Note: This is test shard 8 of 8.
[==========] Running 7 tests from 6 test suites.
[----------] Global test environment set-up.
[----------] 1 test from TableRangeRebalancingTest
[ RUN ] TableRangeRebalancingTest.Basic
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20240502 21:25:20.766214 16382 test_util.cc:268] Using random seed: 1032216797
I20240502 21:25:20.778642 16382 ts_itest-base.cc:112] Starting cluster with:
I20240502 21:25:20.778746 16382 ts_itest-base.cc:113] --------------
I20240502 21:25:20.778872 16382 ts_itest-base.cc:114] 3 tablet servers
I20240502 21:25:20.778939 16382 ts_itest-base.cc:115] 3 replicas per TS
I20240502 21:25:20.779003 16382 ts_itest-base.cc:116] --------------
2024-05-02T21:25:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-02T21:25:20Z Disabled control of system clock
I20240502 21:25:20.802070 16382 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvNciQZ/build/asan/bin/kudu
/tmp/dist-test-taskvNciQZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714685120732928-16382-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714685120732928-16382-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714685120732928-16382-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714685120732928-16382-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.255.190:45405
--webserver_interface=127.15.255.190
--webserver_port=0
--builtin_ntp_servers=127.15.255.148:43351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.255.190:45405
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=true
--enable_range_replica_placement=false with env {}
W20240502 21:25:21.154157 16394 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:25:21.154446 16394 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:25:21.154569 16394 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:25:21.161672 16394 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 21:25:21.161798 16394 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240502 21:25:21.161875 16394 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:25:21.161952 16394 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240502 21:25:21.162015 16394 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240502 21:25:21.170717 16394 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.255.148:43351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714685120732928-16382-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714685120732928-16382-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.15.255.190:45405
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.255.190:45405
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714685120732928-16382-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.15.255.190
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714685120732928-16382-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:25:21.172029 16394 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:25:21.173960 16394 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:25:21.184769 16399 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:21.185184 16400 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:21.185483 16402 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:25:21.186020 16394 server_base.cc:1034] running on GCE node
I20240502 21:25:21.186745 16394 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:25:21.188097 16394 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:25:21.189292 16394 hybrid_clock.cc:648] HybridClock initialized: now 1714685121189247 us; error 39 us; skew 500 ppm
I20240502 21:25:21.189642 16394 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:25:21.191731 16394 webserver.cc:460] Webserver started at http://127.15.255.190:34567/ using document root <none> and password file <none>
I20240502 21:25:21.192489 16394 fs_manager.cc:359] Metadata directory not provided
I20240502 21:25:21.192605 16394 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:25:21.192898 16394 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:25:21.195336 16394 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714685120732928-16382-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "324bea21eb9d4d9bbaeab6cc9ad7d3d1"
format_stamp: "Formatted at 2024-05-02 21:25:21 on dist-test-slave-jdfx"
I20240502 21:25:21.196044 16394 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714685120732928-16382-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "324bea21eb9d4d9bbaeab6cc9ad7d3d1"
format_stamp: "Formatted at 2024-05-02 21:25:21 on dist-test-slave-jdfx"
I20240502 21:25:21.200807 16394 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20240502 21:25:21.203955 16408 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:21.205124 16394 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240502 21:25:21.205287 16394 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714685120732928-16382-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714685120732928-16382-0/raft_consensus-itest-cluster/master-0/wal
uuid: "324bea21eb9d4d9bbaeab6cc9ad7d3d1"
format_stamp: "Formatted at 2024-05-02 21:25:21 on dist-test-slave-jdfx"
I20240502 21:25:21.205463 16394 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714685120732928-16382-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714685120732928-16382-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714685120732928-16382-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:25:21.221899 16394 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:25:21.222616 16394 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:25:21.222878 16394 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:25:21.239320 16394 rpc_server.cc:307] RPC server started. Bound to: 127.15.255.190:45405
I20240502 21:25:21.239327 16461 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.255.190:45405 every 8 connection(s)
I20240502 21:25:21.240684 16394 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714685120732928-16382-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240502 21:25:21.243603 16382 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvNciQZ/build/asan/bin/kudu as pid 16394
I20240502 21:25:21.243855 16382 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714685120732928-16382-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240502 21:25:21.244413 16462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:21.253935 16462 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 324bea21eb9d4d9bbaeab6cc9ad7d3d1: Bootstrap starting.
I20240502 21:25:21.256582 16462 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 324bea21eb9d4d9bbaeab6cc9ad7d3d1: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:21.257849 16462 log.cc:824] T 00000000000000000000000000000000 P 324bea21eb9d4d9bbaeab6cc9ad7d3d1: Log is configured to *not* fsync() on all Append() calls
I20240502 21:25:21.260718 16462 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 324bea21eb9d4d9bbaeab6cc9ad7d3d1: No bootstrap required, opened a new log
I20240502 21:25:21.268047 16462 raft_consensus.cc:348] T 00000000000000000000000000000000 P 324bea21eb9d4d9bbaeab6cc9ad7d3d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "324bea21eb9d4d9bbaeab6cc9ad7d3d1" member_type: VOTER last_known_addr { host: "127.15.255.190" port: 45405 } }
I20240502 21:25:21.268353 16462 raft_consensus.cc:374] T 00000000000000000000000000000000 P 324bea21eb9d4d9bbaeab6cc9ad7d3d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:21.268484 16462 raft_consensus.cc:729] T 00000000000000000000000000000000 P 324bea21eb9d4d9bbaeab6cc9ad7d3d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 324bea21eb9d4d9bbaeab6cc9ad7d3d1, State: Initialized, Role: FOLLOWER
I20240502 21:25:21.269186 16462 consensus_queue.cc:260] T 00000000000000000000000000000000 P 324bea21eb9d4d9bbaeab6cc9ad7d3d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "324bea21eb9d4d9bbaeab6cc9ad7d3d1" member_type: VOTER last_known_addr { host: "127.15.255.190" port: 45405 } }
I20240502 21:25:21.269439 16462 raft_consensus.cc:388] T 00000000000000000000000000000000 P 324bea21eb9d4d9bbaeab6cc9ad7d3d1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 21:25:21.269570 16462 raft_consensus.cc:482] T 00000000000000000000000000000000 P 324bea21eb9d4d9bbaeab6cc9ad7d3d1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 21:25:21.269754 16462 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 324bea21eb9d4d9bbaeab6cc9ad7d3d1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:21.271842 16462 raft_consensus.cc:504] T 00000000000000000000000000000000 P 324bea21eb9d4d9bbaeab6cc9ad7d3d1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "324bea21eb9d4d9bbaeab6cc9ad7d3d1" member_type: VOTER last_known_addr { host: "127.15.255.190" port: 45405 } }
I20240502 21:25:21.272240 16462 leader_election.cc:302] T 00000000000000000000000000000000 P 324bea21eb9d4d9bbaeab6cc9ad7d3d1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 324bea21eb9d4d9bbaeab6cc9ad7d3d1; no voters:
I20240502 21:25:21.272814 16462 leader_election.cc:288] T 00000000000000000000000000000000 P 324bea21eb9d4d9bbaeab6cc9ad7d3d1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 21:25:21.272939 16467 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 324bea21eb9d4d9bbaeab6cc9ad7d3d1 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:25:21.273512 16467 raft_consensus.cc:686] T 00000000000000000000000000000000 P 324bea21eb9d4d9bbaeab6cc9ad7d3d1 [term 1 LEADER]: Becoming Leader. State: Replica: 324bea21eb9d4d9bbaeab6cc9ad7d3d1, State: Running, Role: LEADER
I20240502 21:25:21.274058 16467 consensus_queue.cc:237] T 00000000000000000000000000000000 P 324bea21eb9d4d9bbaeab6cc9ad7d3d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "324bea21eb9d4d9bbaeab6cc9ad7d3d1" member_type: VOTER last_known_addr { host: "127.15.255.190" port: 45405 } }
I20240502 21:25:21.274585 16462 sys_catalog.cc:564] T 00000000000000000000000000000000 P 324bea21eb9d4d9bbaeab6cc9ad7d3d1 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 21:25:21.277352 16469 sys_catalog.cc:455] T 00000000000000000000000000000000 P 324bea21eb9d4d9bbaeab6cc9ad7d3d1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 324bea21eb9d4d9bbaeab6cc9ad7d3d1. Latest consensus state: current_term: 1 leader_uuid: "324bea21eb9d4d9bbaeab6cc9ad7d3d1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "324bea21eb9d4d9bbaeab6cc9ad7d3d1" member_type: VOTER last_known_addr { host: "127.15.255.190" port: 45405 } } }
I20240502 21:25:21.277658 16469 sys_catalog.cc:458] T 00000000000000000000000000000000 P 324bea21eb9d4d9bbaeab6cc9ad7d3d1 [sys.catalog]: This master's current role is: LEADER
I20240502 21:25:21.278051 16468 sys_catalog.cc:455] T 00000000000000000000000000000000 P 324bea21eb9d4d9bbaeab6cc9ad7d3d1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "324bea21eb9d4d9bbaeab6cc9ad7d3d1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "324bea21eb9d4d9bbaeab6cc9ad7d3d1" member_type: VOTER last_known_addr { host: "127.15.255.190" port: 45405 } } }
I20240502 21:25:21.278314 16468 sys_catalog.cc:458] T 00000000000000000000000000000000 P 324bea21eb9d4d9bbaeab6cc9ad7d3d1 [sys.catalog]: This master's current role is: LEADER
I20240502 21:25:21.279060 16476 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 21:25:21.283351 16476 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 21:25:21.290752 16476 catalog_manager.cc:1347] Generated new cluster ID: b174137370794afba4b47920b8fd9353
I20240502 21:25:21.290840 16476 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 21:25:21.298691 16476 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 21:25:21.299742 16476 catalog_manager.cc:1504] Loading token signing keys...
I20240502 21:25:21.306938 16476 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 324bea21eb9d4d9bbaeab6cc9ad7d3d1: Generated new TSK 0
I20240502 21:25:21.307783 16476 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 21:25:21.320402 16382 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvNciQZ/build/asan/bin/kudu
/tmp/dist-test-taskvNciQZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.255.129:0
--local_ip_for_outbound_sockets=127.15.255.129
--webserver_interface=127.15.255.129
--webserver_port=0
--tserver_master_addrs=127.15.255.190:45405
--builtin_ntp_servers=127.15.255.148:43351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240502 21:25:21.637583 16486 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:25:21.637852 16486 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:25:21.637961 16486 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:25:21.645318 16486 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 21:25:21.645431 16486 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:25:21.645581 16486 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.255.129
I20240502 21:25:21.654265 16486 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.255.148:43351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.255.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.15.255.129
--webserver_port=0
--tserver_master_addrs=127.15.255.190:45405
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.255.129
--log_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:25:21.655565 16486 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:25:21.657236 16486 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:25:21.668574 16493 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:21.668887 16494 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:21.669466 16496 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:25:21.670091 16486 server_base.cc:1034] running on GCE node
I20240502 21:25:21.670663 16486 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:25:21.671533 16486 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:25:21.672724 16486 hybrid_clock.cc:648] HybridClock initialized: now 1714685121672673 us; error 38 us; skew 500 ppm
I20240502 21:25:21.673023 16486 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:25:21.675143 16486 webserver.cc:460] Webserver started at http://127.15.255.129:42323/ using document root <none> and password file <none>
I20240502 21:25:21.675843 16486 fs_manager.cc:359] Metadata directory not provided
I20240502 21:25:21.675947 16486 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:25:21.676261 16486 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:25:21.678789 16486 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "9efea4678e0e4dee8fe588222a3a6a67"
format_stamp: "Formatted at 2024-05-02 21:25:21 on dist-test-slave-jdfx"
I20240502 21:25:21.679474 16486 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "9efea4678e0e4dee8fe588222a3a6a67"
format_stamp: "Formatted at 2024-05-02 21:25:21 on dist-test-slave-jdfx"
I20240502 21:25:21.684182 16486 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20240502 21:25:21.687304 16502 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:21.688602 16486 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20240502 21:25:21.688752 16486 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "9efea4678e0e4dee8fe588222a3a6a67"
format_stamp: "Formatted at 2024-05-02 21:25:21 on dist-test-slave-jdfx"
I20240502 21:25:21.688927 16486 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:25:21.701673 16486 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:25:21.702383 16486 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:25:21.702628 16486 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:25:21.703568 16486 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:25:21.704923 16486 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:25:21.705017 16486 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:21.705122 16486 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:25:21.705191 16486 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:21.734544 16486 rpc_server.cc:307] RPC server started. Bound to: 127.15.255.129:37931
I20240502 21:25:21.734599 16616 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.255.129:37931 every 8 connection(s)
I20240502 21:25:21.735812 16486 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240502 21:25:21.736369 16382 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvNciQZ/build/asan/bin/kudu as pid 16486
I20240502 21:25:21.736611 16382 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240502 21:25:21.740355 16382 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvNciQZ/build/asan/bin/kudu
/tmp/dist-test-taskvNciQZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.255.130:0
--local_ip_for_outbound_sockets=127.15.255.130
--webserver_interface=127.15.255.130
--webserver_port=0
--tserver_master_addrs=127.15.255.190:45405
--builtin_ntp_servers=127.15.255.148:43351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 21:25:21.748380 16617 heartbeater.cc:347] Connected to a master server at 127.15.255.190:45405
I20240502 21:25:21.748689 16617 heartbeater.cc:464] Registering TS with master...
I20240502 21:25:21.749472 16617 heartbeater.cc:511] Master 127.15.255.190:45405 requested a full tablet report, sending...
I20240502 21:25:21.751454 16426 ts_manager.cc:194] Registered new tserver with Master: 9efea4678e0e4dee8fe588222a3a6a67 (127.15.255.129:37931)
I20240502 21:25:21.753064 16426 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.255.129:38673
W20240502 21:25:22.059119 16620 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:25:22.059401 16620 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:25:22.059541 16620 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:25:22.067096 16620 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 21:25:22.067232 16620 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:25:22.067381 16620 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.255.130
I20240502 21:25:22.076817 16620 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.255.148:43351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.255.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.15.255.130
--webserver_port=0
--tserver_master_addrs=127.15.255.190:45405
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.255.130
--log_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:25:22.078060 16620 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:25:22.079782 16620 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:25:22.091277 16626 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:22.091799 16625 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:22.092209 16628 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:25:22.092809 16620 server_base.cc:1034] running on GCE node
I20240502 21:25:22.093333 16620 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:25:22.094216 16620 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:25:22.095403 16620 hybrid_clock.cc:648] HybridClock initialized: now 1714685122095358 us; error 38 us; skew 500 ppm
I20240502 21:25:22.095685 16620 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:25:22.097718 16620 webserver.cc:460] Webserver started at http://127.15.255.130:44567/ using document root <none> and password file <none>
I20240502 21:25:22.098356 16620 fs_manager.cc:359] Metadata directory not provided
I20240502 21:25:22.098457 16620 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:25:22.098740 16620 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:25:22.101142 16620 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "1cfec8b74a2342cca5723d915046f117"
format_stamp: "Formatted at 2024-05-02 21:25:22 on dist-test-slave-jdfx"
I20240502 21:25:22.101830 16620 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "1cfec8b74a2342cca5723d915046f117"
format_stamp: "Formatted at 2024-05-02 21:25:22 on dist-test-slave-jdfx"
I20240502 21:25:22.106441 16620 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20240502 21:25:22.109596 16634 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:22.110781 16620 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20240502 21:25:22.110920 16620 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "1cfec8b74a2342cca5723d915046f117"
format_stamp: "Formatted at 2024-05-02 21:25:22 on dist-test-slave-jdfx"
I20240502 21:25:22.111104 16620 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:25:22.128352 16620 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:25:22.129063 16620 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:25:22.129305 16620 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:25:22.130122 16620 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:25:22.131456 16620 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:25:22.131547 16620 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:22.131644 16620 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:25:22.131711 16620 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:22.159770 16620 rpc_server.cc:307] RPC server started. Bound to: 127.15.255.130:43833
I20240502 21:25:22.159830 16748 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.255.130:43833 every 8 connection(s)
I20240502 21:25:22.160943 16620 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240502 21:25:22.166051 16382 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvNciQZ/build/asan/bin/kudu as pid 16620
I20240502 21:25:22.166208 16382 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240502 21:25:22.169204 16382 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvNciQZ/build/asan/bin/kudu
/tmp/dist-test-taskvNciQZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.255.131:0
--local_ip_for_outbound_sockets=127.15.255.131
--webserver_interface=127.15.255.131
--webserver_port=0
--tserver_master_addrs=127.15.255.190:45405
--builtin_ntp_servers=127.15.255.148:43351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 21:25:22.172749 16749 heartbeater.cc:347] Connected to a master server at 127.15.255.190:45405
I20240502 21:25:22.173012 16749 heartbeater.cc:464] Registering TS with master...
I20240502 21:25:22.173893 16749 heartbeater.cc:511] Master 127.15.255.190:45405 requested a full tablet report, sending...
I20240502 21:25:22.175343 16426 ts_manager.cc:194] Registered new tserver with Master: 1cfec8b74a2342cca5723d915046f117 (127.15.255.130:43833)
I20240502 21:25:22.176090 16426 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.255.130:41357
W20240502 21:25:22.480319 16752 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:25:22.480584 16752 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:25:22.480712 16752 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:25:22.487793 16752 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 21:25:22.487898 16752 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:25:22.488037 16752 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.255.131
I20240502 21:25:22.496517 16752 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.255.148:43351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.255.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.15.255.131
--webserver_port=0
--tserver_master_addrs=127.15.255.190:45405
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.255.131
--log_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:25:22.497614 16752 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:25:22.499264 16752 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:25:22.510596 16757 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:22.511106 16758 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:22.511538 16760 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:25:22.512044 16752 server_base.cc:1034] running on GCE node
I20240502 21:25:22.512578 16752 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:25:22.513381 16752 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:25:22.514566 16752 hybrid_clock.cc:648] HybridClock initialized: now 1714685122514521 us; error 35 us; skew 500 ppm
I20240502 21:25:22.514876 16752 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:25:22.516888 16752 webserver.cc:460] Webserver started at http://127.15.255.131:36119/ using document root <none> and password file <none>
I20240502 21:25:22.517517 16752 fs_manager.cc:359] Metadata directory not provided
I20240502 21:25:22.517616 16752 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:25:22.517910 16752 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:25:22.520252 16752 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "8df3306d4bbe4fb1af0afb8d67c81798"
format_stamp: "Formatted at 2024-05-02 21:25:22 on dist-test-slave-jdfx"
I20240502 21:25:22.521695 16752 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "8df3306d4bbe4fb1af0afb8d67c81798"
format_stamp: "Formatted at 2024-05-02 21:25:22 on dist-test-slave-jdfx"
I20240502 21:25:22.526286 16752 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20240502 21:25:22.529255 16766 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:22.530401 16752 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240502 21:25:22.530542 16752 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "8df3306d4bbe4fb1af0afb8d67c81798"
format_stamp: "Formatted at 2024-05-02 21:25:22 on dist-test-slave-jdfx"
I20240502 21:25:22.530722 16752 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:25:22.546126 16752 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:25:22.546806 16752 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:25:22.547045 16752 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:25:22.547875 16752 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:25:22.549219 16752 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:25:22.549311 16752 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:22.549414 16752 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:25:22.549482 16752 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:22.577804 16752 rpc_server.cc:307] RPC server started. Bound to: 127.15.255.131:37673
I20240502 21:25:22.577840 16883 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.255.131:37673 every 8 connection(s)
I20240502 21:25:22.579017 16752 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240502 21:25:22.585094 16382 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvNciQZ/build/asan/bin/kudu as pid 16752
I20240502 21:25:22.585264 16382 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TableRangeRebalancingTest.Basic.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240502 21:25:22.590940 16884 heartbeater.cc:347] Connected to a master server at 127.15.255.190:45405
I20240502 21:25:22.591181 16884 heartbeater.cc:464] Registering TS with master...
I20240502 21:25:22.591893 16884 heartbeater.cc:511] Master 127.15.255.190:45405 requested a full tablet report, sending...
I20240502 21:25:22.593096 16426 ts_manager.cc:194] Registered new tserver with Master: 8df3306d4bbe4fb1af0afb8d67c81798 (127.15.255.131:37673)
I20240502 21:25:22.593881 16426 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.255.131:45789
I20240502 21:25:22.600853 16382 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20240502 21:25:22.618633 16382 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20240502 21:25:22.755993 16617 heartbeater.cc:503] Master 127.15.255.190:45405 was elected leader, sending a full tablet report...
I20240502 21:25:23.178261 16749 heartbeater.cc:503] Master 127.15.255.190:45405 was elected leader, sending a full tablet report...
I20240502 21:25:23.595916 16884 heartbeater.cc:503] Master 127.15.255.190:45405 was elected leader, sending a full tablet report...
I20240502 21:25:26.372682 16426 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60624:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\200\007\001\000\000\000\000\000\006\001\000\000\000\000\000\007\001\000\377\377\377\177"
indirect_data: "<redacted>"""
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 8
seed: 0
}
}
W20240502 21:25:26.379189 16426 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table rebalance_test_table_0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240502 21:25:26.425980 16812 tablet_service.cc:1467] Processing CreateTablet for tablet 71d05c4f1e6f4f7186ed7b8e157b7d2f (DEFAULT_TABLE table=rebalance_test_table_0 [id=a460252f532043d789e8c746446d5964]), partition=HASH (key) PARTITION 3, RANGE (key) PARTITION -2147483648 <= VALUES < 0
I20240502 21:25:26.427918 16812 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71d05c4f1e6f4f7186ed7b8e157b7d2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:26.427346 16549 tablet_service.cc:1467] Processing CreateTablet for tablet 38999367825249e1a0f5f156bd3c3b25 (DEFAULT_TABLE table=rebalance_test_table_0 [id=a460252f532043d789e8c746446d5964]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION 0 <= VALUES < 2147483647
I20240502 21:25:26.428843 16811 tablet_service.cc:1467] Processing CreateTablet for tablet 61268934b6b74fad90db0a52dd48ac82 (DEFAULT_TABLE table=rebalance_test_table_0 [id=a460252f532043d789e8c746446d5964]), partition=HASH (key) PARTITION 3, RANGE (key) PARTITION 0 <= VALUES < 2147483647
I20240502 21:25:26.429132 16549 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38999367825249e1a0f5f156bd3c3b25. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:26.429314 16811 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61268934b6b74fad90db0a52dd48ac82. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:26.431107 16681 tablet_service.cc:1467] Processing CreateTablet for tablet 38999367825249e1a0f5f156bd3c3b25 (DEFAULT_TABLE table=rebalance_test_table_0 [id=a460252f532043d789e8c746446d5964]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION 0 <= VALUES < 2147483647
I20240502 21:25:26.431159 16682 tablet_service.cc:1467] Processing CreateTablet for tablet 05a7626e52ae43a6829feacaf1837ea5 (DEFAULT_TABLE table=rebalance_test_table_0 [id=a460252f532043d789e8c746446d5964]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION -2147483648 <= VALUES < 0
I20240502 21:25:26.432937 16681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38999367825249e1a0f5f156bd3c3b25. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:26.432835 16810 tablet_service.cc:1467] Processing CreateTablet for tablet c06e83b5216e4138817cc58a61c02e79 (DEFAULT_TABLE table=rebalance_test_table_0 [id=a460252f532043d789e8c746446d5964]), partition=HASH (key) PARTITION 4, RANGE (key) PARTITION -2147483648 <= VALUES < 0
I20240502 21:25:26.433135 16548 tablet_service.cc:1467] Processing CreateTablet for tablet 1b9f6d5e769e48f394fe59a835c22fc0 (DEFAULT_TABLE table=rebalance_test_table_0 [id=a460252f532043d789e8c746446d5964]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION -2147483648 <= VALUES < 0
I20240502 21:25:26.433329 16810 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c06e83b5216e4138817cc58a61c02e79. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:26.433681 16548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b9f6d5e769e48f394fe59a835c22fc0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:26.433730 16682 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05a7626e52ae43a6829feacaf1837ea5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:26.434832 16809 tablet_service.cc:1467] Processing CreateTablet for tablet ed40aed15ca3447aa01283771ac257ed (DEFAULT_TABLE table=rebalance_test_table_0 [id=a460252f532043d789e8c746446d5964]), partition=HASH (key) PARTITION 4, RANGE (key) PARTITION 0 <= VALUES < 2147483647
I20240502 21:25:26.435439 16809 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed40aed15ca3447aa01283771ac257ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:26.436743 16808 tablet_service.cc:1467] Processing CreateTablet for tablet 1173ce275c4f40f0a0a31729310fbefd (DEFAULT_TABLE table=rebalance_test_table_0 [id=a460252f532043d789e8c746446d5964]), partition=HASH (key) PARTITION 5, RANGE (key) PARTITION -2147483648 <= VALUES < 0
I20240502 21:25:26.437201 16808 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1173ce275c4f40f0a0a31729310fbefd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:26.438103 16547 tablet_service.cc:1467] Processing CreateTablet for tablet c5ea5b6dcf934533ba8df41e52711337 (DEFAULT_TABLE table=rebalance_test_table_0 [id=a460252f532043d789e8c746446d5964]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION 0 <= VALUES < 2147483647
I20240502 21:25:26.438160 16813 tablet_service.cc:1467] Processing CreateTablet for tablet cb358d933789476898f74949ffa81e49 (DEFAULT_TABLE table=rebalance_test_table_0 [id=a460252f532043d789e8c746446d5964]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION 0 <= VALUES < 2147483647
I20240502 21:25:26.438612 16813 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb358d933789476898f74949ffa81e49. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:26.422441 16550 tablet_service.cc:1467] Processing CreateTablet for tablet 05a7626e52ae43a6829feacaf1837ea5 (DEFAULT_TABLE table=rebalance_test_table_0 [id=a460252f532043d789e8c746446d5964]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION -2147483648 <= VALUES < 0
I20240502 21:25:26.440021 16815 tablet_service.cc:1467] Processing CreateTablet for tablet c5ea5b6dcf934533ba8df41e52711337 (DEFAULT_TABLE table=rebalance_test_table_0 [id=a460252f532043d789e8c746446d5964]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION 0 <= VALUES < 2147483647
I20240502 21:25:26.440440 16815 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5ea5b6dcf934533ba8df41e52711337. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:26.441119 16817 tablet_service.cc:1467] Processing CreateTablet for tablet 38999367825249e1a0f5f156bd3c3b25 (DEFAULT_TABLE table=rebalance_test_table_0 [id=a460252f532043d789e8c746446d5964]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION 0 <= VALUES < 2147483647
I20240502 21:25:26.441542 16817 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38999367825249e1a0f5f156bd3c3b25. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:26.438453 16545 tablet_service.cc:1467] Processing CreateTablet for tablet cb358d933789476898f74949ffa81e49 (DEFAULT_TABLE table=rebalance_test_table_0 [id=a460252f532043d789e8c746446d5964]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION 0 <= VALUES < 2147483647
I20240502 21:25:26.439131 16543 tablet_service.cc:1467] Processing CreateTablet for tablet 61268934b6b74fad90db0a52dd48ac82 (DEFAULT_TABLE table=rebalance_test_table_0 [id=a460252f532043d789e8c746446d5964]), partition=HASH (key) PARTITION 3, RANGE (key) PARTITION 0 <= VALUES < 2147483647
I20240502 21:25:26.442453 16807 tablet_service.cc:1467] Processing CreateTablet for tablet 4cfc1f2146dd449c87a46196e25280c2 (DEFAULT_TABLE table=rebalance_test_table_0 [id=a460252f532043d789e8c746446d5964]), partition=HASH (key) PARTITION 5, RANGE (key) PARTITION 0 <= VALUES < 2147483647
I20240502 21:25:26.439263 16546 tablet_service.cc:1467] Processing CreateTablet for tablet 2b1fbeb84317494595f7262fe7857ebd (DEFAULT_TABLE table=rebalance_test_table_0 [id=a460252f532043d789e8c746446d5964]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION -2147483648 <= VALUES < 0
I20240502 21:25:26.442898 16807 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4cfc1f2146dd449c87a46196e25280c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:26.442919 16547 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5ea5b6dcf934533ba8df41e52711337. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:26.426432 16814 tablet_service.cc:1467] Processing CreateTablet for tablet 2b1fbeb84317494595f7262fe7857ebd (DEFAULT_TABLE table=rebalance_test_table_0 [id=a460252f532043d789e8c746446d5964]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION -2147483648 <= VALUES < 0
I20240502 21:25:26.444347 16814 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b1fbeb84317494595f7262fe7857ebd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:26.439647 16541 tablet_service.cc:1467] Processing CreateTablet for tablet ed40aed15ca3447aa01283771ac257ed (DEFAULT_TABLE table=rebalance_test_table_0 [id=a460252f532043d789e8c746446d5964]), partition=HASH (key) PARTITION 4, RANGE (key) PARTITION 0 <= VALUES < 2147483647
I20240502 21:25:26.445112 16816 tablet_service.cc:1467] Processing CreateTablet for tablet 1b9f6d5e769e48f394fe59a835c22fc0 (DEFAULT_TABLE table=rebalance_test_table_0 [id=a460252f532043d789e8c746446d5964]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION -2147483648 <= VALUES < 0
I20240502 21:25:26.445552 16816 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b9f6d5e769e48f394fe59a835c22fc0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:26.446460 16805 tablet_service.cc:1467] Processing CreateTablet for tablet e882da2f771249f8a3626d1a7a35de34 (DEFAULT_TABLE table=rebalance_test_table_0 [id=a460252f532043d789e8c746446d5964]), partition=HASH (key) PARTITION 6, RANGE (key) PARTITION 0 <= VALUES < 2147483647
I20240502 21:25:26.446930 16805 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e882da2f771249f8a3626d1a7a35de34. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:26.447182 16680 tablet_service.cc:1467] Processing CreateTablet for tablet 1b9f6d5e769e48f394fe59a835c22fc0 (DEFAULT_TABLE table=rebalance_test_table_0 [id=a460252f532043d789e8c746446d5964]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION -2147483648 <= VALUES < 0
I20240502 21:25:26.447678 16680 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b9f6d5e769e48f394fe59a835c22fc0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:26.447573 16818 tablet_service.cc:1467] Processing CreateTablet for tablet 05a7626e52ae43a6829feacaf1837ea5 (DEFAULT_TABLE table=rebalance_test_table_0 [id=a460252f532043d789e8c746446d5964]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION -2147483648 <= VALUES < 0
I20240502 21:25:26.447970 16818 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05a7626e52ae43a6829feacaf1837ea5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:26.448575 16550 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05a7626e52ae43a6829feacaf1837ea5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:26.448966 16806 tablet_service.cc:1467] Processing CreateTablet for tablet 610bb5f6731047b3b86883445a9e05bb (DEFAULT_TABLE table=rebalance_test_table_0 [id=a460252f532043d789e8c746446d5964]), partition=HASH (key) PARTITION 6, RANGE (key) PARTITION -2147483648 <= VALUES < 0
I20240502 21:25:26.439980 16544 tablet_service.cc:1467] Processing CreateTablet for tablet 71d05c4f1e6f4f7186ed7b8e157b7d2f (DEFAULT_TABLE table=rebalance_test_table_0 [id=a460252f532043d789e8c746446d5964]), partition=HASH (key) PARTITION 3, RANGE (key) PARTITION -2147483648 <= VALUES < 0
I20240502 21:25:26.440676 16542 tablet_service.cc:1467] Processing CreateTablet for tablet c06e83b5216e4138817cc58a61c02e79 (DEFAULT_TABLE table=rebalance_test_table_0 [id=a460252f532043d789e8c746446d5964]), partition=HASH (key) PARTITION 4, RANGE (key) PARTITION -2147483648 <= VALUES < 0
I20240502 21:25:26.450009 16802 tablet_service.cc:1467] Processing CreateTablet for tablet 72f6a167c03d4ee6bd72d6e5793f784d (DEFAULT_TABLE table=rebalance_test_table_0 [id=a460252f532043d789e8c746446d5964]), partition=HASH (key) PARTITION 7, RANGE (key) PARTITION 0 <= VALUES < 2147483647
I20240502 21:25:26.450524 16802 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72f6a167c03d4ee6bd72d6e5793f784d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:26.441498 16540 tablet_service.cc:1467] Processing CreateTablet for tablet 1173ce275c4f40f0a0a31729310fbefd (DEFAULT_TABLE table=rebalance_test_table_0 [id=a460252f532043d789e8c746446d5964]), partition=HASH (key) PARTITION 5, RANGE (key) PARTITION -2147483648 <= VALUES < 0
I20240502 21:25:26.451262 16806 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 610bb5f6731047b3b86883445a9e05bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:26.451934 16903 tablet_bootstrap.cc:492] T 05a7626e52ae43a6829feacaf1837ea5 P 1cfec8b74a2342cca5723d915046f117: Bootstrap starting.
I20240502 21:25:26.452173 16803 tablet_service.cc:1467] Processing CreateTablet for tablet 62a4fb5953dc4fa7badc5e2ee1ed4261 (DEFAULT_TABLE table=rebalance_test_table_0 [id=a460252f532043d789e8c746446d5964]), partition=HASH (key) PARTITION 7, RANGE (key) PARTITION -2147483648 <= VALUES < 0
I20240502 21:25:26.452867 16803 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62a4fb5953dc4fa7badc5e2ee1ed4261. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:26.454398 16680 tablet_service.cc:1467] Processing CreateTablet for tablet c5ea5b6dcf934533ba8df41e52711337 (DEFAULT_TABLE table=rebalance_test_table_0 [id=a460252f532043d789e8c746446d5964]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION 0 <= VALUES < 2147483647
I20240502 21:25:26.454735 16680 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5ea5b6dcf934533ba8df41e52711337. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:26.469331 16679 tablet_service.cc:1467] Processing CreateTablet for tablet 2b1fbeb84317494595f7262fe7857ebd (DEFAULT_TABLE table=rebalance_test_table_0 [id=a460252f532043d789e8c746446d5964]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION -2147483648 <= VALUES < 0
I20240502 21:25:26.469767 16679 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b1fbeb84317494595f7262fe7857ebd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:26.476315 16678 tablet_service.cc:1467] Processing CreateTablet for tablet 71d05c4f1e6f4f7186ed7b8e157b7d2f (DEFAULT_TABLE table=rebalance_test_table_0 [id=a460252f532043d789e8c746446d5964]), partition=HASH (key) PARTITION 3, RANGE (key) PARTITION -2147483648 <= VALUES < 0
I20240502 21:25:26.476824 16678 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71d05c4f1e6f4f7186ed7b8e157b7d2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:26.482892 16677 tablet_service.cc:1467] Processing CreateTablet for tablet 61268934b6b74fad90db0a52dd48ac82 (DEFAULT_TABLE table=rebalance_test_table_0 [id=a460252f532043d789e8c746446d5964]), partition=HASH (key) PARTITION 3, RANGE (key) PARTITION 0 <= VALUES < 2147483647
I20240502 21:25:26.483335 16677 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61268934b6b74fad90db0a52dd48ac82. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:26.489331 16676 tablet_service.cc:1467] Processing CreateTablet for tablet ed40aed15ca3447aa01283771ac257ed (DEFAULT_TABLE table=rebalance_test_table_0 [id=a460252f532043d789e8c746446d5964]), partition=HASH (key) PARTITION 4, RANGE (key) PARTITION 0 <= VALUES < 2147483647
I20240502 21:25:26.489953 16676 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed40aed15ca3447aa01283771ac257ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:26.490049 16673 tablet_service.cc:1467] Processing CreateTablet for tablet 610bb5f6731047b3b86883445a9e05bb (DEFAULT_TABLE table=rebalance_test_table_0 [id=a460252f532043d789e8c746446d5964]), partition=HASH (key) PARTITION 6, RANGE (key) PARTITION -2147483648 <= VALUES < 0
I20240502 21:25:26.490552 16673 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 610bb5f6731047b3b86883445a9e05bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:26.491962 16674 tablet_service.cc:1467] Processing CreateTablet for tablet 4cfc1f2146dd449c87a46196e25280c2 (DEFAULT_TABLE table=rebalance_test_table_0 [id=a460252f532043d789e8c746446d5964]), partition=HASH (key) PARTITION 5, RANGE (key) PARTITION 0 <= VALUES < 2147483647
I20240502 21:25:26.492542 16674 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4cfc1f2146dd449c87a46196e25280c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:26.495640 16538 tablet_service.cc:1467] Processing CreateTablet for tablet e882da2f771249f8a3626d1a7a35de34 (DEFAULT_TABLE table=rebalance_test_table_0 [id=a460252f532043d789e8c746446d5964]), partition=HASH (key) PARTITION 6, RANGE (key) PARTITION 0 <= VALUES < 2147483647
I20240502 21:25:26.497119 16675 tablet_service.cc:1467] Processing CreateTablet for tablet 1173ce275c4f40f0a0a31729310fbefd (DEFAULT_TABLE table=rebalance_test_table_0 [id=a460252f532043d789e8c746446d5964]), partition=HASH (key) PARTITION 5, RANGE (key) PARTITION -2147483648 <= VALUES < 0
I20240502 21:25:26.497670 16675 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1173ce275c4f40f0a0a31729310fbefd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:26.498368 16545 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb358d933789476898f74949ffa81e49. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:26.499073 16547 tablet_service.cc:1467] Processing CreateTablet for tablet 610bb5f6731047b3b86883445a9e05bb (DEFAULT_TABLE table=rebalance_test_table_0 [id=a460252f532043d789e8c746446d5964]), partition=HASH (key) PARTITION 6, RANGE (key) PARTITION -2147483648 <= VALUES < 0
I20240502 21:25:26.499756 16543 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61268934b6b74fad90db0a52dd48ac82. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:26.501359 16678 tablet_service.cc:1467] Processing CreateTablet for tablet c06e83b5216e4138817cc58a61c02e79 (DEFAULT_TABLE table=rebalance_test_table_0 [id=a460252f532043d789e8c746446d5964]), partition=HASH (key) PARTITION 4, RANGE (key) PARTITION -2147483648 <= VALUES < 0
I20240502 21:25:26.501394 16536 tablet_service.cc:1467] Processing CreateTablet for tablet 72f6a167c03d4ee6bd72d6e5793f784d (DEFAULT_TABLE table=rebalance_test_table_0 [id=a460252f532043d789e8c746446d5964]), partition=HASH (key) PARTITION 7, RANGE (key) PARTITION 0 <= VALUES < 2147483647
I20240502 21:25:26.501718 16678 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c06e83b5216e4138817cc58a61c02e79. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:26.502872 16539 tablet_service.cc:1467] Processing CreateTablet for tablet 4cfc1f2146dd449c87a46196e25280c2 (DEFAULT_TABLE table=rebalance_test_table_0 [id=a460252f532043d789e8c746446d5964]), partition=HASH (key) PARTITION 5, RANGE (key) PARTITION 0 <= VALUES < 2147483647
I20240502 21:25:26.505064 16542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c06e83b5216e4138817cc58a61c02e79. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:26.504973 16537 tablet_service.cc:1467] Processing CreateTablet for tablet 62a4fb5953dc4fa7badc5e2ee1ed4261 (DEFAULT_TABLE table=rebalance_test_table_0 [id=a460252f532043d789e8c746446d5964]), partition=HASH (key) PARTITION 7, RANGE (key) PARTITION -2147483648 <= VALUES < 0
I20240502 21:25:26.506517 16540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1173ce275c4f40f0a0a31729310fbefd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:26.507575 16546 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b1fbeb84317494595f7262fe7857ebd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:26.504871 16679 tablet_service.cc:1467] Processing CreateTablet for tablet cb358d933789476898f74949ffa81e49 (DEFAULT_TABLE table=rebalance_test_table_0 [id=a460252f532043d789e8c746446d5964]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION 0 <= VALUES < 2147483647
I20240502 21:25:26.508971 16544 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71d05c4f1e6f4f7186ed7b8e157b7d2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:26.509071 16679 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb358d933789476898f74949ffa81e49. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:26.510013 16541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed40aed15ca3447aa01283771ac257ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:26.510035 16676 tablet_service.cc:1467] Processing CreateTablet for tablet 72f6a167c03d4ee6bd72d6e5793f784d (DEFAULT_TABLE table=rebalance_test_table_0 [id=a460252f532043d789e8c746446d5964]), partition=HASH (key) PARTITION 7, RANGE (key) PARTITION 0 <= VALUES < 2147483647
I20240502 21:25:26.509485 16674 tablet_service.cc:1467] Processing CreateTablet for tablet 62a4fb5953dc4fa7badc5e2ee1ed4261 (DEFAULT_TABLE table=rebalance_test_table_0 [id=a460252f532043d789e8c746446d5964]), partition=HASH (key) PARTITION 7, RANGE (key) PARTITION -2147483648 <= VALUES < 0
I20240502 21:25:26.510581 16675 tablet_service.cc:1467] Processing CreateTablet for tablet e882da2f771249f8a3626d1a7a35de34 (DEFAULT_TABLE table=rebalance_test_table_0 [id=a460252f532043d789e8c746446d5964]), partition=HASH (key) PARTITION 6, RANGE (key) PARTITION 0 <= VALUES < 2147483647
I20240502 21:25:26.512910 16536 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72f6a167c03d4ee6bd72d6e5793f784d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:26.512570 16676 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72f6a167c03d4ee6bd72d6e5793f784d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:26.513989 16539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4cfc1f2146dd449c87a46196e25280c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:26.515071 16537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62a4fb5953dc4fa7badc5e2ee1ed4261. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:26.515165 16674 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62a4fb5953dc4fa7badc5e2ee1ed4261. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:26.516129 16538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e882da2f771249f8a3626d1a7a35de34. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:26.516227 16903 tablet_bootstrap.cc:654] T 05a7626e52ae43a6829feacaf1837ea5 P 1cfec8b74a2342cca5723d915046f117: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:26.516255 16675 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e882da2f771249f8a3626d1a7a35de34. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:26.519397 16547 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 610bb5f6731047b3b86883445a9e05bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:26.538504 16903 log.cc:824] T 05a7626e52ae43a6829feacaf1837ea5 P 1cfec8b74a2342cca5723d915046f117: Log is configured to *not* fsync() on all Append() calls
I20240502 21:25:26.539959 16905 tablet_bootstrap.cc:492] T 1b9f6d5e769e48f394fe59a835c22fc0 P 9efea4678e0e4dee8fe588222a3a6a67: Bootstrap starting.
I20240502 21:25:26.542315 16903 tablet_bootstrap.cc:492] T 05a7626e52ae43a6829feacaf1837ea5 P 1cfec8b74a2342cca5723d915046f117: No bootstrap required, opened a new log
I20240502 21:25:26.542610 16903 ts_tablet_manager.cc:1367] T 05a7626e52ae43a6829feacaf1837ea5 P 1cfec8b74a2342cca5723d915046f117: Time spent bootstrapping tablet: real 0.091s user 0.007s sys 0.000s
I20240502 21:25:26.551544 16903 raft_consensus.cc:348] T 05a7626e52ae43a6829feacaf1837ea5 P 1cfec8b74a2342cca5723d915046f117 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9efea4678e0e4dee8fe588222a3a6a67" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37931 } } peers { permanent_uuid: "8df3306d4bbe4fb1af0afb8d67c81798" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37673 } } peers { permanent_uuid: "1cfec8b74a2342cca5723d915046f117" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 43833 } }
I20240502 21:25:26.551993 16903 raft_consensus.cc:374] T 05a7626e52ae43a6829feacaf1837ea5 P 1cfec8b74a2342cca5723d915046f117 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:26.552122 16903 raft_consensus.cc:729] T 05a7626e52ae43a6829feacaf1837ea5 P 1cfec8b74a2342cca5723d915046f117 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cfec8b74a2342cca5723d915046f117, State: Initialized, Role: FOLLOWER
I20240502 21:25:26.552845 16903 consensus_queue.cc:260] T 05a7626e52ae43a6829feacaf1837ea5 P 1cfec8b74a2342cca5723d915046f117 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9efea4678e0e4dee8fe588222a3a6a67" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37931 } } peers { permanent_uuid: "8df3306d4bbe4fb1af0afb8d67c81798" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37673 } } peers { permanent_uuid: "1cfec8b74a2342cca5723d915046f117" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 43833 } }
I20240502 21:25:26.557448 16904 tablet_bootstrap.cc:492] T 610bb5f6731047b3b86883445a9e05bb P 8df3306d4bbe4fb1af0afb8d67c81798: Bootstrap starting.
I20240502 21:25:26.558729 16903 ts_tablet_manager.cc:1398] T 05a7626e52ae43a6829feacaf1837ea5 P 1cfec8b74a2342cca5723d915046f117: Time spent starting tablet: real 0.016s user 0.009s sys 0.006s
I20240502 21:25:26.559092 16903 tablet_bootstrap.cc:492] T 1b9f6d5e769e48f394fe59a835c22fc0 P 1cfec8b74a2342cca5723d915046f117: Bootstrap starting.
I20240502 21:25:26.560983 16904 tablet_bootstrap.cc:654] T 610bb5f6731047b3b86883445a9e05bb P 8df3306d4bbe4fb1af0afb8d67c81798: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:26.561301 16903 tablet_bootstrap.cc:654] T 1b9f6d5e769e48f394fe59a835c22fc0 P 1cfec8b74a2342cca5723d915046f117: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:26.562191 16904 log.cc:824] T 610bb5f6731047b3b86883445a9e05bb P 8df3306d4bbe4fb1af0afb8d67c81798: Log is configured to *not* fsync() on all Append() calls
I20240502 21:25:26.565974 16903 tablet_bootstrap.cc:492] T 1b9f6d5e769e48f394fe59a835c22fc0 P 1cfec8b74a2342cca5723d915046f117: No bootstrap required, opened a new log
I20240502 21:25:26.566112 16903 ts_tablet_manager.cc:1367] T 1b9f6d5e769e48f394fe59a835c22fc0 P 1cfec8b74a2342cca5723d915046f117: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.003s
I20240502 21:25:26.567011 16903 raft_consensus.cc:348] T 1b9f6d5e769e48f394fe59a835c22fc0 P 1cfec8b74a2342cca5723d915046f117 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9efea4678e0e4dee8fe588222a3a6a67" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37931 } } peers { permanent_uuid: "1cfec8b74a2342cca5723d915046f117" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 43833 } } peers { permanent_uuid: "8df3306d4bbe4fb1af0afb8d67c81798" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37673 } }
I20240502 21:25:26.567215 16903 raft_consensus.cc:374] T 1b9f6d5e769e48f394fe59a835c22fc0 P 1cfec8b74a2342cca5723d915046f117 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:26.567309 16903 raft_consensus.cc:729] T 1b9f6d5e769e48f394fe59a835c22fc0 P 1cfec8b74a2342cca5723d915046f117 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cfec8b74a2342cca5723d915046f117, State: Initialized, Role: FOLLOWER
I20240502 21:25:26.567453 16905 tablet_bootstrap.cc:654] T 1b9f6d5e769e48f394fe59a835c22fc0 P 9efea4678e0e4dee8fe588222a3a6a67: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:26.567541 16903 consensus_queue.cc:260] T 1b9f6d5e769e48f394fe59a835c22fc0 P 1cfec8b74a2342cca5723d915046f117 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9efea4678e0e4dee8fe588222a3a6a67" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37931 } } peers { permanent_uuid: "1cfec8b74a2342cca5723d915046f117" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 43833 } } peers { permanent_uuid: "8df3306d4bbe4fb1af0afb8d67c81798" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37673 } }
I20240502 21:25:26.568022 16903 ts_tablet_manager.cc:1398] T 1b9f6d5e769e48f394fe59a835c22fc0 P 1cfec8b74a2342cca5723d915046f117: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 21:25:26.568290 16903 tablet_bootstrap.cc:492] T 2b1fbeb84317494595f7262fe7857ebd P 1cfec8b74a2342cca5723d915046f117: Bootstrap starting.
I20240502 21:25:26.568548 16904 tablet_bootstrap.cc:492] T 610bb5f6731047b3b86883445a9e05bb P 8df3306d4bbe4fb1af0afb8d67c81798: No bootstrap required, opened a new log
I20240502 21:25:26.568879 16904 ts_tablet_manager.cc:1367] T 610bb5f6731047b3b86883445a9e05bb P 8df3306d4bbe4fb1af0afb8d67c81798: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.004s
I20240502 21:25:26.569825 16903 tablet_bootstrap.cc:654] T 2b1fbeb84317494595f7262fe7857ebd P 1cfec8b74a2342cca5723d915046f117: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:26.572199 16905 log.cc:824] T 1b9f6d5e769e48f394fe59a835c22fc0 P 9efea4678e0e4dee8fe588222a3a6a67: Log is configured to *not* fsync() on all Append() calls
I20240502 21:25:26.574249 16903 tablet_bootstrap.cc:492] T 2b1fbeb84317494595f7262fe7857ebd P 1cfec8b74a2342cca5723d915046f117: No bootstrap required, opened a new log
I20240502 21:25:26.574398 16903 ts_tablet_manager.cc:1367] T 2b1fbeb84317494595f7262fe7857ebd P 1cfec8b74a2342cca5723d915046f117: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20240502 21:25:26.575052 16903 raft_consensus.cc:348] T 2b1fbeb84317494595f7262fe7857ebd P 1cfec8b74a2342cca5723d915046f117 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cfec8b74a2342cca5723d915046f117" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 43833 } } peers { permanent_uuid: "9efea4678e0e4dee8fe588222a3a6a67" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37931 } } peers { permanent_uuid: "8df3306d4bbe4fb1af0afb8d67c81798" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37673 } }
I20240502 21:25:26.575266 16903 raft_consensus.cc:374] T 2b1fbeb84317494595f7262fe7857ebd P 1cfec8b74a2342cca5723d915046f117 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:26.575342 16903 raft_consensus.cc:729] T 2b1fbeb84317494595f7262fe7857ebd P 1cfec8b74a2342cca5723d915046f117 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cfec8b74a2342cca5723d915046f117, State: Initialized, Role: FOLLOWER
I20240502 21:25:26.575366 16905 tablet_bootstrap.cc:492] T 1b9f6d5e769e48f394fe59a835c22fc0 P 9efea4678e0e4dee8fe588222a3a6a67: No bootstrap required, opened a new log
I20240502 21:25:26.575608 16905 ts_tablet_manager.cc:1367] T 1b9f6d5e769e48f394fe59a835c22fc0 P 9efea4678e0e4dee8fe588222a3a6a67: Time spent bootstrapping tablet: real 0.036s user 0.000s sys 0.008s
I20240502 21:25:26.575579 16903 consensus_queue.cc:260] T 2b1fbeb84317494595f7262fe7857ebd P 1cfec8b74a2342cca5723d915046f117 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cfec8b74a2342cca5723d915046f117" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 43833 } } peers { permanent_uuid: "9efea4678e0e4dee8fe588222a3a6a67" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37931 } } peers { permanent_uuid: "8df3306d4bbe4fb1af0afb8d67c81798" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37673 } }
I20240502 21:25:26.576088 16903 ts_tablet_manager.cc:1398] T 2b1fbeb84317494595f7262fe7857ebd P 1cfec8b74a2342cca5723d915046f117: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 21:25:26.576351 16903 tablet_bootstrap.cc:492] T 71d05c4f1e6f4f7186ed7b8e157b7d2f P 1cfec8b74a2342cca5723d915046f117: Bootstrap starting.
I20240502 21:25:26.579102 16904 raft_consensus.cc:348] T 610bb5f6731047b3b86883445a9e05bb P 8df3306d4bbe4fb1af0afb8d67c81798 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9efea4678e0e4dee8fe588222a3a6a67" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37931 } } peers { permanent_uuid: "1cfec8b74a2342cca5723d915046f117" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 43833 } } peers { permanent_uuid: "8df3306d4bbe4fb1af0afb8d67c81798" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37673 } }
I20240502 21:25:26.579512 16904 raft_consensus.cc:374] T 610bb5f6731047b3b86883445a9e05bb P 8df3306d4bbe4fb1af0afb8d67c81798 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:26.579612 16904 raft_consensus.cc:729] T 610bb5f6731047b3b86883445a9e05bb P 8df3306d4bbe4fb1af0afb8d67c81798 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8df3306d4bbe4fb1af0afb8d67c81798, State: Initialized, Role: FOLLOWER
I20240502 21:25:26.580307 16904 consensus_queue.cc:260] T 610bb5f6731047b3b86883445a9e05bb P 8df3306d4bbe4fb1af0afb8d67c81798 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9efea4678e0e4dee8fe588222a3a6a67" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37931 } } peers { permanent_uuid: "1cfec8b74a2342cca5723d915046f117" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 43833 } } peers { permanent_uuid: "8df3306d4bbe4fb1af0afb8d67c81798" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37673 } }
I20240502 21:25:26.581125 16903 tablet_bootstrap.cc:654] T 71d05c4f1e6f4f7186ed7b8e157b7d2f P 1cfec8b74a2342cca5723d915046f117: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:26.582847 16904 ts_tablet_manager.cc:1398] T 610bb5f6731047b3b86883445a9e05bb P 8df3306d4bbe4fb1af0afb8d67c81798: Time spent starting tablet: real 0.014s user 0.007s sys 0.003s
W20240502 21:25:26.582960 16885 tablet.cc:2369] T 610bb5f6731047b3b86883445a9e05bb P 8df3306d4bbe4fb1af0afb8d67c81798: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 21:25:26.583232 16904 tablet_bootstrap.cc:492] T cb358d933789476898f74949ffa81e49 P 8df3306d4bbe4fb1af0afb8d67c81798: Bootstrap starting.
I20240502 21:25:26.584235 16905 raft_consensus.cc:348] T 1b9f6d5e769e48f394fe59a835c22fc0 P 9efea4678e0e4dee8fe588222a3a6a67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9efea4678e0e4dee8fe588222a3a6a67" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37931 } } peers { permanent_uuid: "1cfec8b74a2342cca5723d915046f117" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 43833 } } peers { permanent_uuid: "8df3306d4bbe4fb1af0afb8d67c81798" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37673 } }
I20240502 21:25:26.584699 16905 raft_consensus.cc:374] T 1b9f6d5e769e48f394fe59a835c22fc0 P 9efea4678e0e4dee8fe588222a3a6a67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:26.584757 16904 tablet_bootstrap.cc:654] T cb358d933789476898f74949ffa81e49 P 8df3306d4bbe4fb1af0afb8d67c81798: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:26.584807 16905 raft_consensus.cc:729] T 1b9f6d5e769e48f394fe59a835c22fc0 P 9efea4678e0e4dee8fe588222a3a6a67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9efea4678e0e4dee8fe588222a3a6a67, State: Initialized, Role: FOLLOWER
I20240502 21:25:26.585462 16905 consensus_queue.cc:260] T 1b9f6d5e769e48f394fe59a835c22fc0 P 9efea4678e0e4dee8fe588222a3a6a67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9efea4678e0e4dee8fe588222a3a6a67" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37931 } } peers { permanent_uuid: "1cfec8b74a2342cca5723d915046f117" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 43833 } } peers { permanent_uuid: "8df3306d4bbe4fb1af0afb8d67c81798" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37673 } }
I20240502 21:25:26.589290 16904 tablet_bootstrap.cc:492] T cb358d933789476898f74949ffa81e49 P 8df3306d4bbe4fb1af0afb8d67c81798: No bootstrap required, opened a new log
I20240502 21:25:26.589390 16903 tablet_bootstrap.cc:492] T 71d05c4f1e6f4f7186ed7b8e157b7d2f P 1cfec8b74a2342cca5723d915046f117: No bootstrap required, opened a new log
I20240502 21:25:26.589419 16904 ts_tablet_manager.cc:1367] T cb358d933789476898f74949ffa81e49 P 8df3306d4bbe4fb1af0afb8d67c81798: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20240502 21:25:26.589599 16903 ts_tablet_manager.cc:1367] T 71d05c4f1e6f4f7186ed7b8e157b7d2f P 1cfec8b74a2342cca5723d915046f117: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.000s
I20240502 21:25:26.589627 16905 ts_tablet_manager.cc:1398] T 1b9f6d5e769e48f394fe59a835c22fc0 P 9efea4678e0e4dee8fe588222a3a6a67: Time spent starting tablet: real 0.014s user 0.011s sys 0.003s
I20240502 21:25:26.590251 16905 tablet_bootstrap.cc:492] T c5ea5b6dcf934533ba8df41e52711337 P 9efea4678e0e4dee8fe588222a3a6a67: Bootstrap starting.
I20240502 21:25:26.591130 16907 raft_consensus.cc:482] T 1b9f6d5e769e48f394fe59a835c22fc0 P 1cfec8b74a2342cca5723d915046f117 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:25:26.591356 16907 raft_consensus.cc:504] T 1b9f6d5e769e48f394fe59a835c22fc0 P 1cfec8b74a2342cca5723d915046f117 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9efea4678e0e4dee8fe588222a3a6a67" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37931 } } peers { permanent_uuid: "1cfec8b74a2342cca5723d915046f117" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 43833 } } peers { permanent_uuid: "8df3306d4bbe4fb1af0afb8d67c81798" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37673 } }
I20240502 21:25:26.591830 16905 tablet_bootstrap.cc:654] T c5ea5b6dcf934533ba8df41e52711337 P 9efea4678e0e4dee8fe588222a3a6a67: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:26.591938 16904 raft_consensus.cc:348] T cb358d933789476898f74949ffa81e49 P 8df3306d4bbe4fb1af0afb8d67c81798 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9efea4678e0e4dee8fe588222a3a6a67" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37931 } } peers { permanent_uuid: "1cfec8b74a2342cca5723d915046f117" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 43833 } } peers { permanent_uuid: "8df3306d4bbe4fb1af0afb8d67c81798" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37673 } }
I20240502 21:25:26.592152 16904 raft_consensus.cc:374] T cb358d933789476898f74949ffa81e49 P 8df3306d4bbe4fb1af0afb8d67c81798 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:26.592259 16904 raft_consensus.cc:729] T cb358d933789476898f74949ffa81e49 P 8df3306d4bbe4fb1af0afb8d67c81798 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8df3306d4bbe4fb1af0afb8d67c81798, State: Initialized, Role: FOLLOWER
I20240502 21:25:26.592501 16904 consensus_queue.cc:260] T cb358d933789476898f74949ffa81e49 P 8df3306d4bbe4fb1af0afb8d67c81798 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9efea4678e0e4dee8fe588222a3a6a67" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37931 } } peers { permanent_uuid: "1cfec8b74a2342cca5723d915046f117" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 43833 } } peers { permanent_uuid: "8df3306d4bbe4fb1af0afb8d67c81798" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37673 } }
I20240502 21:25:26.593137 16904 ts_tablet_manager.cc:1398] T cb358d933789476898f74949ffa81e49 P 8df3306d4bbe4fb1af0afb8d67c81798: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 21:25:26.593369 16904 tablet_bootstrap.cc:492] T 1173ce275c4f40f0a0a31729310fbefd P 8df3306d4bbe4fb1af0afb8d67c81798: Bootstrap starting.
I20240502 21:25:26.593365 16907 leader_election.cc:288] T 1b9f6d5e769e48f394fe59a835c22fc0 P 1cfec8b74a2342cca5723d915046f117 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9efea4678e0e4dee8fe588222a3a6a67 (127.15.255.129:37931), 8df3306d4bbe4fb1af0afb8d67c81798 (127.15.255.131:37673)
I20240502 21:25:26.594859 16904 tablet_bootstrap.cc:654] T 1173ce275c4f40f0a0a31729310fbefd P 8df3306d4bbe4fb1af0afb8d67c81798: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:26.595762 16903 raft_consensus.cc:348] T 71d05c4f1e6f4f7186ed7b8e157b7d2f P 1cfec8b74a2342cca5723d915046f117 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9efea4678e0e4dee8fe588222a3a6a67" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37931 } } peers { permanent_uuid: "1cfec8b74a2342cca5723d915046f117" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 43833 } } peers { permanent_uuid: "8df3306d4bbe4fb1af0afb8d67c81798" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37673 } }
I20240502 21:25:26.596096 16903 raft_consensus.cc:374] T 71d05c4f1e6f4f7186ed7b8e157b7d2f P 1cfec8b74a2342cca5723d915046f117 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:26.596253 16903 raft_consensus.cc:729] T 71d05c4f1e6f4f7186ed7b8e157b7d2f P 1cfec8b74a2342cca5723d915046f117 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cfec8b74a2342cca5723d915046f117, State: Initialized, Role: FOLLOWER
I20240502 21:25:26.596302 16916 raft_consensus.cc:482] T 1b9f6d5e769e48f394fe59a835c22fc0 P 9efea4678e0e4dee8fe588222a3a6a67 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:25:26.596601 16903 consensus_queue.cc:260] T 71d05c4f1e6f4f7186ed7b8e157b7d2f P 1cfec8b74a2342cca5723d915046f117 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9efea4678e0e4dee8fe588222a3a6a67" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37931 } } peers { permanent_uuid: "1cfec8b74a2342cca5723d915046f117" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 43833 } } peers { permanent_uuid: "8df3306d4bbe4fb1af0afb8d67c81798" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37673 } }
I20240502 21:25:26.596526 16916 raft_consensus.cc:504] T 1b9f6d5e769e48f394fe59a835c22fc0 P 9efea4678e0e4dee8fe588222a3a6a67 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9efea4678e0e4dee8fe588222a3a6a67" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37931 } } peers { permanent_uuid: "1cfec8b74a2342cca5723d915046f117" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 43833 } } peers { permanent_uuid: "8df3306d4bbe4fb1af0afb8d67c81798" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37673 } }
I20240502 21:25:26.597827 16903 ts_tablet_manager.cc:1398] T 71d05c4f1e6f4f7186ed7b8e157b7d2f P 1cfec8b74a2342cca5723d915046f117: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20240502 21:25:26.598243 16903 tablet_bootstrap.cc:492] T 610bb5f6731047b3b86883445a9e05bb P 1cfec8b74a2342cca5723d915046f117: Bootstrap starting.
I20240502 21:25:26.599892 16903 tablet_bootstrap.cc:654] T 610bb5f6731047b3b86883445a9e05bb P 1cfec8b74a2342cca5723d915046f117: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:26.610797 16916 leader_election.cc:288] T 1b9f6d5e769e48f394fe59a835c22fc0 P 9efea4678e0e4dee8fe588222a3a6a67 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cfec8b74a2342cca5723d915046f117 (127.15.255.130:43833), 8df3306d4bbe4fb1af0afb8d67c81798 (127.15.255.131:37673)
I20240502 21:25:26.634984 16838 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b9f6d5e769e48f394fe59a835c22fc0" candidate_uuid: "1cfec8b74a2342cca5723d915046f117" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8df3306d4bbe4fb1af0afb8d67c81798" is_pre_election: true
W20240502 21:25:26.636364 16636 leader_election.cc:341] T 1b9f6d5e769e48f394fe59a835c22fc0 P 1cfec8b74a2342cca5723d915046f117 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8df3306d4bbe4fb1af0afb8d67c81798 (127.15.255.131:37673): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:25:26.638833 16570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b9f6d5e769e48f394fe59a835c22fc0" candidate_uuid: "1cfec8b74a2342cca5723d915046f117" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9efea4678e0e4dee8fe588222a3a6a67" is_pre_election: true
I20240502 21:25:26.639271 16570 raft_consensus.cc:2454] T 1b9f6d5e769e48f394fe59a835c22fc0 P 9efea4678e0e4dee8fe588222a3a6a67 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cfec8b74a2342cca5723d915046f117 in term 0.
I20240502 21:25:26.639966 16635 leader_election.cc:302] T 1b9f6d5e769e48f394fe59a835c22fc0 P 1cfec8b74a2342cca5723d915046f117 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cfec8b74a2342cca5723d915046f117, 9efea4678e0e4dee8fe588222a3a6a67; no voters: 8df3306d4bbe4fb1af0afb8d67c81798
I20240502 21:25:26.640542 16907 raft_consensus.cc:2781] T 1b9f6d5e769e48f394fe59a835c22fc0 P 1cfec8b74a2342cca5723d915046f117 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:25:26.640688 16907 raft_consensus.cc:482] T 1b9f6d5e769e48f394fe59a835c22fc0 P 1cfec8b74a2342cca5723d915046f117 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:25:26.640797 16907 raft_consensus.cc:3037] T 1b9f6d5e769e48f394fe59a835c22fc0 P 1cfec8b74a2342cca5723d915046f117 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:26.644042 16907 raft_consensus.cc:504] T 1b9f6d5e769e48f394fe59a835c22fc0 P 1cfec8b74a2342cca5723d915046f117 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9efea4678e0e4dee8fe588222a3a6a67" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37931 } } peers { permanent_uuid: "1cfec8b74a2342cca5723d915046f117" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 43833 } } peers { permanent_uuid: "8df3306d4bbe4fb1af0afb8d67c81798" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37673 } }
I20240502 21:25:26.645448 16838 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b9f6d5e769e48f394fe59a835c22fc0" candidate_uuid: "1cfec8b74a2342cca5723d915046f117" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8df3306d4bbe4fb1af0afb8d67c81798"
W20240502 21:25:26.646075 16636 leader_election.cc:341] T 1b9f6d5e769e48f394fe59a835c22fc0 P 1cfec8b74a2342cca5723d915046f117 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8df3306d4bbe4fb1af0afb8d67c81798 (127.15.255.131:37673): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:25:26.646884 16838 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b9f6d5e769e48f394fe59a835c22fc0" candidate_uuid: "9efea4678e0e4dee8fe588222a3a6a67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8df3306d4bbe4fb1af0afb8d67c81798" is_pre_election: true
W20240502 21:25:26.647629 16504 leader_election.cc:341] T 1b9f6d5e769e48f394fe59a835c22fc0 P 9efea4678e0e4dee8fe588222a3a6a67 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8df3306d4bbe4fb1af0afb8d67c81798 (127.15.255.131:37673): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:25:26.647907 16907 leader_election.cc:288] T 1b9f6d5e769e48f394fe59a835c22fc0 P 1cfec8b74a2342cca5723d915046f117 [CANDIDATE]: Term 1 election: Requested vote from peers 9efea4678e0e4dee8fe588222a3a6a67 (127.15.255.129:37931), 8df3306d4bbe4fb1af0afb8d67c81798 (127.15.255.131:37673)
I20240502 21:25:26.648552 16702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b9f6d5e769e48f394fe59a835c22fc0" candidate_uuid: "9efea4678e0e4dee8fe588222a3a6a67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cfec8b74a2342cca5723d915046f117" is_pre_election: true
I20240502 21:25:26.649053 16702 raft_consensus.cc:2379] T 1b9f6d5e769e48f394fe59a835c22fc0 P 1cfec8b74a2342cca5723d915046f117 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9efea4678e0e4dee8fe588222a3a6a67 in current term 1: Already voted for candidate 1cfec8b74a2342cca5723d915046f117 in this term.
I20240502 21:25:26.650329 16506 leader_election.cc:302] T 1b9f6d5e769e48f394fe59a835c22fc0 P 9efea4678e0e4dee8fe588222a3a6a67 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9efea4678e0e4dee8fe588222a3a6a67; no voters: 1cfec8b74a2342cca5723d915046f117, 8df3306d4bbe4fb1af0afb8d67c81798
I20240502 21:25:26.650475 16905 tablet_bootstrap.cc:492] T c5ea5b6dcf934533ba8df41e52711337 P 9efea4678e0e4dee8fe588222a3a6a67: No bootstrap required, opened a new log
I20240502 21:25:26.650641 16905 ts_tablet_manager.cc:1367] T c5ea5b6dcf934533ba8df41e52711337 P 9efea4678e0e4dee8fe588222a3a6a67: Time spent bootstrapping tablet: real 0.060s user 0.002s sys 0.007s
I20240502 21:25:26.650836 16916 raft_consensus.cc:3037] T 1b9f6d5e769e48f394fe59a835c22fc0 P 9efea4678e0e4dee8fe588222a3a6a67 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:26.651800 16905 raft_consensus.cc:348] T c5ea5b6dcf934533ba8df41e52711337 P 9efea4678e0e4dee8fe588222a3a6a67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cfec8b74a2342cca5723d915046f117" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 43833 } } peers { permanent_uuid: "9efea4678e0e4dee8fe588222a3a6a67" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37931 } } peers { permanent_uuid: "8df3306d4bbe4fb1af0afb8d67c81798" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37673 } }
I20240502 21:25:26.652025 16905 raft_consensus.cc:374] T c5ea5b6dcf934533ba8df41e52711337 P 9efea4678e0e4dee8fe588222a3a6a67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:26.652122 16905 raft_consensus.cc:729] T c5ea5b6dcf934533ba8df41e52711337 P 9efea4678e0e4dee8fe588222a3a6a67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9efea4678e0e4dee8fe588222a3a6a67, State: Initialized, Role: FOLLOWER
I20240502 21:25:26.652477 16905 consensus_queue.cc:260] T c5ea5b6dcf934533ba8df41e52711337 P 9efea4678e0e4dee8fe588222a3a6a67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cfec8b74a2342cca5723d915046f117" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 43833 } } peers { permanent_uuid: "9efea4678e0e4dee8fe588222a3a6a67" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37931 } } peers { permanent_uuid: "8df3306d4bbe4fb1af0afb8d67c81798" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37673 } }
I20240502 21:25:26.653940 16916 raft_consensus.cc:2726] T 1b9f6d5e769e48f394fe59a835c22fc0 P 9efea4678e0e4dee8fe588222a3a6a67 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 21:25:26.653997 16903 tablet_bootstrap.cc:492] T 610bb5f6731047b3b86883445a9e05bb P 1cfec8b74a2342cca5723d915046f117: No bootstrap required, opened a new log
I20240502 21:25:26.654145 16903 ts_tablet_manager.cc:1367] T 610bb5f6731047b3b86883445a9e05bb P 1cfec8b74a2342cca5723d915046f117: Time spent bootstrapping tablet: real 0.056s user 0.017s sys 0.037s
I20240502 21:25:26.654364 16905 ts_tablet_manager.cc:1398] T c5ea5b6dcf934533ba8df41e52711337 P 9efea4678e0e4dee8fe588222a3a6a67: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20240502 21:25:26.654640 16905 tablet_bootstrap.cc:492] T 38999367825249e1a0f5f156bd3c3b25 P 9efea4678e0e4dee8fe588222a3a6a67: Bootstrap starting.
I20240502 21:25:26.654902 16903 raft_consensus.cc:348] T 610bb5f6731047b3b86883445a9e05bb P 1cfec8b74a2342cca5723d915046f117 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9efea4678e0e4dee8fe588222a3a6a67" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37931 } } peers { permanent_uuid: "1cfec8b74a2342cca5723d915046f117" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 43833 } } peers { permanent_uuid: "8df3306d4bbe4fb1af0afb8d67c81798" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37673 } }
I20240502 21:25:26.655071 16904 tablet_bootstrap.cc:492] T 1173ce275c4f40f0a0a31729310fbefd P 8df3306d4bbe4fb1af0afb8d67c81798: No bootstrap required, opened a new log
I20240502 21:25:26.655107 16903 raft_consensus.cc:374] T 610bb5f6731047b3b86883445a9e05bb P 1cfec8b74a2342cca5723d915046f117 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:26.655179 16903 raft_consensus.cc:729] T 610bb5f6731047b3b86883445a9e05bb P 1cfec8b74a2342cca5723d915046f117 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cfec8b74a2342cca5723d915046f117, State: Initialized, Role: FOLLOWER
I20240502 21:25:26.655221 16904 ts_tablet_manager.cc:1367] T 1173ce275c4f40f0a0a31729310fbefd P 8df3306d4bbe4fb1af0afb8d67c81798: Time spent bootstrapping tablet: real 0.062s user 0.022s sys 0.036s
I20240502 21:25:26.655432 16903 consensus_queue.cc:260] T 610bb5f6731047b3b86883445a9e05bb P 1cfec8b74a2342cca5723d915046f117 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9efea4678e0e4dee8fe588222a3a6a67" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37931 } } peers { permanent_uuid: "1cfec8b74a2342cca5723d915046f117" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 43833 } } peers { permanent_uuid: "8df3306d4bbe4fb1af0afb8d67c81798" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37673 } }
I20240502 21:25:26.656018 16903 ts_tablet_manager.cc:1398] T 610bb5f6731047b3b86883445a9e05bb P 1cfec8b74a2342cca5723d915046f117: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 21:25:26.656242 16905 tablet_bootstrap.cc:654] T 38999367825249e1a0f5f156bd3c3b25 P 9efea4678e0e4dee8fe588222a3a6a67: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:26.656320 16903 tablet_bootstrap.cc:492] T 61268934b6b74fad90db0a52dd48ac82 P 1cfec8b74a2342cca5723d915046f117: Bootstrap starting.
I20240502 21:25:26.656803 16904 raft_consensus.cc:348] T 1173ce275c4f40f0a0a31729310fbefd P 8df3306d4bbe4fb1af0afb8d67c81798 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9efea4678e0e4dee8fe588222a3a6a67" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37931 } } peers { permanent_uuid: "1cfec8b74a2342cca5723d915046f117" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 43833 } } peers { permanent_uuid: "8df3306d4bbe4fb1af0afb8d67c81798" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37673 } }
I20240502 21:25:26.657023 16904 raft_consensus.cc:374] T 1173ce275c4f40f0a0a31729310fbefd P 8df3306d4bbe4fb1af0afb8d67c81798 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:26.657090 16904 raft_consensus.cc:729] T 1173ce275c4f40f0a0a31729310fbefd P 8df3306d4bbe4fb1af0afb8d67c81798 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8df3306d4bbe4fb1af0afb8d67c81798, State: Initialized, Role: FOLLOWER
I20240502 21:25:26.657402 16904 consensus_queue.cc:260] T 1173ce275c4f40f0a0a31729310fbefd P 8df3306d4bbe4fb1af0afb8d67c81798 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9efea4678e0e4dee8fe588222a3a6a67" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37931 } } peers { permanent_uuid: "1cfec8b74a2342cca5723d915046f117" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 43833 } } peers { permanent_uuid: "8df3306d4bbe4fb1af0afb8d67c81798" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37673 } }
I20240502 21:25:26.657996 16903 tablet_bootstrap.cc:654] T 61268934b6b74fad90db0a52dd48ac82 P 1cfec8b74a2342cca5723d915046f117: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:26.659050 16904 ts_tablet_manager.cc:1398] T 1173ce275c4f40f0a0a31729310fbefd P 8df3306d4bbe4fb1af0afb8d67c81798: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240502 21:25:26.659353 16904 tablet_bootstrap.cc:492] T c5ea5b6dcf934533ba8df41e52711337 P 8df3306d4bbe4fb1af0afb8d67c81798: Bootstrap starting.
I20240502 21:25:26.660925 16904 tablet_bootstrap.cc:654] T c5ea5b6dcf934533ba8df41e52711337 P 8df3306d4bbe4fb1af0afb8d67c81798: Neither blocks nor log segments found. Creating new log.
W20240502 21:25:26.665257 16750 tablet.cc:2369] T 05a7626e52ae43a6829feacaf1837ea5 P 1cfec8b74a2342cca5723d915046f117: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 21:25:26.665772 16570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b9f6d5e769e48f394fe59a835c22fc0" candidate_uuid: "1cfec8b74a2342cca5723d915046f117" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9efea4678e0e4dee8fe588222a3a6a67"
I20240502 21:25:26.666047 16903 tablet_bootstrap.cc:492] T 61268934b6b74fad90db0a52dd48ac82 P 1cfec8b74a2342cca5723d915046f117: No bootstrap required, opened a new log
I20240502 21:25:26.666188 16903 ts_tablet_manager.cc:1367] T 61268934b6b74fad90db0a52dd48ac82 P 1cfec8b74a2342cca5723d915046f117: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20240502 21:25:26.666836 16903 raft_consensus.cc:348] T 61268934b6b74fad90db0a52dd48ac82 P 1cfec8b74a2342cca5723d915046f117 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cfec8b74a2342cca5723d915046f117" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 43833 } } peers { permanent_uuid: "9efea4678e0e4dee8fe588222a3a6a67" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37931 } } peers { permanent_uuid: "8df3306d4bbe4fb1af0afb8d67c81798" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37673 } }
I20240502 21:25:26.667016 16903 raft_consensus.cc:374] T 61268934b6b74fad90db0a52dd48ac82 P 1cfec8b74a2342cca5723d915046f117 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:26.667093 16903 raft_consensus.cc:729] T 61268934b6b74fad90db0a52dd48ac82 P 1cfec8b74a2342cca5723d915046f117 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cfec8b74a2342cca5723d915046f117, State: Initialized, Role: FOLLOWER
I20240502 21:25:26.667325 16903 consensus_queue.cc:260] T 61268934b6b74fad90db0a52dd48ac82 P 1cfec8b74a2342cca5723d915046f117 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cfec8b74a2342cca5723d915046f117" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 43833 } } peers { permanent_uuid: "9efea4678e0e4dee8fe588222a3a6a67" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37931 } } peers { permanent_uuid: "8df3306d4bbe4fb1af0afb8d67c81798" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37673 } }
I20240502 21:25:26.667933 16903 ts_tablet_manager.cc:1398] T 61268934b6b74fad90db0a52dd48ac82 P 1cfec8b74a2342cca5723d915046f117: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:25:26.668983 16570 raft_consensus.cc:2454] T 1b9f6d5e769e48f394fe59a835c22fc0 P 9efea4678e0e4dee8fe588222a3a6a67 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cfec8b74a2342cca5723d915046f117 in term 1.
I20240502 21:25:26.669639 16903 tablet_bootstrap.cc:492] T ed40aed15ca3447aa01283771ac257ed P 1cfec8b74a2342cca5723d915046f117: Bootstrap starting.
I20240502 21:25:26.669802 16635 leader_election.cc:302] T 1b9f6d5e769e48f394fe59a835c22fc0 P 1cfec8b74a2342cca5723d915046f117 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cfec8b74a2342cca5723d915046f117, 9efea4678e0e4dee8fe588222a3a6a67; no voters: 8df3306d4bbe4fb1af0afb8d67c81798
I20240502 21:25:26.670114 16907 raft_consensus.cc:2781] T 1b9f6d5e769e48f394fe59a835c22fc0 P 1cfec8b74a2342cca5723d915046f117 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:25:26.673177 16903 tablet_bootstrap.cc:654] T ed40aed15ca3447aa01283771ac257ed P 1cfec8b74a2342cca5723d915046f117: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:26.675453 16904 tablet_bootstrap.cc:492] T c5ea5b6dcf934533ba8df41e52711337 P 8df3306d4bbe4fb1af0afb8d67c81798: No bootstrap required, opened a new log
I20240502 21:25:26.675601 16904 ts_tablet_manager.cc:1367] T c5ea5b6dcf934533ba8df41e52711337 P 8df3306d4bbe4fb1af0afb8d67c81798: Time spent bootstrapping tablet: real 0.016s user 0.013s sys 0.000s
I20240502 21:25:26.676376 16904 raft_consensus.cc:348] T c5ea5b6dcf934533ba8df41e52711337 P 8df3306d4bbe4fb1af0afb8d67c81798 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cfec8b74a2342cca5723d915046f117" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 43833 } } peers { permanent_uuid: "9efea4678e0e4dee8fe588222a3a6a67" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37931 } } peers { permanent_uuid: "8df3306d4bbe4fb1af0afb8d67c81798" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37673 } }
I20240502 21:25:26.676599 16904 raft_consensus.cc:374] T c5ea5b6dcf934533ba8df41e52711337 P 8df3306d4bbe4fb1af0afb8d67c81798 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:26.676687 16904 raft_consensus.cc:729] T c5ea5b6dcf934533ba8df41e52711337 P 8df3306d4bbe4fb1af0afb8d67c81798 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8df3306d4bbe4fb1af0afb8d67c81798, State: Initialized, Role: FOLLOWER
I20240502 21:25:26.676963 16904 consensus_queue.cc:260] T c5ea5b6dcf934533ba8df41e52711337 P 8df3306d4bbe4fb1af0afb8d67c81798 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cfec8b74a2342cca5723d915046f117" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 43833 } } peers { permanent_uuid: "9efea4678e0e4dee8fe588222a3a6a67" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37931 } } peers { permanent_uuid: "8df3306d4bbe4fb1af0afb8d67c81798" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37673 } }
I20240502 21:25:26.677594 16904 ts_tablet_manager.cc:1398] T c5ea5b6dcf934533ba8df41e52711337 P 8df3306d4bbe4fb1af0afb8d67c81798: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:25:26.677906 16904 tablet_bootstrap.cc:492] T 62a4fb5953dc4fa7badc5e2ee1ed4261 P 8df3306d4bbe4fb1af0afb8d67c81798: Bootstrap starting.
I20240502 21:25:26.679477 16904 tablet_bootstrap.cc:654] T 62a4fb5953dc4fa7badc5e2ee1ed4261 P 8df3306d4bbe4fb1af0afb8d67c81798: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:26.685094 16907 raft_consensus.cc:686] T 1b9f6d5e769e48f394fe59a835c22fc0 P 1cfec8b74a2342cca5723d915046f117 [term 1 LEADER]: Becoming Leader. State: Replica: 1cfec8b74a2342cca5723d915046f117, State: Running, Role: LEADER
I20240502 21:25:26.685916 16907 consensus_queue.cc:237] T 1b9f6d5e769e48f394fe59a835c22fc0 P 1cfec8b74a2342cca5723d915046f117 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9efea4678e0e4dee8fe588222a3a6a67" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37931 } } peers { permanent_uuid: "1cfec8b74a2342cca5723d915046f117" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 43833 } } peers { permanent_uuid: "8df3306d4bbe4fb1af0afb8d67c81798" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37673 } }
I20240502 21:25:26.687323 16904 tablet_bootstrap.cc:492] T 62a4fb5953dc4fa7badc5e2ee1ed4261 P 8df3306d4bbe4fb1af0afb8d67c81798: No bootstrap required, opened a new log
I20240502 21:25:26.687475 16904 ts_tablet_manager.cc:1367] T 62a4fb5953dc4fa7badc5e2ee1ed4261 P 8df3306d4bbe4fb1af0afb8d67c81798: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.002s
I20240502 21:25:26.688117 16904 raft_consensus.cc:348] T 62a4fb5953dc4fa7badc5e2ee1ed4261 P 8df3306d4bbe4fb1af0afb8d67c81798 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9efea4678e0e4dee8fe588222a3a6a67" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37931 } } peers { permanent_uuid: "1cfec8b74a2342cca5723d915046f117" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 43833 } } peers { permanent_uuid: "8df3306d4bbe4fb1af0afb8d67c81798" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37673 } }
I20240502 21:25:26.688315 16904 raft_consensus.cc:374] T 62a4fb5953dc4fa7badc5e2ee1ed4261 P 8df3306d4bbe4fb1af0afb8d67c81798 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:26.688405 16904 raft_consensus.cc:729] T 62a4fb5953dc4fa7badc5e2ee1ed4261 P 8df3306d4bbe4fb1af0afb8d67c81798 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8df3306d4bbe4fb1af0afb8d67c81798, State: Initialized, Role: FOLLOWER
I20240502 21:25:26.688699 16904 consensus_queue.cc:260] T 62a4fb5953dc4fa7badc5e2ee1ed4261 P 8df3306d4bbe4fb1af0afb8d67c81798 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9efea4678e0e4dee8fe588222a3a6a67" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37931 } } peers { permanent_uuid: "1cfec8b74a2342cca5723d915046f117" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 43833 } } peers { permanent_uuid: "8df3306d4bbe4fb1af0afb8d67c81798" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37673 } }
I20240502 21:25:26.689656 16904 ts_tablet_manager.cc:1398] T 62a4fb5953dc4fa7badc5e2ee1ed4261 P 8df3306d4bbe4fb1af0afb8d67c81798: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 21:25:26.689707 16903 tablet_bootstrap.cc:492] T ed40aed15ca3447aa01283771ac257ed P 1cfec8b74a2342cca5723d915046f117: No bootstrap required, opened a new log
I20240502 21:25:26.689846 16903 ts_tablet_manager.cc:1367] T ed40aed15ca3447aa01283771ac257ed P 1cfec8b74a2342cca5723d915046f117: Time spent bootstrapping tablet: real 0.020s user 0.003s sys 0.014s
I20240502 21:25:26.689878 16905 tablet_bootstrap.cc:492] T 38999367825249e1a0f5f156bd3c3b25 P 9efea4678e0e4dee8fe588222a3a6a67: No bootstrap required, opened a new log
I20240502 21:25:26.689922 16904 tablet_bootstrap.cc:492] T 61268934b6b74fad90db0a52dd48ac82 P 8df3306d4bbe4fb1af0afb8d67c81798: Bootstrap starting.
I20240502 21:25:26.690019 16905 ts_tablet_manager.cc:1367] T 38999367825249e1a0f5f156bd3c3b25 P 9efea4678e0e4dee8fe588222a3a6a67: Time spent bootstrapping tablet: real 0.035s user 0.013s sys 0.020s
I20240502 21:25:26.690554 16903 raft_consensus.cc:348] T ed40aed15ca3447aa01283771ac257ed P 1cfec8b74a2342cca5723d915046f117 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9efea4678e0e4dee8fe588222a3a6a67" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37931 } } peers { permanent_uuid: "1cfec8b74a2342cca5723d915046f117" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 43833 } } peers { permanent_uuid: "8df3306d4bbe4fb1af0afb8d67c81798" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37673 } }
I20240502 21:25:26.690759 16903 raft_consensus.cc:374] T ed40aed15ca3447aa01283771ac257ed P 1cfec8b74a2342cca5723d915046f117 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:26.690668 16905 raft_consensus.cc:348] T 38999367825249e1a0f5f156bd3c3b25 P 9efea4678e0e4dee8fe588222a3a6a67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9efea4678e0e4dee8fe588222a3a6a67" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37931 } } peers { permanent_uuid: "1cfec8b74a2342cca5723d915046f117" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 43833 } } peers { permanent_uuid: "8df3306d4bbe4fb1af0afb8d67c81798" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37673 } }
I20240502 21:25:26.690831 16903 raft_consensus.cc:729] T ed40aed15ca3447aa01283771ac257ed P 1cfec8b74a2342cca5723d915046f117 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cfec8b74a2342cca5723d915046f117, State: Initialized, Role: FOLLOWER
I20240502 21:25:26.690884 16905 raft_consensus.cc:374] T 38999367825249e1a0f5f156bd3c3b25 P 9efea4678e0e4dee8fe588222a3a6a67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:26.690945 16905 raft_consensus.cc:729] T 38999367825249e1a0f5f156bd3c3b25 P 9efea4678e0e4dee8fe588222a3a6a67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9efea4678e0e4dee8fe588222a3a6a67, State: Initialized, Role: FOLLOWER
I20240502 21:25:26.691066 16903 consensus_queue.cc:260] T ed40aed15ca3447aa01283771ac257ed P 1cfec8b74a2342cca5723d915046f117 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9efea4678e0e4dee8fe588222a3a6a67" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37931 } } peers { permanent_uuid: "1cfec8b74a2342cca5723d915046f117" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 43833 } } peers { permanent_uuid: "8df3306d4bbe4fb1af0afb8d67c81798" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37673 } }
I20240502 21:25:26.691201 16905 consensus_queue.cc:260] T 38999367825249e1a0f5f156bd3c3b25 P 9efea4678e0e4dee8fe588222a3a6a67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9efea4678e0e4dee8fe588222a3a6a67" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37931 } } peers { permanent_uuid: "1cfec8b74a2342cca5723d915046f117" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 43833 } } peers { permanent_uuid: "8df3306d4bbe4fb1af0afb8d67c81798" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37673 } }
I20240502 21:25:26.691562 16903 ts_tablet_manager.cc:1398] T ed40aed15ca3447aa01283771ac257ed P 1cfec8b74a2342cca5723d915046f117: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 21:25:26.691838 16903 tablet_bootstrap.cc:492] T 4cfc1f2146dd449c87a46196e25280c2 P 1cfec8b74a2342cca5723d915046f117: Bootstrap starting.
I20240502 21:25:26.691848 16905 ts_tablet_manager.cc:1398] T 38999367825249e1a0f5f156bd3c3b25 P 9efea4678e0e4dee8fe588222a3a6a67: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 21:25:26.692116 16905 tablet_bootstrap.cc:492] T 05a7626e52ae43a6829feacaf1837ea5 P 9efea4678e0e4dee8fe588222a3a6a67: Bootstrap starting.
I20240502 21:25:26.692209 16904 tablet_bootstrap.cc:654] T 61268934b6b74fad90db0a52dd48ac82 P 8df3306d4bbe4fb1af0afb8d67c81798: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:26.693423 16903 tablet_bootstrap.cc:654] T 4cfc1f2146dd449c87a46196e25280c2 P 1cfec8b74a2342cca5723d915046f117: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:26.693801 16905 tablet_bootstrap.cc:654] T 05a7626e52ae43a6829feacaf1837ea5 P 9efea4678e0e4dee8fe588222a3a6a67: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:26.698232 16904 tablet_bootstrap.cc:492] T 61268934b6b74fad90db0a52dd48ac82 P 8df3306d4bbe4fb1af0afb8d67c81798: No bootstrap required, opened a new log
I20240502 21:25:26.698369 16904 ts_tablet_manager.cc:1367] T 61268934b6b74fad90db0a52dd48ac82 P 8df3306d4bbe4fb1af0afb8d67c81798: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.005s
I20240502 21:25:26.698962 16903 tablet_bootstrap.cc:492] T 4cfc1f2146dd449c87a46196e25280c2 P 1cfec8b74a2342cca5723d915046f117: No bootstrap required, opened a new log
I20240502 21:25:26.699198 16904 raft_consensus.cc:348] T 61268934b6b74fad90db0a52dd48ac82 P 8df3306d4bbe4fb1af0afb8d67c81798 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cfec8b74a2342cca5723d915046f117" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 43833 } } peers { permanent_uuid: "9efea4678e0e4dee8fe588222a3a6a67" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37931 } } peers { permanent_uuid: "8df3306d4bbe4fb1af0afb8d67c81798" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37673 } }
I20240502 21:25:26.699388 16904 raft_consensus.cc:374] T 61268934b6b74fad90db0a52dd48ac82 P 8df3306d4bbe4fb1af0afb8d67c81798 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:26.699477 16904 raft_consensus.cc:729] T 61268934b6b74fad90db0a52dd48ac82 P 8df3306d4bbe4fb1af0afb8d67c81798 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8df3306d4bbe4fb1af0afb8d67c81798, State: Initialized, Role: FOLLOWER
I20240502 21:25:26.699748 16903 ts_tablet_manager.cc:1367] T 4cfc1f2146dd449c87a46196e25280c2 P 1cfec8b74a2342cca5723d915046f117: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20240502 21:25:26.699710 16904 consensus_queue.cc:260] T 61268934b6b74fad90db0a52dd48ac82 P 8df3306d4bbe4fb1af0afb8d67c81798 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cfec8b74a2342cca5723d915046f117" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 43833 } } peers { permanent_uuid: "9efea4678e0e4dee8fe588222a3a6a67" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37931 } } peers { permanent_uuid: "8df3306d4bbe4fb1af0afb8d67c81798" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37673 } }
I20240502 21:25:26.700278 16905 tablet_bootstrap.cc:492] T 05a7626e52ae43a6829feacaf1837ea5 P 9efea4678e0e4dee8fe588222a3a6a67: No bootstrap required, opened a new log
I20240502 21:25:26.700470 16905 ts_tablet_manager.cc:1367] T 05a7626e52ae43a6829feacaf1837ea5 P 9efea4678e0e4dee8fe588222a3a6a67: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20240502 21:25:26.700464 16903 raft_consensus.cc:348] T 4cfc1f2146dd449c87a46196e25280c2 P 1cfec8b74a2342cca5723d915046f117 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cfec8b74a2342cca5723d915046f117" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 43833 } } peers { permanent_uuid: "9efea4678e0e4dee8fe588222a3a6a67" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37931 } } peers { permanent_uuid: "8df3306d4bbe4fb1af0afb8d67c81798" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37673 } }
I20240502 21:25:26.700656 16903 raft_consensus.cc:374] T 4cfc1f2146dd449c87a46196e25280c2 P 1cfec8b74a2342cca5723d915046f117 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:26.700768 16903 raft_consensus.cc:729] T 4cfc1f2146dd449c87a46196e25280c2 P 1cfec8b74a2342cca5723d915046f117 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cfec8b74a2342cca5723d915046f117, State: Initialized, Role: FOLLOWER
I20240502 21:25:26.700083 16425 catalog_manager.cc:5484] T 1b9f6d5e769e48f394fe59a835c22fc0 P 1cfec8b74a2342cca5723d915046f117 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1cfec8b74a2342cca5723d915046f117 (127.15.255.130). New cstate: current_term: 1 leader_uuid: "1cfec8b74a2342cca5723d915046f117" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9efea4678e0e4dee8fe588222a3a6a67" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37931 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1cfec8b74a2342cca5723d915046f117" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 43833 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8df3306d4bbe4fb1af0afb8d67c81798" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37673 } health_report { overall_health: UNKNOWN } } }
I20240502 21:25:26.700974 16903 consensus_queue.cc:260] T 4cfc1f2146dd449c87a46196e25280c2 P 1cfec8b74a2342cca5723d915046f117 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cfec8b74a2342cca5723d915046f117" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 43833 } } peers { permanent_uuid: "9efea4678e0e4dee8fe588222a3a6a67" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37931 } } peers { permanent_uuid: "8df3306d4bbe4fb1af0afb8d67c81798" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37673 } }
I20240502 21:25:26.701301 16905 raft_consensus.cc:348] T 05a7626e52ae43a6829feacaf1837ea5 P 9efea4678e0e4dee8fe588222a3a6a67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9efea4678e0e4dee8fe588222a3a6a67" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37931 } } peers { permanent_uuid: "8df3306d4bbe4fb1af0afb8d67c81798" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37673 } } peers { permanent_uuid: "1cfec8b74a2342cca5723d915046f117" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 43833 } }
I20240502 21:25:26.701479 16905 raft_consensus.cc:374] T 05a7626e52ae43a6829feacaf1837ea5 P 9efea4678e0e4dee8fe588222a3a6a67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:26.701515 16904 ts_tablet_manager.cc:1398] T 61268934b6b74fad90db0a52dd48ac82 P 8df3306d4bbe4fb1af0afb8d67c81798: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240502 21:25:26.701524 16903 ts_tablet_manager.cc:1398] T 4cfc1f2146dd449c87a46196e25280c2 P 1cfec8b74a2342cca5723d915046f117: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 21:25:26.701565 16905 raft_consensus.cc:729] T 05a7626e52ae43a6829feacaf1837ea5 P 9efea4678e0e4dee8fe588222a3a6a67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9efea4678e0e4dee8fe588222a3a6a67, State: Initialized, Role: FOLLOWER
I20240502 21:25:26.701756 16904 tablet_bootstrap.cc:492] T e882da2f771249f8a3626d1a7a35de34 P 8df3306d4bbe4fb1af0afb8d67c81798: Bootstrap starting.
I20240502 21:25:26.701774 16903 tablet_bootstrap.cc:492] T 1173ce275c4f40f0a0a31729310fbefd P 1cfec8b74a2342cca5723d915046f117: Bootstrap starting.
I20240502 21:25:26.701787 16905 consensus_queue.cc:260] T 05a7626e52ae43a6829feacaf1837ea5 P 9efea4678e0e4dee8fe588222a3a6a67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9efea4678e0e4dee8fe588222a3a6a67" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37931 } } peers { permanent_uuid: "8df3306d4bbe4fb1af0afb8d67c81798" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37673 } } peers { permanent_uuid: "1cfec8b74a2342cca5723d915046f117" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 43833 } }
I20240502 21:25:26.702374 16905 ts_tablet_manager.cc:1398] T 05a7626e52ae43a6829feacaf1837ea5 P 9efea4678e0e4dee8fe588222a3a6a67: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 21:25:26.703809 16903 tablet_bootstrap.cc:654] T 1173ce275c4f40f0a0a31729310fbefd P 1cfec8b74a2342cca5723d915046f117: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:26.704483 16904 tablet_bootstrap.cc:654] T e882da2f771249f8a3626d1a7a35de34 P 8df3306d4bbe4fb1af0afb8d67c81798: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:26.704453 16905 tablet_bootstrap.cc:492] T 610bb5f6731047b3b86883445a9e05bb P 9efea4678e0e4dee8fe588222a3a6a67: Bootstrap starting.
I20240502 21:25:26.707770 16903 tablet_bootstrap.cc:492] T 1173ce275c4f40f0a0a31729310fbefd P 1cfec8b74a2342cca5723d915046f117: No bootstrap required, opened a new log
I20240502 21:25:26.707876 16905 tablet_bootstrap.cc:654] T 610bb5f6731047b3b86883445a9e05bb P 9efea4678e0e4dee8fe588222a3a6a67: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:26.707924 16903 ts_tablet_manager.cc:1367] T 1173ce275c4f40f0a0a31729310fbefd P 1cfec8b74a2342cca5723d915046f117: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.003s
I20240502 21:25:26.708163 16904 tablet_bootstrap.cc:492] T e882da2f771249f8a3626d1a7a35de34 P 8df3306d4bbe4fb1af0afb8d67c81798: No bootstrap required, opened a new log
I20240502 21:25:26.708309 16904 ts_tablet_manager.cc:1367] T e882da2f771249f8a3626d1a7a35de34 P 8df3306d4bbe4fb1af0afb8d67c81798: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.003s
I20240502 21:25:26.708624 16903 raft_consensus.cc:348] T 1173ce275c4f40f0a0a31729310fbefd P 1cfec8b74a2342cca5723d915046f117 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9efea4678e0e4dee8fe588222a3a6a67" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37931 } } peers { permanent_uuid: "1cfec8b74a2342cca5723d915046f117" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 43833 } } peers { permanent_uuid: "8df3306d4bbe4fb1af0afb8d67c81798" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37673 } }
I20240502 21:25:26.708853 16903 raft_consensus.cc:374] T 1173ce275c4f40f0a0a31729310fbefd P 1cfec8b74a2342cca5723d915046f117 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:26.708933 16903 raft_consensus.cc:729] T 1173ce275c4f40f0a0a31729310fbefd P 1cfec8b74a2342cca5723d915046f117 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cfec8b74a2342cca5723d915046f117, State: Initialized, Role: FOLLOWER
I20240502 21:25:26.709014 16904 raft_consensus.cc:348] T e882da2f771249f8a3626d1a7a35de34 P 8df3306d4bbe4fb1af0afb8d67c81798 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cfec8b74a2342cca5723d915046f117" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 43833 } } peers { permanent_uuid: "9efea4678e0e4dee8fe588222a3a6a67" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37931 } } peers { permanent_uuid: "8df3306d4bbe4fb1af0afb8d67c81798" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37673 } }
I20240502 21:25:26.709213 16904 raft_consensus.cc:374] T e882da2f771249f8a3626d1a7a35de34 P 8df3306d4bbe4fb1af0afb8d67c81798 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:26.709164 16903 consensus_queue.cc:260] T 1173ce275c4f40f0a0a31729310fbefd P 1cfec8b74a2342cca5723d915046f117 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9efea4678e0e4dee8fe588222a3a6a67" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37931 } } peers { permanent_uuid: "1cfec8b74a2342cca5723d915046f117" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 43833 } } peers { permanent_uuid: "8df3306d4bbe4fb1af0afb8d67c81798" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37673 } }
I20240502 21:25:26.709283 16904 raft_consensus.cc:729] T e882da2f771249f8a3626d1a7a35de34 P 8df3306d4bbe4fb1af0afb8d67c81798 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8df3306d4bbe4fb1af0afb8d67c81798, State: Initialized, Role: FOLLOWER
I20240502 21:25:26.709537 16905 tablet_bootstrap.cc:492] T 610bb5f6731047b3b86883445a9e05bb P 9efea4678e0e4dee8fe588222a3a6a67: No bootstrap required, opened a new log
I20240502 21:25:26.709504 16904 consensus_queue.cc:260] T e882da2f771249f8a3626d1a7a35de34 P 8df3306d4bbe4fb1af0afb8d67c81798 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cfec8b74a2342cca5723d915046f117" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 43833 } } peers { permanent_uuid: "9efea4678e0e4dee8fe588222a3a6a67" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37931 } } peers { permanent_uuid: "8df3306d4bbe4fb1af0afb8d67c81798" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37673 } }
I20240502 21:25:26.709692 16905 ts_tablet_manager.cc:1367] T 610bb5f6731047b3b86883445a9e05bb P 9efea4678e0e4dee8fe588222a3a6a67: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240502 21:25:26.709749 16903 ts_tablet_manager.cc:1398] T 1173ce275c4f40f0a0a31729310fbefd P 1cfec8b74a2342cca5723d915046f117: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:25:26.710032 16903 tablet_bootstrap.cc:492] T c5ea5b6dcf934533ba8df41e52711337 P 1cfec8b74a2342cca5723d915046f117: Bootstrap starting.
I20240502 21:25:26.710393 16905 raft_consensus.cc:348] T 610bb5f6731047b3b86883445a9e05bb P 9efea4678e0e4dee8fe588222a3a6a67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9efea4678e0e4dee8fe588222a3a6a67" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37931 } } peers { permanent_uuid: "1cfec8b74a2342cca5723d915046f117" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 43833 } } peers { permanent_uuid: "8df3306d4bbe4fb1af0afb8d67c81798" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37673 } }
I20240502 21:25:26.710574 16905 raft_consensus.cc:374] T 610bb5f6731047b3b86883445a9e05bb P 9efea4678e0e4dee8fe588222a3a6a67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:26.710649 16905 raft_consensus.cc:729] T 610bb5f6731047b3b86883445a9e05bb P 9efea4678e0e4dee8fe588222a3a6a67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9efea4678e0e4dee8fe588222a3a6a67, State: Initialized, Role: FOLLOWER
I20240502 21:25:26.710888 16916 raft_consensus.cc:482] T 38999367825249e1a0f5f156bd3c3b25 P 9efea4678e0e4dee8fe588222a3a6a67 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:25:26.710891 16905 consensus_queue.cc:260] T 610bb5f6731047b3b86883445a9e05bb P 9efea4678e0e4dee8fe588222a3a6a67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9efea4678e0e4dee8fe588222a3a6a67" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37931 } } peers { permanent_uuid: "1cfec8b74a2342cca5723d915046f117" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 43833 } } peers { permanent_uuid: "8df3306d4bbe4fb1af0afb8d67c81798" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37673 } }
I20240502 21:25:26.711067 16916 raft_consensus.cc:504] T 38999367825249e1a0f5f156bd3c3b25 P 9efea4678e0e4dee8fe588222a3a6a67 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9efea4678e0e4dee8fe588222a3a6a67" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37931 } } peers { permanent_uuid: "1cfec8b74a2342cca5723d915046f117" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 43833 } } peers { permanent_uuid: "8df3306d4bbe4fb1af0afb8d67c81798" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37673 } }
I20240502 21:25:26.711351 16905 ts_tablet_manager.cc:1398] T 610bb5f6731047b3b86883445a9e05bb P 9efea4678e0e4dee8fe588222a3a6a67: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 21:25:26.711639 16905 tablet_bootstrap.cc:492] T 1173ce275c4f40f0a0a31729310fbefd P 9efea4678e0e4dee8fe588222a3a6a67: Bootstrap starting.
I20240502 21:25:26.711733 16916 leader_election.cc:288] T 38999367825249e1a0f5f156bd3c3b25 P 9efea4678e0e4dee8fe588222a3a6a67 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cfec8b74a2342cca5723d915046f117 (127.15.255.130:43833), 8df3306d4bbe4fb1af0afb8d67c81798 (127.15.255.131:37673)
I20240502 21:25:26.711910 16903 tablet_bootstrap.cc:654] T c5ea5b6dcf934533ba8df41e52711337 P 1cfec8b74a2342cca5723d915046f117: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:26.712596 16702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38999367825249e1a0f5f156bd3c3b25" candidate_uuid: "9efea4678e0e4dee8fe588222a3a6a67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cfec8b74a2342cca5723d915046f117" is_pre_election: true
I20240502 21:25:26.712568 16904 ts_tablet_manager.cc:1398] T e882da2f771249f8a3626d1a7a35de34 P 8df3306d4bbe4fb1af0afb8d67c81798: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240502 21:25:26.713205 16904 tablet_bootstrap.cc:492] T 71d05c4f1e6f4f7186ed7b8e157b7d2f P 8df3306d4bbe4fb1af0afb8d67c81798: Bootstrap starting.
I20240502 21:25:26.713374 16905 tablet_bootstrap.cc:654] T 1173ce275c4f40f0a0a31729310fbefd P 9efea4678e0e4dee8fe588222a3a6a67: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:26.713351 16838 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38999367825249e1a0f5f156bd3c3b25" candidate_uuid: "9efea4678e0e4dee8fe588222a3a6a67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8df3306d4bbe4fb1af0afb8d67c81798" is_pre_election: true
W20240502 21:25:26.714357 16506 leader_election.cc:341] T 38999367825249e1a0f5f156bd3c3b25 P 9efea4678e0e4dee8fe588222a3a6a67 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cfec8b74a2342cca5723d915046f117 (127.15.255.130:43833): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:25:26.714813 16904 tablet_bootstrap.cc:654] T 71d05c4f1e6f4f7186ed7b8e157b7d2f P 8df3306d4bbe4fb1af0afb8d67c81798: Neither blocks nor log segments found. Creating new log.
W20240502 21:25:26.715061 16504 leader_election.cc:341] T 38999367825249e1a0f5f156bd3c3b25 P 9efea4678e0e4dee8fe588222a3a6a67 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8df3306d4bbe4fb1af0afb8d67c81798 (127.15.255.131:37673): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:25:26.715209 16504 leader_election.cc:302] T 38999367825249e1a0f5f156bd3c3b25 P 9efea4678e0e4dee8fe588222a3a6a67 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9efea4678e0e4dee8fe588222a3a6a67; no voters: 1cfec8b74a2342cca5723d915046f117, 8df3306d4bbe4fb1af0afb8d67c81798
I20240502 21:25:26.715620 16925 raft_consensus.cc:2726] T 38999367825249e1a0f5f156bd3c3b25 P 9efea4678e0e4dee8fe588222a3a6a67 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 21:25:26.718174 16905 tablet_bootstrap.cc:492] T 1173ce275c4f40f0a0a31729310fbefd P 9efea4678e0e4dee8fe588222a3a6a67: No bootstrap required, opened a new log
I20240502 21:25:26.718309 16905 ts_tablet_manager.cc:1367] T 1173ce275c4f40f0a0a31729310fbefd P 9efea4678e0e4dee8fe588222a3a6a67: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.001s
I20240502 21:25:26.718981 16905 raft_consensus.cc:348] T 1173ce275c4f40f0a0a31729310fbefd P 9efea4678e0e4dee8fe588222a3a6a67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9efea4678e0e4dee8fe588222a3a6a67" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37931 } } peers { permanent_uuid: "1cfec8b74a2342cca5723d915046f117" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 43833 } } peers { permanent_uuid: "8df3306d4bbe4fb1af0afb8d67c81798" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37673 } }
I20240502 21:25:26.719151 16905 raft_consensus.cc:374] T 1173ce275c4f40f0a0a31729310fbefd P 9efea4678e0e4dee8fe588222a3a6a67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:26.719246 16905 raft_consensus.cc:729] T 1173ce275c4f40f0a0a31729310fbefd P 9efea4678e0e4dee8fe588222a3a6a67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9efea4678e0e4dee8fe588222a3a6a67, State: Initialized, Role: FOLLOWER
I20240502 21:25:26.719473 16905 consensus_queue.cc:260] T 1173ce275c4f40f0a0a31729310fbefd P 9efea4678e0e4dee8fe588222a3a6a67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9efea4678e0e4dee8fe588222a3a6a67" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37931 } } peers { permanent_uuid: "1cfec8b74a2342cca5723d915046f117" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 43833 } } peers { permanent_uuid: "8df3306d4bbe4fb1af0afb8d67c81798" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37673 } }
I20240502 21:25:26.720007 16905 ts_tablet_manager.cc:1398] T 1173ce275c4f40f0a0a31729310fbefd P 9efea4678e0e4dee8fe588222a3a6a67: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 21:25:26.720265 16905 tablet_bootstrap.cc:492] T ed40aed15ca3447aa01283771ac257ed P 9efea4678e0e4dee8fe588222a3a6a67: Bootstrap starting.
I20240502 21:25:26.721618 16905 tablet_bootstrap.cc:654] T ed40aed15ca3447aa01283771ac257ed P 9efea4678e0e4dee8fe588222a3a6a67: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:26.723505 16903 tablet_bootstrap.cc:492] T c5ea5b6dcf934533ba8df41e52711337 P 1cfec8b74a2342cca5723d915046f117: No bootstrap required, opened a new log
I20240502 21:25:26.723685 16903 ts_tablet_manager.cc:1367] T c5ea5b6dcf934533ba8df41e52711337 P 1cfec8b74a2342cca5723d915046f117: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.007s
I20240502 21:25:26.723968 16905 tablet_bootstrap.cc:492] T ed40aed15ca3447aa01283771ac257ed P 9efea4678e0e4dee8fe588222a3a6a67: No bootstrap required, opened a new log
I20240502 21:25:26.724138 16905 ts_tablet_manager.cc:1367] T ed40aed15ca3447aa01283771ac257ed P 9efea4678e0e4dee8fe588222a3a6a67: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 21:25:26.724295 16904 tablet_bootstrap.cc:492] T 71d05c4f1e6f4f7186ed7b8e157b7d2f P 8df3306d4bbe4fb1af0afb8d67c81798: No bootstrap required, opened a new log
I20240502 21:25:26.724314 16903 raft_consensus.cc:348] T c5ea5b6dcf934533ba8df41e52711337 P 1cfec8b74a2342cca5723d915046f117 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cfec8b74a2342cca5723d915046f117" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 43833 } } peers { permanent_uuid: "9efea4678e0e4dee8fe588222a3a6a67" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37931 } } peers { permanent_uuid: "8df3306d4bbe4fb1af0afb8d67c81798" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37673 } }
I20240502 21:25:26.724450 16904 ts_tablet_manager.cc:1367] T 71d05c4f1e6f4f7186ed7b8e157b7d2f P 8df3306d4bbe4fb1af0afb8d67c81798: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.008s
I20240502 21:25:26.724521 16903 raft_consensus.cc:374] T c5ea5b6dcf934533ba8df41e52711337 P 1cfec8b74a2342cca5723d915046f117 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:26.724613 16903 raft_consensus.cc:729] T c5ea5b6dcf934533ba8df41e52711337 P 1cfec8b74a2342cca5723d915046f117 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cfec8b74a2342cca5723d915046f117, State: Initialized, Role: FOLLOWER
I20240502 21:25:26.724825 16903 consensus_queue.cc:260] T c5ea5b6dcf934533ba8df41e52711337 P 1cfec8b74a2342cca5723d915046f117 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cfec8b74a2342cca5723d915046f117" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 43833 } } peers { permanent_uuid: "9efea4678e0e4dee8fe588222a3a6a67" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37931 } } peers { permanent_uuid: "8df3306d4bbe4fb1af0afb8d67c81798" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37673 } }
I20240502 21:25:26.724830 16905 raft_consensus.cc:348] T ed40aed15ca3447aa01283771ac257ed P 9efea4678e0e4dee8fe588222a3a6a67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9efea4678e0e4dee8fe588222a3a6a67" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37931 } } peers { permanent_uuid: "1cfec8b74a2342cca5723d915046f117" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 43833 } } peers { permanent_uuid: "8df3306d4bbe4fb1af0afb8d67c81798" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37673 } }
I20240502 21:25:26.725064 16905 raft_consensus.cc:374] T ed40aed15ca3447aa01283771ac257ed P 9efea4678e0e4dee8fe588222a3a6a67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:26.725143 16905 raft_consensus.cc:729] T ed40aed15ca3447aa01283771ac257ed P 9efea4678e0e4dee8fe588222a3a6a67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9efea4678e0e4dee8fe588222a3a6a67, State: Initialized, Role: FOLLOWER
I20240502 21:25:26.725284 16903 ts_tablet_manager.cc:1398] T c5ea5b6dcf934533ba8df41e52711337 P 1cfec8b74a2342cca5723d915046f117: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 21:25:26.725277 16904 raft_consensus.cc:348] T 71d05c4f1e6f4f7186ed7b8e157b7d2f P 8df3306d4bbe4fb1af0afb8d67c81798 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9efea4678e0e4dee8fe588222a3a6a67" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37931 } } peers { permanent_uuid: "1cfec8b74a2342cca5723d915046f117" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 43833 } } peers { permanent_uuid: "8df3306d4bbe4fb1af0afb8d67c81798" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37673 } }
I20240502 21:25:26.725445 16904 raft_consensus.cc:374] T 71d05c4f1e6f4f7186ed7b8e157b7d2f P 8df3306d4bbe4fb1af0afb8d67c81798 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:26.725373 16905 consensus_queue.cc:260] T ed40aed15ca3447aa01283771ac257ed P 9efea4678e0e4dee8fe588222a3a6a67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9efea4678e0e4dee8fe588222a3a6a67" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37931 } } peers { permanent_uuid: "1cfec8b74a2342cca5723d915046f117" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 43833 } } peers { permanent_uuid: "8df3306d4bbe4fb1af0afb8d67c81798" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37673 } }
I20240502 21:25:26.725519 16904 raft_consensus.cc:729] T 71d05c4f1e6f4f7186ed7b8e157b7d2f P 8df3306d4bbe4fb1af0afb8d67c81798 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8df3306d4bbe4fb1af0afb8d67c81798, State: Initialized, Role: FOLLOWER
I20240502 21:25:26.725550 16903 tablet_bootstrap.cc:492] T cb358d933789476898f74949ffa81e49 P 1cfec8b74a2342cca5723d915046f117: Bootstrap starting.
I20240502 21:25:26.725843 16905 ts_tablet_manager.cc:1398] T ed40aed15ca3447aa01283771ac257ed P 9efea4678e0e4dee8fe588222a3a6a67: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:25:26.726138 16905 tablet_bootstrap.cc:492] T cb358d933789476898f74949ffa81e49 P 9efea4678e0e4dee8fe588222a3a6a67: Bootstrap starting.
I20240502 21:25:26.726250 16904 consensus_queue.cc:260] T 71d05c4f1e6f4f7186ed7b8e157b7d2f P 8df3306d4bbe4fb1af0afb8d67c81798 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9efea4678e0e4dee8fe588222a3a6a67" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37931 } } peers { permanent_uuid: "1cfec8b74a2342cca5723d915046f117" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 43833 } } peers { permanent_uuid: "8df3306d4bbe4fb1af0afb8d67c81798" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37673 } }
I20240502 21:25:26.727317 16904 ts_tablet_manager.cc:1398] T 71d05c4f1e6f4f7186ed7b8e157b7d2f P 8df3306d4bbe4fb1af0afb8d67c81798: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240502 21:25:26.727620 16905 tablet_bootstrap.cc:654] T cb358d933789476898f74949ffa81e49 P 9efea4678e0e4dee8fe588222a3a6a67: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:26.727836 16903 tablet_bootstrap.cc:654] T cb358d933789476898f74949ffa81e49 P 1cfec8b74a2342cca5723d915046f117: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:26.727594 16904 tablet_bootstrap.cc:492] T ed40aed15ca3447aa01283771ac257ed P 8df3306d4bbe4fb1af0afb8d67c81798: Bootstrap starting.
I20240502 21:25:26.729652 16904 tablet_bootstrap.cc:654] T ed40aed15ca3447aa01283771ac257ed P 8df3306d4bbe4fb1af0afb8d67c81798: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:26.738708 16903 tablet_bootstrap.cc:492] T cb358d933789476898f74949ffa81e49 P 1cfec8b74a2342cca5723d915046f117: No bootstrap required, opened a new log
I20240502 21:25:26.738868 16903 ts_tablet_manager.cc:1367] T cb358d933789476898f74949ffa81e49 P 1cfec8b74a2342cca5723d915046f117: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.008s
I20240502 21:25:26.738920 16907 raft_consensus.cc:482] T c5ea5b6dcf934533ba8df41e52711337 P 1cfec8b74a2342cca5723d915046f117 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:25:26.739073 16907 raft_consensus.cc:504] T c5ea5b6dcf934533ba8df41e52711337 P 1cfec8b74a2342cca5723d915046f117 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cfec8b74a2342cca5723d915046f117" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 43833 } } peers { permanent_uuid: "9efea4678e0e4dee8fe588222a3a6a67" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37931 } } peers { permanent_uuid: "8df3306d4bbe4fb1af0afb8d67c81798" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37673 } }
I20240502 21:25:26.739172 16904 tablet_bootstrap.cc:492] T ed40aed15ca3447aa01283771ac257ed P 8df3306d4bbe4fb1af0afb8d67c81798: No bootstrap required, opened a new log
I20240502 21:25:26.739306 16904 ts_tablet_manager.cc:1367] T ed40aed15ca3447aa01283771ac257ed P 8df3306d4bbe4fb1af0afb8d67c81798: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.010s
I20240502 21:25:26.739454 16905 tablet_bootstrap.cc:492] T cb358d933789476898f74949ffa81e49 P 9efea4678e0e4dee8fe588222a3a6a67: No bootstrap required, opened a new log
I20240502 21:25:26.739565 16905 ts_tablet_manager.cc:1367] T cb358d933789476898f74949ffa81e49 P 9efea4678e0e4dee8fe588222a3a6a67: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.006s
I20240502 21:25:26.739675 16907 leader_election.cc:288] T c5ea5b6dcf934533ba8df41e52711337 P 1cfec8b74a2342cca5723d915046f117 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9efea4678e0e4dee8fe588222a3a6a67 (127.15.255.129:37931), 8df3306d4bbe4fb1af0afb8d67c81798 (127.15.255.131:37673)
I20240502 21:25:26.739609 16903 raft_consensus.cc:348] T cb358d933789476898f74949ffa81e49 P 1cfec8b74a2342cca5723d915046f117 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9efea4678e0e4dee8fe588222a3a6a67" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37931 } } peers { permanent_uuid: "1cfec8b74a2342cca5723d915046f117" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 43833 } } peers { permanent_uuid: "8df3306d4bbe4fb1af0afb8d67c81798" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37673 } }
I20240502 21:25:26.739820 16903 raft_consensus.cc:374] T cb358d933789476898f74949ffa81e49 P 1cfec8b74a2342cca5723d915046f117 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:26.740113 16903 raft_consensus.cc:729] T cb358d933789476898f74949ffa81e49 P 1cfec8b74a2342cca5723d915046f117 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cfec8b74a2342cca5723d915046f117, State: Initialized, Role: FOLLOWER
I20240502 21:25:26.740140 16838 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5ea5b6dcf934533ba8df41e52711337" candidate_uuid: "1cfec8b74a2342cca5723d915046f117" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8df3306d4bbe4fb1af0afb8d67c81798" is_pre_election: true
I20240502 21:25:26.740234 16570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5ea5b6dcf934533ba8df41e52711337" candidate_uuid: "1cfec8b74a2342cca5723d915046f117" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9efea4678e0e4dee8fe588222a3a6a67" is_pre_election: true
I20240502 21:25:26.740414 16570 raft_consensus.cc:2454] T c5ea5b6dcf934533ba8df41e52711337 P 9efea4678e0e4dee8fe588222a3a6a67 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cfec8b74a2342cca5723d915046f117 in term 0.
I20240502 21:25:26.740466 16838 raft_consensus.cc:2454] T c5ea5b6dcf934533ba8df41e52711337 P 8df3306d4bbe4fb1af0afb8d67c81798 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cfec8b74a2342cca5723d915046f117 in term 0.
I20240502 21:25:26.740375 16903 consensus_queue.cc:260] T cb358d933789476898f74949ffa81e49 P 1cfec8b74a2342cca5723d915046f117 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9efea4678e0e4dee8fe588222a3a6a67" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37931 } } peers { permanent_uuid: "1cfec8b74a2342cca5723d915046f117" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 43833 } } peers { permanent_uuid: "8df3306d4bbe4fb1af0afb8d67c81798" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37673 } }
I20240502 21:25:26.740885 16903 ts_tablet_manager.cc:1398] T cb358d933789476898f74949ffa81e49 P 1cfec8b74a2342cca5723d915046f117: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:25:26.740890 16635 leader_election.cc:302] T c5ea5b6dcf934533ba8df41e52711337 P 1cfec8b74a2342cca5723d915046f117 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cfec8b74a2342cca5723d915046f117, 9efea4678e0e4dee8fe588222a3a6a67; no voters:
I20240502 21:25:26.741190 16903 tablet_bootstrap.cc:492] T 72f6a167c03d4ee6bd72d6e5793f784d P 1cfec8b74a2342cca5723d915046f117: Bootstrap starting.
I20240502 21:25:26.741429 16907 raft_consensus.cc:2781] T c5ea5b6dcf934533ba8df41e52711337 P 1cfec8b74a2342cca5723d915046f117 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:25:26.741531 16907 raft_consensus.cc:482] T c5ea5b6dcf934533ba8df41e52711337 P 1cfec8b74a2342cca5723d915046f117 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:25:26.741644 16907 raft_consensus.cc:3037] T c5ea5b6dcf934533ba8df41e52711337 P 1cfec8b74a2342cca5723d915046f117 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:26.741848 16905 raft_consensus.cc:348] T cb358d933789476898f74949ffa81e49 P 9efea4678e0e4dee8fe588222a3a6a67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9efea4678e0e4dee8fe588222a3a6a67" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37931 } } peers { permanent_uuid: "1cfec8b74a2342cca5723d915046f117" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 43833 } } peers { permanent_uuid: "8df3306d4bbe4fb1af0afb8d67c81798" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37673 } }
I20240502 21:25:26.742097 16905 raft_consensus.cc:374] T cb358d933789476898f74949ffa81e49 P 9efea4678e0e4dee8fe588222a3a6a67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:26.742224 16905 raft_consensus.cc:729] T cb358d933789476898f74949ffa81e49 P 9efea4678e0e4dee8fe588222a3a6a67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9efea4678e0e4dee8fe588222a3a6a67, State: Initialized, Role: FOLLOWER
I20240502 21:25:26.742478 16905 consensus_queue.cc:260] T cb358d933789476898f74949ffa81e49 P 9efea4678e0e4dee8fe588222a3a6a67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9efea4678e0e4dee8fe588222a3a6a67" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37931 } } peers { permanent_uuid: "1cfec8b74a2342cca5723d915046f117" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 43833 } } peers { permanent_uuid: "8df3306d4bbe4fb1af0afb8d67c81798" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37673 } }
I20240502 21:25:26.742760 16903 tablet_bootstrap.cc:654] T 72f6a167c03d4ee6bd72d6e5793f784d P 1cfec8b74a2342cca5723d915046f117: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:26.743121 16905 ts_tablet_manager.cc:1398] T cb358d933789476898f74949ffa81e49 P 9efea4678e0e4dee8fe588222a3a6a67: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240502 21:25:26.743417 16905 tablet_bootstrap.cc:492] T 71d05c4f1e6f4f7186ed7b8e157b7d2f P 9efea4678e0e4dee8fe588222a3a6a67: Bootstrap starting.
I20240502 21:25:26.744498 16907 raft_consensus.cc:504] T c5ea5b6dcf934533ba8df41e52711337 P 1cfec8b74a2342cca5723d915046f117 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cfec8b74a2342cca5723d915046f117" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 43833 } } peers { permanent_uuid: "9efea4678e0e4dee8fe588222a3a6a67" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37931 } } peers { permanent_uuid: "8df3306d4bbe4fb1af0afb8d67c81798" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37673 } }
I20240502 21:25:26.744951 16905 tablet_bootstrap.cc:654] T 71d05c4f1e6f4f7186ed7b8e157b7d2f P 9efea4678e0e4dee8fe588222a3a6a67: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:26.746168 16570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5ea5b6dcf934533ba8df41e52711337" candidate_uuid: "1cfec8b74a2342cca5723d915046f117" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9efea4678e0e4dee8fe588222a3a6a67"
I20240502 21:25:26.746364 16570 raft_consensus.cc:3037] T c5ea5b6dcf934533ba8df41e52711337 P 9efea4678e0e4dee8fe588222a3a6a67 [term 0 FOLLOWER]: Advancing to term 1
W20240502 21:25:26.746870 16618 tablet.cc:2369] T 05a7626e52ae43a6829feacaf1837ea5 P 9efea4678e0e4dee8fe588222a3a6a67: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 21:25:26.749415 16570 raft_consensus.cc:2454] T c5ea5b6dcf934533ba8df41e52711337 P 9efea4678e0e4dee8fe588222a3a6a67 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cfec8b74a2342cca5723d915046f117 in term 1.
I20240502 21:25:26.749945 16635 leader_election.cc:302] T c5ea5b6dcf934533ba8df41e52711337 P 1cfec8b74a2342cca5723d915046f117 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cfec8b74a2342cca5723d915046f117, 9efea4678e0e4dee8fe588222a3a6a67; no voters:
I20240502 21:25:26.750252 16929 raft_consensus.cc:2781] T c5ea5b6dcf934533ba8df41e52711337 P 1cfec8b74a2342cca5723d915046f117 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:25:26.750303 16905 tablet_bootstrap.cc:492] T 71d05c4f1e6f4f7186ed7b8e157b7d2f P 9efea4678e0e4dee8fe588222a3a6a67: No bootstrap required, opened a new log
I20240502 21:25:26.750530 16905 ts_tablet_manager.cc:1367] T 71d05c4f1e6f4f7186ed7b8e157b7d2f P 9efea4678e0e4dee8fe588222a3a6a67: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20240502 21:25:26.751317 16905 raft_consensus.cc:348] T 71d05c4f1e6f4f7186ed7b8e157b7d2f P 9efea4678e0e4dee8fe588222a3a6a67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9efea4678e0e4dee8fe588222a3a6a67" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37931 } } peers { permanent_uuid: "1cfec8b74a2342cca5723d915046f117" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 43833 } } peers { permanent_uuid: "8df3306d4bbe4fb1af0afb8d67c81798" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37673 } }
I20240502 21:25:26.751595 16905 raft_consensus.cc:374] T 71d05c4f1e6f4f7186ed7b8e157b7d2f P 9efea4678e0e4dee8fe588222a3a6a67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:26.751715 16905 raft_consensus.cc:729] T 71d05c4f1e6f4f7186ed7b8e157b7d2f P 9efea4678e0e4dee8fe588222a3a6a67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9efea4678e0e4dee8fe588222a3a6a67, State: Initialized, Role: FOLLOWER
I20240502 21:25:26.752454 16907 leader_election.cc:288] T c5ea5b6dcf934533ba8df41e52711337 P 1cfec8b74a2342cca5723d915046f117 [CANDIDATE]: Term 1 election: Requested vote from peers 9efea4678e0e4dee8fe588222a3a6a67 (127.15.255.129:37931), 8df3306d4bbe4fb1af0afb8d67c81798 (127.15.255.131:37673)
I20240502 21:25:26.752564 16838 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5ea5b6dcf934533ba8df41e52711337" candidate_uuid: "1cfec8b74a2342cca5723d915046f117" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8df3306d4bbe4fb1af0afb8d67c81798"
I20240502 21:25:26.752748 16838 raft_consensus.cc:3037] T c5ea5b6dcf934533ba8df41e52711337 P 8df3306d4bbe4fb1af0afb8d67c81798 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:26.756088 16838 raft_consensus.cc:2454] T c5ea5b6dcf934533ba8df41e52711337 P 8df3306d4bbe4fb1af0afb8d67c81798 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cfec8b74a2342cca5723d915046f117 in term 1.
I20240502 21:25:26.752039 16905 consensus_queue.cc:260] T 71d05c4f1e6f4f7186ed7b8e157b7d2f P 9efea4678e0e4dee8fe588222a3a6a67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9efea4678e0e4dee8fe588222a3a6a67" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37931 } } peers { permanent_uuid: "1cfec8b74a2342cca5723d915046f117" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 43833 } } peers { permanent_uuid: "8df3306d4bbe4fb1af0afb8d67c81798" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37673 } }
I20240502 21:25:26.757284 16905 ts_tablet_manager.cc:1398] T 71d05c4f1e6f4f7186ed7b8e157b7d2f P 9efea4678e0e4dee8fe588222a3a6a67: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240502 21:25:26.739964 16904 raft_consensus.cc:348] T ed40aed15ca3447aa01283771ac257ed P 8df3306d4bbe4fb1af0afb8d67c81798 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9efea4678e0e4dee8fe588222a3a6a67" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37931 } } peers { permanent_uuid: "1cfec8b74a2342cca5723d915046f117" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 43833 } } peers { permanent_uuid: "8df3306d4bbe4fb1af0afb8d67c81798" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37673 } }
I20240502 21:25:26.757601 16904 raft_consensus.cc:374] T ed40aed15ca3447aa01283771ac257ed P 8df3306d4bbe4fb1af0afb8d67c81798 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:26.757696 16904 raft_consensus.cc:729] T ed40aed15ca3447aa01283771ac257ed P 8df3306d4bbe4fb1af0afb8d67c81798 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8df3306d4bbe4fb1af0afb8d67c81798, State: Initialized, Role: FOLLOWER
I20240502 21:25:26.757720 16905 tablet_bootstrap.cc:492] T 2b1fbeb84317494595f7262fe7857ebd P 9efea4678e0e4dee8fe588222a3a6a67: Bootstrap starting.
I20240502 21:25:26.757933 16904 consensus_queue.cc:260] T ed40aed15ca3447aa01283771ac257ed P 8df3306d4bbe4fb1af0afb8d67c81798 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9efea4678e0e4dee8fe588222a3a6a67" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37931 } } peers { permanent_uuid: "1cfec8b74a2342cca5723d915046f117" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 43833 } } peers { permanent_uuid: "8df3306d4bbe4fb1af0afb8d67c81798" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37673 } }
I20240502 21:25:26.759361 16905 tablet_bootstrap.cc:654] T 2b1fbeb84317494595f7262fe7857ebd P 9efea4678e0e4dee8fe588222a3a6a67: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:26.759519 16904 ts_tablet_manager.cc:1398] T ed40aed15ca3447aa01283771ac257ed P 8df3306d4bbe4fb1af0afb8d67c81798: Time spent starting tablet: real 0.020s user 0.001s sys 0.001s
I20240502 21:25:26.759832 16904 tablet_bootstrap.cc:492] T 38999367825249e1a0f5f156bd3c3b25 P 8df3306d4bbe4fb1af0afb8d67c81798: Bootstrap starting.
I20240502 21:25:26.761035 16903 tablet_bootstrap.cc:492] T 72f6a167c03d4ee6bd72d6e5793f784d P 1cfec8b74a2342cca5723d915046f117: No bootstrap required, opened a new log
I20240502 21:25:26.761044 16929 raft_consensus.cc:686] T c5ea5b6dcf934533ba8df41e52711337 P 1cfec8b74a2342cca5723d915046f117 [term 1 LEADER]: Becoming Leader. State: Replica: 1cfec8b74a2342cca5723d915046f117, State: Running, Role: LEADER
I20240502 21:25:26.761209 16903 ts_tablet_manager.cc:1367] T 72f6a167c03d4ee6bd72d6e5793f784d P 1cfec8b74a2342cca5723d915046f117: Time spent bootstrapping tablet: real 0.020s user 0.017s sys 0.001s
I20240502 21:25:26.761466 16929 consensus_queue.cc:237] T c5ea5b6dcf934533ba8df41e52711337 P 1cfec8b74a2342cca5723d915046f117 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cfec8b74a2342cca5723d915046f117" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 43833 } } peers { permanent_uuid: "9efea4678e0e4dee8fe588222a3a6a67" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37931 } } peers { permanent_uuid: "8df3306d4bbe4fb1af0afb8d67c81798" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37673 } }
I20240502 21:25:26.761998 16903 raft_consensus.cc:348] T 72f6a167c03d4ee6bd72d6e5793f784d P 1cfec8b74a2342cca5723d915046f117 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9efea4678e0e4dee8fe588222a3a6a67" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37931 } } peers { permanent_uuid: "1cfec8b74a2342cca5723d915046f117" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 43833 } } peers { permanent_uuid: "8df3306d4bbe4fb1af0afb8d67c81798" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37673 } }
I20240502 21:25:26.762202 16903 raft_consensus.cc:374] T 72f6a167c03d4ee6bd72d6e5793f784d P 1cfec8b74a2342cca5723d915046f117 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:26.762316 16903 raft_consensus.cc:729] T 72f6a167c03d4ee6bd72d6e5793f784d P 1cfec8b74a2342cca5723d915046f117 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cfec8b74a2342cca5723d915046f117, State: Initialized, Role: FOLLOWER
I20240502 21:25:26.762547 16903 consensus_queue.cc:260] T 72f6a167c03d4ee6bd72d6e5793f784d P 1cfec8b74a2342cca5723d915046f117 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9efea4678e0e4dee8fe588222a3a6a67" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37931 } } peers { permanent_uuid: "1cfec8b74a2342cca5723d915046f117" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 43833 } } peers { permanent_uuid: "8df3306d4bbe4fb1af0afb8d67c81798" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37673 } }
I20240502 21:25:26.763131 16903 ts_tablet_manager.cc:1398] T 72f6a167c03d4ee6bd72d6e5793f784d P 1cfec8b74a2342cca5723d915046f117: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:25:26.763244 16904 tablet_bootstrap.cc:654] T 38999367825249e1a0f5f156bd3c3b25 P 8df3306d4bbe4fb1af0afb8d67c81798: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:26.763404 16903 tablet_bootstrap.cc:492] T e882da2f771249f8a3626d1a7a35de34 P 1cfec8b74a2342cca5723d915046f117: Bootstrap starting.
I20240502 21:25:26.764964 16903 tablet_bootstrap.cc:654] T e882da2f771249f8a3626d1a7a35de34 P 1cfec8b74a2342cca5723d915046f117: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:26.765806 16905 tablet_bootstrap.cc:492] T 2b1fbeb84317494595f7262fe7857ebd P 9efea4678e0e4dee8fe588222a3a6a67: No bootstrap required, opened a new log
I20240502 21:25:26.766002 16905 ts_tablet_manager.cc:1367] T 2b1fbeb84317494595f7262fe7857ebd P 9efea4678e0e4dee8fe588222a3a6a67: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20240502 21:25:26.766731 16905 raft_consensus.cc:348] T 2b1fbeb84317494595f7262fe7857ebd P 9efea4678e0e4dee8fe588222a3a6a67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cfec8b74a2342cca5723d915046f117" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 43833 } } peers { permanent_uuid: "9efea4678e0e4dee8fe588222a3a6a67" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37931 } } peers { permanent_uuid: "8df3306d4bbe4fb1af0afb8d67c81798" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37673 } }
I20240502 21:25:26.766985 16905 raft_consensus.cc:374] T 2b1fbeb84317494595f7262fe7857ebd P 9efea4678e0e4dee8fe588222a3a6a67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:26.767112 16905 raft_consensus.cc:729] T 2b1fbeb84317494595f7262fe7857ebd P 9efea4678e0e4dee8fe588222a3a6a67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9efea4678e0e4dee8fe588222a3a6a67, State: Initialized, Role: FOLLOWER
I20240502 21:25:26.767381 16905 consensus_queue.cc:260] T 2b1fbeb84317494595f7262fe7857ebd P 9efea4678e0e4dee8fe588222a3a6a67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cfec8b74a2342cca5723d915046f117" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 43833 } } peers { permanent_uuid: "9efea4678e0e4dee8fe588222a3a6a67" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37931 } } peers { permanent_uuid: "8df3306d4bbe4fb1af0afb8d67c81798" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37673 } }
I20240502 21:25:26.768050 16905 ts_tablet_manager.cc:1398] T 2b1fbeb84317494595f7262fe7857ebd P 9efea4678e0e4dee8fe588222a3a6a67: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 21:25:26.768337 16905 tablet_bootstrap.cc:492] T 62a4fb5953dc4fa7badc5e2ee1ed4261 P 9efea4678e0e4dee8fe588222a3a6a67: Bootstrap starting.
I20240502 21:25:26.769832 16905 tablet_bootstrap.cc:654] T 62a4fb5953dc4fa7badc5e2ee1ed4261 P 9efea4678e0e4dee8fe588222a3a6a67: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:26.772920 16904 tablet_bootstrap.cc:492] T 38999367825249e1a0f5f156bd3c3b25 P 8df3306d4bbe4fb1af0afb8d67c81798: No bootstrap required, opened a new log
I20240502 21:25:26.773100 16904 ts_tablet_manager.cc:1367] T 38999367825249e1a0f5f156bd3c3b25 P 8df3306d4bbe4fb1af0afb8d67c81798: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.001s
I20240502 21:25:26.773335 16426 catalog_manager.cc:5484] T c5ea5b6dcf934533ba8df41e52711337 P 1cfec8b74a2342cca5723d915046f117 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1cfec8b74a2342cca5723d915046f117 (127.15.255.130). New cstate: current_term: 1 leader_uuid: "1cfec8b74a2342cca5723d915046f117" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cfec8b74a2342cca5723d915046f117" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 43833 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9efea4678e0e4dee8fe588222a3a6a67" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37931 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8df3306d4bbe4fb1af0afb8d67c81798" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37673 } health_report { overall_health: UNKNOWN } } }
I20240502 21:25:26.774073 16904 raft_consensus.cc:348] T 38999367825249e1a0f5f156bd3c3b25 P 8df3306d4bbe4fb1af0afb8d67c81798 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9efea4678e0e4dee8fe588222a3a6a67" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37931 } } peers { permanent_uuid: "1cfec8b74a2342cca5723d915046f117" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 43833 } } peers { permanent_uuid: "8df3306d4bbe4fb1af0afb8d67c81798" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37673 } }
I20240502 21:25:26.774292 16904 raft_consensus.cc:374] T 38999367825249e1a0f5f156bd3c3b25 P 8df3306d4bbe4fb1af0afb8d67c81798 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:26.774382 16904 raft_consensus.cc:729] T 38999367825249e1a0f5f156bd3c3b25 P 8df3306d4bbe4fb1af0afb8d67c81798 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8df3306d4bbe4fb1af0afb8d67c81798, State: Initialized, Role: FOLLOWER
I20240502 21:25:26.774632 16904 consensus_queue.cc:260] T 38999367825249e1a0f5f156bd3c3b25 P 8df3306d4bbe4fb1af0afb8d67c81798 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9efea4678e0e4dee8fe588222a3a6a67" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37931 } } peers { permanent_uuid: "1cfec8b74a2342cca5723d915046f117" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 43833 } } peers { permanent_uuid: "8df3306d4bbe4fb1af0afb8d67c81798" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37673 } }
I20240502 21:25:26.775269 16904 ts_tablet_manager.cc:1398] T 38999367825249e1a0f5f156bd3c3b25 P 8df3306d4bbe4fb1af0afb8d67c81798: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:25:26.775552 16904 tablet_bootstrap.cc:492] T 72f6a167c03d4ee6bd72d6e5793f784d P 8df3306d4bbe4fb1af0afb8d67c81798: Bootstrap starting.
I20240502 21:25:26.775684 16925 raft_consensus.cc:482] T 05a7626e52ae43a6829feacaf1837ea5 P 9efea4678e0e4dee8fe588222a3a6a67 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:25:26.775825 16925 raft_consensus.cc:504] T 05a7626e52ae43a6829feacaf1837ea5 P 9efea4678e0e4dee8fe588222a3a6a67 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9efea4678e0e4dee8fe588222a3a6a67" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37931 } } peers { permanent_uuid: "8df3306d4bbe4fb1af0afb8d67c81798" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37673 } } peers { permanent_uuid: "1cfec8b74a2342cca5723d915046f117" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 43833 } }
I20240502 21:25:26.776723 16925 leader_election.cc:288] T 05a7626e52ae43a6829feacaf1837ea5 P 9efea4678e0e4dee8fe588222a3a6a67 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8df3306d4bbe4fb1af0afb8d67c81798 (127.15.255.131:37673), 1cfec8b74a2342cca5723d915046f117 (127.15.255.130:43833)
I20240502 21:25:26.777212 16904 tablet_bootstrap.cc:654] T 72f6a167c03d4ee6bd72d6e5793f784d P 8df3306d4bbe4fb1af0afb8d67c81798: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:26.777498 16838 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05a7626e52ae43a6829feacaf1837ea5" candidate_uuid: "9efea4678e0e4dee8fe588222a3a6a67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8df3306d4bbe4fb1af0afb8d67c81798" is_pre_election: true
W20240502 21:25:26.778002 16504 leader_election.cc:341] T 05a7626e52ae43a6829feacaf1837ea5 P 9efea4678e0e4dee8fe588222a3a6a67 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8df3306d4bbe4fb1af0afb8d67c81798 (127.15.255.131:37673): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:25:26.778599 16905 tablet_bootstrap.cc:492] T 62a4fb5953dc4fa7badc5e2ee1ed4261 P 9efea4678e0e4dee8fe588222a3a6a67: No bootstrap required, opened a new log
I20240502 21:25:26.778632 16702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05a7626e52ae43a6829feacaf1837ea5" candidate_uuid: "9efea4678e0e4dee8fe588222a3a6a67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cfec8b74a2342cca5723d915046f117" is_pre_election: true
I20240502 21:25:26.778810 16905 ts_tablet_manager.cc:1367] T 62a4fb5953dc4fa7badc5e2ee1ed4261 P 9efea4678e0e4dee8fe588222a3a6a67: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.003s
I20240502 21:25:26.778896 16702 raft_consensus.cc:2454] T 05a7626e52ae43a6829feacaf1837ea5 P 1cfec8b74a2342cca5723d915046f117 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9efea4678e0e4dee8fe588222a3a6a67 in term 0.
I20240502 21:25:26.779536 16905 raft_consensus.cc:348] T 62a4fb5953dc4fa7badc5e2ee1ed4261 P 9efea4678e0e4dee8fe588222a3a6a67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9efea4678e0e4dee8fe588222a3a6a67" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37931 } } peers { permanent_uuid: "1cfec8b74a2342cca5723d915046f117" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 43833 } } peers { permanent_uuid: "8df3306d4bbe4fb1af0afb8d67c81798" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37673 } }
I20240502 21:25:26.779659 16506 leader_election.cc:302] T 05a7626e52ae43a6829feacaf1837ea5 P 9efea4678e0e4dee8fe588222a3a6a67 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cfec8b74a2342cca5723d915046f117, 9efea4678e0e4dee8fe588222a3a6a67; no voters: 8df3306d4bbe4fb1af0afb8d67c81798
I20240502 21:25:26.779769 16905 raft_consensus.cc:374] T 62a4fb5953dc4fa7badc5e2ee1ed4261 P 9efea4678e0e4dee8fe588222a3a6a67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:26.779842 16905 raft_consensus.cc:729] T 62a4fb5953dc4fa7badc5e2ee1ed4261 P 9efea4678e0e4dee8fe588222a3a6a67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9efea4678e0e4dee8fe588222a3a6a67, State: Initialized, Role: FOLLOWER
I20240502 21:25:26.780114 16925 raft_consensus.cc:2781] T 05a7626e52ae43a6829feacaf1837ea5 P 9efea4678e0e4dee8fe588222a3a6a67 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:25:26.780215 16925 raft_consensus.cc:482] T 05a7626e52ae43a6829feacaf1837ea5 P 9efea4678e0e4dee8fe588222a3a6a67 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:25:26.780143 16905 consensus_queue.cc:260] T 62a4fb5953dc4fa7badc5e2ee1ed4261 P 9efea4678e0e4dee8fe588222a3a6a67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9efea4678e0e4dee8fe588222a3a6a67" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37931 } } peers { permanent_uuid: "1cfec8b74a2342cca5723d915046f117" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 43833 } } peers { permanent_uuid: "8df3306d4bbe4fb1af0afb8d67c81798" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37673 } }
I20240502 21:25:26.780325 16925 raft_consensus.cc:3037] T 05a7626e52ae43a6829feacaf1837ea5 P 9efea4678e0e4dee8fe588222a3a6a67 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:26.781296 16905 ts_tablet_manager.cc:1398] T 62a4fb5953dc4fa7badc5e2ee1ed4261 P 9efea4678e0e4dee8fe588222a3a6a67: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 21:25:26.781595 16905 tablet_bootstrap.cc:492] T e882da2f771249f8a3626d1a7a35de34 P 9efea4678e0e4dee8fe588222a3a6a67: Bootstrap starting.
I20240502 21:25:26.783146 16905 tablet_bootstrap.cc:654] T e882da2f771249f8a3626d1a7a35de34 P 9efea4678e0e4dee8fe588222a3a6a67: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:26.783486 16925 raft_consensus.cc:504] T 05a7626e52ae43a6829feacaf1837ea5 P 9efea4678e0e4dee8fe588222a3a6a67 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9efea4678e0e4dee8fe588222a3a6a67" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37931 } } peers { permanent_uuid: "8df3306d4bbe4fb1af0afb8d67c81798" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37673 } } peers { permanent_uuid: "1cfec8b74a2342cca5723d915046f117" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 43833 } }
I20240502 21:25:26.784775 16838 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05a7626e52ae43a6829feacaf1837ea5" candidate_uuid: "9efea4678e0e4dee8fe588222a3a6a67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8df3306d4bbe4fb1af0afb8d67c81798"
W20240502 21:25:26.785338 16504 leader_election.cc:341] T 05a7626e52ae43a6829feacaf1837ea5 P 9efea4678e0e4dee8fe588222a3a6a67 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8df3306d4bbe4fb1af0afb8d67c81798 (127.15.255.131:37673): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:25:26.785725 16925 leader_election.cc:288] T 05a7626e52ae43a6829feacaf1837ea5 P 9efea4678e0e4dee8fe588222a3a6a67 [CANDIDATE]: Term 1 election: Requested vote from peers 8df3306d4bbe4fb1af0afb8d67c81798 (127.15.255.131:37673), 1cfec8b74a2342cca5723d915046f117 (127.15.255.130:43833)
I20240502 21:25:26.788342 16702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05a7626e52ae43a6829feacaf1837ea5" candidate_uuid: "9efea4678e0e4dee8fe588222a3a6a67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cfec8b74a2342cca5723d915046f117"
I20240502 21:25:26.788579 16702 raft_consensus.cc:3037] T 05a7626e52ae43a6829feacaf1837ea5 P 1cfec8b74a2342cca5723d915046f117 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:26.789037 16904 tablet_bootstrap.cc:492] T 72f6a167c03d4ee6bd72d6e5793f784d P 8df3306d4bbe4fb1af0afb8d67c81798: No bootstrap required, opened a new log
I20240502 21:25:26.789186 16904 ts_tablet_manager.cc:1367] T 72f6a167c03d4ee6bd72d6e5793f784d P 8df3306d4bbe4fb1af0afb8d67c81798: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.008s
I20240502 21:25:26.789878 16904 raft_consensus.cc:348] T 72f6a167c03d4ee6bd72d6e5793f784d P 8df3306d4bbe4fb1af0afb8d67c81798 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9efea4678e0e4dee8fe588222a3a6a67" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37931 } } peers { permanent_uuid: "1cfec8b74a2342cca5723d915046f117" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 43833 } } peers { permanent_uuid: "8df3306d4bbe4fb1af0afb8d67c81798" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37673 } }
I20240502 21:25:26.790092 16904 raft_consensus.cc:374] T 72f6a167c03d4ee6bd72d6e5793f784d P 8df3306d4bbe4fb1af0afb8d67c81798 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:26.790166 16904 raft_consensus.cc:729] T 72f6a167c03d4ee6bd72d6e5793f784d P 8df3306d4bbe4fb1af0afb8d67c81798 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8df3306d4bbe4fb1af0afb8d67c81798, State: Initialized, Role: FOLLOWER
I20240502 21:25:26.787956 16905 tablet_bootstrap.cc:492] T e882da2f771249f8a3626d1a7a35de34 P 9efea4678e0e4dee8fe588222a3a6a67: No bootstrap required, opened a new log
I20240502 21:25:26.790362 16905 ts_tablet_manager.cc:1367] T e882da2f771249f8a3626d1a7a35de34 P 9efea4678e0e4dee8fe588222a3a6a67: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.001s
I20240502 21:25:26.790381 16904 consensus_queue.cc:260] T 72f6a167c03d4ee6bd72d6e5793f784d P 8df3306d4bbe4fb1af0afb8d67c81798 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9efea4678e0e4dee8fe588222a3a6a67" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37931 } } peers { permanent_uuid: "1cfec8b74a2342cca5723d915046f117" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 43833 } } peers { permanent_uuid: "8df3306d4bbe4fb1af0afb8d67c81798" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37673 } }
I20240502 21:25:26.791019 16905 raft_consensus.cc:348] T e882da2f771249f8a3626d1a7a35de34 P 9efea4678e0e4dee8fe588222a3a6a67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cfec8b74a2342cca5723d915046f117" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 43833 } } peers { permanent_uuid: "9efea4678e0e4dee8fe588222a3a6a67" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37931 } } peers { permanent_uuid: "8df3306d4bbe4fb1af0afb8d67c81798" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37673 } }
I20240502 21:25:26.791244 16905 raft_consensus.cc:374] T e882da2f771249f8a3626d1a7a35de34 P 9efea4678e0e4dee8fe588222a3a6a67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:26.791323 16905 raft_consensus.cc:729] T e882da2f771249f8a3626d1a7a35de34 P 9efea4678e0e4dee8fe588222a3a6a67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9efea4678e0e4dee8fe588222a3a6a67, State: Initialized, Role: FOLLOWER
I20240502 21:25:26.791496 16905 consensus_queue.cc:260] T e882da2f771249f8a3626d1a7a35de34 P 9efea4678e0e4dee8fe588222a3a6a67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cfec8b74a2342cca5723d915046f117" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 43833 } } peers { permanent_uuid: "9efea4678e0e4dee8fe588222a3a6a67" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37931 } } peers { permanent_uuid: "8df3306d4bbe4fb1af0afb8d67c81798" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37673 } }
I20240502 21:25:26.791568 16702 raft_consensus.cc:2454] T 05a7626e52ae43a6829feacaf1837ea5 P 1cfec8b74a2342cca5723d915046f117 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9efea4678e0e4dee8fe588222a3a6a67 in term 1.
I20240502 21:25:26.791682 16904 ts_tablet_manager.cc:1398] T 72f6a167c03d4ee6bd72d6e5793f784d P 8df3306d4bbe4fb1af0afb8d67c81798: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 21:25:26.791980 16904 tablet_bootstrap.cc:492] T 05a7626e52ae43a6829feacaf1837ea5 P 8df3306d4bbe4fb1af0afb8d67c81798: Bootstrap starting.
I20240502 21:25:26.792110 16905 ts_tablet_manager.cc:1398] T e882da2f771249f8a3626d1a7a35de34 P 9efea4678e0e4dee8fe588222a3a6a67: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 21:25:26.792333 16905 tablet_bootstrap.cc:492] T 72f6a167c03d4ee6bd72d6e5793f784d P 9efea4678e0e4dee8fe588222a3a6a67: Bootstrap starting.
I20240502 21:25:26.793493 16904 tablet_bootstrap.cc:654] T 05a7626e52ae43a6829feacaf1837ea5 P 8df3306d4bbe4fb1af0afb8d67c81798: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:26.793849 16905 tablet_bootstrap.cc:654] T 72f6a167c03d4ee6bd72d6e5793f784d P 9efea4678e0e4dee8fe588222a3a6a67: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:26.794343 16506 leader_election.cc:302] T 05a7626e52ae43a6829feacaf1837ea5 P 9efea4678e0e4dee8fe588222a3a6a67 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cfec8b74a2342cca5723d915046f117, 9efea4678e0e4dee8fe588222a3a6a67; no voters: 8df3306d4bbe4fb1af0afb8d67c81798
I20240502 21:25:26.799053 16925 raft_consensus.cc:2781] T 05a7626e52ae43a6829feacaf1837ea5 P 9efea4678e0e4dee8fe588222a3a6a67 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:25:26.799379 16925 raft_consensus.cc:686] T 05a7626e52ae43a6829feacaf1837ea5 P 9efea4678e0e4dee8fe588222a3a6a67 [term 1 LEADER]: Becoming Leader. State: Replica: 9efea4678e0e4dee8fe588222a3a6a67, State: Running, Role: LEADER
I20240502 21:25:26.799710 16913 raft_consensus.cc:482] T e882da2f771249f8a3626d1a7a35de34 P 8df3306d4bbe4fb1af0afb8d67c81798 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:25:26.799857 16913 raft_consensus.cc:504] T e882da2f771249f8a3626d1a7a35de34 P 8df3306d4bbe4fb1af0afb8d67c81798 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cfec8b74a2342cca5723d915046f117" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 43833 } } peers { permanent_uuid: "9efea4678e0e4dee8fe588222a3a6a67" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37931 } } peers { permanent_uuid: "8df3306d4bbe4fb1af0afb8d67c81798" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37673 } }
I20240502 21:25:26.799927 16925 consensus_queue.cc:237] T 05a7626e52ae43a6829feacaf1837ea5 P 9efea4678e0e4dee8fe588222a3a6a67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9efea4678e0e4dee8fe588222a3a6a67" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37931 } } peers { permanent_uuid: "8df3306d4bbe4fb1af0afb8d67c81798" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37673 } } peers { permanent_uuid: "1cfec8b74a2342cca5723d915046f117" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 43833 } }
I20240502 21:25:26.801771 16913 leader_election.cc:288] T e882da2f771249f8a3626d1a7a35de34 P 8df3306d4bbe4fb1af0afb8d67c81798 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cfec8b74a2342cca5723d915046f117 (127.15.255.130:43833), 9efea4678e0e4dee8fe588222a3a6a67 (127.15.255.129:37931)
I20240502 21:25:26.807965 16929 raft_consensus.cc:482] T 610bb5f6731047b3b86883445a9e05bb P 1cfec8b74a2342cca5723d915046f117 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:25:26.808145 16929 raft_consensus.cc:504] T 610bb5f6731047b3b86883445a9e05bb P 1cfec8b74a2342cca5723d915046f117 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9efea4678e0e4dee8fe588222a3a6a67" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37931 } } peers { permanent_uuid: "1cfec8b74a2342cca5723d915046f117" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 43833 } } peers { permanent_uuid: "8df3306d4bbe4fb1af0afb8d67c81798" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37673 } }
I20240502 21:25:26.808856 16929 leader_election.cc:288] T 610bb5f6731047b3b86883445a9e05bb P 1cfec8b74a2342cca5723d915046f117 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9efea4678e0e4dee8fe588222a3a6a67 (127.15.255.129:37931), 8df3306d4bbe4fb1af0afb8d67c81798 (127.15.255.131:37673)
I20240502 21:25:26.809548 16570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "610bb5f6731047b3b86883445a9e05bb" candidate_uuid: "1cfec8b74a2342cca5723d915046f117" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9efea4678e0e4dee8fe588222a3a6a67" is_pre_election: true
I20240502 21:25:26.809805 16570 raft_consensus.cc:2454] T 610bb5f6731047b3b86883445a9e05bb P 9efea4678e0e4dee8fe588222a3a6a67 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cfec8b74a2342cca5723d915046f117 in term 0.
I20240502 21:25:26.810252 16635 leader_election.cc:302] T 610bb5f6731047b3b86883445a9e05bb P 1cfec8b74a2342cca5723d915046f117 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cfec8b74a2342cca5723d915046f117, 9efea4678e0e4dee8fe588222a3a6a67; no voters:
I20240502 21:25:26.810551 16929 raft_consensus.cc:2781] T 610bb5f6731047b3b86883445a9e05bb P 1cfec8b74a2342cca5723d915046f117 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:25:26.810640 16929 raft_consensus.cc:482] T 610bb5f6731047b3b86883445a9e05bb P 1cfec8b74a2342cca5723d915046f117 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:25:26.810752 16929 raft_consensus.cc:3037] T 610bb5f6731047b3b86883445a9e05bb P 1cfec8b74a2342cca5723d915046f117 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:26.813576 16929 raft_consensus.cc:504] T 610bb5f6731047b3b86883445a9e05bb P 1cfec8b74a2342cca5723d915046f117 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9efea4678e0e4dee8fe588222a3a6a67" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37931 } } peers { permanent_uuid: "1cfec8b74a2342cca5723d915046f117" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 43833 } } peers { permanent_uuid: "8df3306d4bbe4fb1af0afb8d67c81798" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37673 } }
I20240502 21:25:26.814574 16570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "610bb5f6731047b3b86883445a9e05bb" candidate_uuid: "1cfec8b74a2342cca5723d915046f117" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9efea4678e0e4dee8fe588222a3a6a67"
I20240502 21:25:26.814795 16570 raft_consensus.cc:3037] T 610bb5f6731047b3b86883445a9e05bb P 9efea4678e0e4dee8fe588222a3a6a67 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:26.816635 16925 raft_consensus.cc:482] T 62a4fb5953dc4fa7badc5e2ee1ed4261 P 9efea4678e0e4dee8fe588222a3a6a67 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:25:26.816802 16925 raft_consensus.cc:504] T 62a4fb5953dc4fa7badc5e2ee1ed4261 P 9efea4678e0e4dee8fe588222a3a6a67 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9efea4678e0e4dee8fe588222a3a6a67" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37931 } } peers { permanent_uuid: "1cfec8b74a2342cca5723d915046f117" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 43833 } } peers { permanent_uuid: "8df3306d4bbe4fb1af0afb8d67c81798" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37673 } }
I20240502 21:25:26.817588 16570 raft_consensus.cc:2454] T 610bb5f6731047b3b86883445a9e05bb P 9efea4678e0e4dee8fe588222a3a6a67 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cfec8b74a2342cca5723d915046f117 in term 1.
I20240502 21:25:26.818034 16635 leader_election.cc:302] T 610bb5f6731047b3b86883445a9e05bb P 1cfec8b74a2342cca5723d915046f117 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cfec8b74a2342cca5723d915046f117, 9efea4678e0e4dee8fe588222a3a6a67; no voters:
I20240502 21:25:26.818078 16426 catalog_manager.cc:5484] T 05a7626e52ae43a6829feacaf1837ea5 P 9efea4678e0e4dee8fe588222a3a6a67 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9efea4678e0e4dee8fe588222a3a6a67 (127.15.255.129). New cstate: current_term: 1 leader_uuid: "9efea4678e0e4dee8fe588222a3a6a67" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9efea4678e0e4dee8fe588222a3a6a67" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37931 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8df3306d4bbe4fb1af0afb8d67c81798" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1cfec8b74a2342cca5723d915046f117" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 43833 } health_report { overall_health: UNKNOWN } } }
I20240502 21:25:26.818346 16948 raft_consensus.cc:2781] T 610bb5f6731047b3b86883445a9e05bb P 1cfec8b74a2342cca5723d915046f117 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:25:26.818476 16948 raft_consensus.cc:686] T 610bb5f6731047b3b86883445a9e05bb P 1cfec8b74a2342cca5723d915046f117 [term 1 LEADER]: Becoming Leader. State: Replica: 1cfec8b74a2342cca5723d915046f117, State: Running, Role: LEADER
I20240502 21:25:26.818789 16948 consensus_queue.cc:237] T 610bb5f6731047b3b86883445a9e05bb P 1cfec8b74a2342cca5723d915046f117 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9efea4678e0e4dee8fe588222a3a6a67" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37931 } } peers { permanent_uuid: "1cfec8b74a2342cca5723d915046f117" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 43833 } } peers { permanent_uuid: "8df3306d4bbe4fb1af0afb8d67c81798" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37673 } }
I20240502 21:25:26.820688 16929 leader_election.cc:288] T 610bb5f6731047b3b86883445a9e05bb P 1cfec8b74a2342cca5723d915046f117 [CANDIDATE]: Term 1 election: Requested vote from peers 9efea4678e0e4dee8fe588222a3a6a67 (127.15.255.129:37931), 8df3306d4bbe4fb1af0afb8d67c81798 (127.15.255.131:37673)
I20240502 21:25:26.823649 16426 catalog_manager.cc:5484] T 610bb5f6731047b3b86883445a9e05bb P 1cfec8b74a2342cca5723d915046f117 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1cfec8b74a2342cca5723d915046f117 (127.15.255.130). New cstate: current_term: 1 leader_uuid: "1cfec8b74a2342cca5723d915046f117" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9efea4678e0e4dee8fe588222a3a6a67" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37931 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1cfec8b74a2342cca5723d915046f117" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 43833 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8df3306d4bbe4fb1af0afb8d67c81798" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37673 } health_report { overall_health: UNKNOWN } } }
I20240502 21:25:26.825497 16903 tablet_bootstrap.cc:492] T e882da2f771249f8a3626d1a7a35de34 P 1cfec8b74a2342cca5723d915046f117: No bootstrap required, opened a new log
I20240502 21:25:26.825711 16903 ts_tablet_manager.cc:1367] T e882da2f771249f8a3626d1a7a35de34 P 1cfec8b74a2342cca5723d915046f117: Time spent bootstrapping tablet: real 0.062s user 0.022s sys 0.021s
I20240502 21:25:26.826532 16903 raft_consensus.cc:348] T e882da2f771249f8a3626d1a7a35de34 P 1cfec8b74a2342cca5723d915046f117 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cfec8b74a2342cca5723d915046f117" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 43833 } } peers { permanent_uuid: "9efea4678e0e4dee8fe588222a3a6a67" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37931 } } peers { permanent_uuid: "8df3306d4bbe4fb1af0afb8d67c81798" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37673 } }
I20240502 21:25:26.826825 16903 raft_consensus.cc:374] T e882da2f771249f8a3626d1a7a35de34 P 1cfec8b74a2342cca5723d915046f117 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:26.826946 16916 raft_consensus.cc:482] T e882da2f771249f8a3626d1a7a35de34 P 9efea4678e0e4dee8fe588222a3a6a67 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:25:26.826961 16903 raft_consensus.cc:729] T e882da2f771249f8a3626d1a7a35de34 P 1cfec8b74a2342cca5723d915046f117 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cfec8b74a2342cca5723d915046f117, State: Initialized, Role: FOLLOWER
I20240502 21:25:26.827082 16916 raft_consensus.cc:504] T e882da2f771249f8a3626d1a7a35de34 P 9efea4678e0e4dee8fe588222a3a6a67 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cfec8b74a2342cca5723d915046f117" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 43833 } } peers { permanent_uuid: "9efea4678e0e4dee8fe588222a3a6a67" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37931 } } peers { permanent_uuid: "8df3306d4bbe4fb1af0afb8d67c81798" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37673 } }
I20240502 21:25:26.827322 16903 consensus_queue.cc:260] T e882da2f771249f8a3626d1a7a35de34 P 1cfec8b74a2342cca5723d915046f117 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cfec8b74a2342cca5723d915046f117" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 43833 } } peers { permanent_uuid: "9efea4678e0e4dee8fe588222a3a6a67" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37931 } } peers { permanent_uuid: "8df3306d4bbe4fb1af0afb8d67c81798" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37673 } }
I20240502 21:25:26.827803 16916 leader_election.cc:288] T e882da2f771249f8a3626d1a7a35de34 P 9efea4678e0e4dee8fe588222a3a6a67 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cfec8b74a2342cca5723d915046f117 (127.15.255.130:43833), 8df3306d4bbe4fb1af0afb8d67c81798 (127.15.255.131:37673)
I20240502 21:25:26.828272 16903 ts_tablet_manager.cc:1398] T e882da2f771249f8a3626d1a7a35de34 P 1cfec8b74a2342cca5723d915046f117: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 21:25:26.828532 16702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e882da2f771249f8a3626d1a7a35de34" candidate_uuid: "9efea4678e0e4dee8fe588222a3a6a67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cfec8b74a2342cca5723d915046f117" is_pre_election: true
I20240502 21:25:26.828830 16702 raft_consensus.cc:2454] T e882da2f771249f8a3626d1a7a35de34 P 1cfec8b74a2342cca5723d915046f117 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9efea4678e0e4dee8fe588222a3a6a67 in term 0.
I20240502 21:25:26.828939 16903 tablet_bootstrap.cc:492] T c06e83b5216e4138817cc58a61c02e79 P 1cfec8b74a2342cca5723d915046f117: Bootstrap starting.
I20240502 21:25:26.829373 16506 leader_election.cc:302] T e882da2f771249f8a3626d1a7a35de34 P 9efea4678e0e4dee8fe588222a3a6a67 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cfec8b74a2342cca5723d915046f117, 9efea4678e0e4dee8fe588222a3a6a67; no voters:
I20240502 21:25:26.829701 16916 raft_consensus.cc:2781] T e882da2f771249f8a3626d1a7a35de34 P 9efea4678e0e4dee8fe588222a3a6a67 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:25:26.829811 16916 raft_consensus.cc:482] T e882da2f771249f8a3626d1a7a35de34 P 9efea4678e0e4dee8fe588222a3a6a67 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:25:26.829895 16916 raft_consensus.cc:3037] T e882da2f771249f8a3626d1a7a35de34 P 9efea4678e0e4dee8fe588222a3a6a67 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:26.830772 16903 tablet_bootstrap.cc:654] T c06e83b5216e4138817cc58a61c02e79 P 1cfec8b74a2342cca5723d915046f117: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:26.832808 16916 raft_consensus.cc:504] T e882da2f771249f8a3626d1a7a35de34 P 9efea4678e0e4dee8fe588222a3a6a67 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cfec8b74a2342cca5723d915046f117" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 43833 } } peers { permanent_uuid: "9efea4678e0e4dee8fe588222a3a6a67" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37931 } } peers { permanent_uuid: "8df3306d4bbe4fb1af0afb8d67c81798" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37673 } }
I20240502 21:25:26.833638 16916 leader_election.cc:288] T e882da2f771249f8a3626d1a7a35de34 P 9efea4678e0e4dee8fe588222a3a6a67 [CANDIDATE]: Term 1 election: Requested vote from peers 1cfec8b74a2342cca5723d915046f117 (127.15.255.130:43833), 8df3306d4bbe4fb1af0afb8d67c81798 (127.15.255.131:37673)
I20240502 21:25:26.834277 16702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e882da2f771249f8a3626d1a7a35de34" candidate_uuid: "9efea4678e0e4dee8fe588222a3a6a67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cfec8b74a2342cca5723d915046f117"
I20240502 21:25:26.834502 16702 raft_consensus.cc:3037] T e882da2f771249f8a3626d1a7a35de34 P 1cfec8b74a2342cca5723d915046f117 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:26.837246 16702 raft_consensus.cc:2454] T e882da2f771249f8a3626d1a7a35de34 P 1cfec8b74a2342cca5723d915046f117 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9efea4678e0e4dee8fe588222a3a6a67 in term 1.
I20240502 21:25:26.837697 16506 leader_election.cc:302] T e882da2f771249f8a3626d1a7a35de34 P 9efea4678e0e4dee8fe588222a3a6a67 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cfec8b74a2342cca5723d915046f117, 9efea4678e0e4dee8fe588222a3a6a67; no voters:
I20240502 21:25:26.838032 16916 raft_consensus.cc:2781] T e882da2f771249f8a3626d1a7a35de34 P 9efea4678e0e4dee8fe588222a3a6a67 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:25:26.843382 16838 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e882da2f771249f8a3626d1a7a35de34" candidate_uuid: "9efea4678e0e4dee8fe588222a3a6a67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8df3306d4bbe4fb1af0afb8d67c81798" is_pre_election: true
I20240502 21:25:26.843649 16838 raft_consensus.cc:2454] T e882da2f771249f8a3626d1a7a35de34 P 8df3306d4bbe4fb1af0afb8d67c81798 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9efea4678e0e4dee8fe588222a3a6a67 in term 0.
I20240502 21:25:26.844297 16838 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e882da2f771249f8a3626d1a7a35de34" candidate_uuid: "9efea4678e0e4dee8fe588222a3a6a67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8df3306d4bbe4fb1af0afb8d67c81798"
I20240502 21:25:26.844511 16838 raft_consensus.cc:3037] T e882da2f771249f8a3626d1a7a35de34 P 8df3306d4bbe4fb1af0afb8d67c81798 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:26.847319 16838 raft_consensus.cc:2454] T e882da2f771249f8a3626d1a7a35de34 P 8df3306d4bbe4fb1af0afb8d67c81798 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9efea4678e0e4dee8fe588222a3a6a67 in term 1.
I20240502 21:25:26.850287 16963 raft_consensus.cc:482] T 1173ce275c4f40f0a0a31729310fbefd P 9efea4678e0e4dee8fe588222a3a6a67 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:25:26.850452 16916 raft_consensus.cc:686] T e882da2f771249f8a3626d1a7a35de34 P 9efea4678e0e4dee8fe588222a3a6a67 [term 1 LEADER]: Becoming Leader. State: Replica: 9efea4678e0e4dee8fe588222a3a6a67, State: Running, Role: LEADER
I20240502 21:25:26.850813 16916 consensus_queue.cc:237] T e882da2f771249f8a3626d1a7a35de34 P 9efea4678e0e4dee8fe588222a3a6a67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cfec8b74a2342cca5723d915046f117" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 43833 } } peers { permanent_uuid: "9efea4678e0e4dee8fe588222a3a6a67" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37931 } } peers { permanent_uuid: "8df3306d4bbe4fb1af0afb8d67c81798" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37673 } }
I20240502 21:25:26.851068 16913 raft_consensus.cc:482] T cb358d933789476898f74949ffa81e49 P 8df3306d4bbe4fb1af0afb8d67c81798 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:25:26.851243 16913 raft_consensus.cc:504] T cb358d933789476898f74949ffa81e49 P 8df3306d4bbe4fb1af0afb8d67c81798 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9efea4678e0e4dee8fe588222a3a6a67" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37931 } } peers { permanent_uuid: "1cfec8b74a2342cca5723d915046f117" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 43833 } } peers { permanent_uuid: "8df3306d4bbe4fb1af0afb8d67c81798" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37673 } }
I20240502 21:25:26.851833 16913 leader_election.cc:288] T cb358d933789476898f74949ffa81e49 P 8df3306d4bbe4fb1af0afb8d67c81798 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9efea4678e0e4dee8fe588222a3a6a67 (127.15.255.129:37931), 1cfec8b74a2342cca5723d915046f117 (127.15.255.130:43833)
I20240502 21:25:26.852237 16963 raft_consensus.cc:504] T 1173ce275c4f40f0a0a31729310fbefd P 9efea4678e0e4dee8fe588222a3a6a67 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9efea4678e0e4dee8fe588222a3a6a67" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37931 } } peers { permanent_uuid: "1cfec8b74a2342cca5723d915046f117" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 43833 } } peers { permanent_uuid: "8df3306d4bbe4fb1af0afb8d67c81798" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37673 } }
I20240502 21:25:26.853523 16702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1173ce275c4f40f0a0a31729310fbefd" candidate_uuid: "9efea4678e0e4dee8fe588222a3a6a67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cfec8b74a2342cca5723d915046f117" is_pre_election: true
I20240502 21:25:26.853808 16702 raft_consensus.cc:2454] T 1173ce275c4f40f0a0a31729310fbefd P 1cfec8b74a2342cca5723d915046f117 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9efea4678e0e4dee8fe588222a3a6a67 in term 0.
I20240502 21:25:26.853986 16913 raft_consensus.cc:482] T 38999367825249e1a0f5f156bd3c3b25 P 8df3306d4bbe4fb1af0afb8d67c81798 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:25:26.854138 16913 raft_consensus.cc:504] T 38999367825249e1a0f5f156bd3c3b25 P 8df3306d4bbe4fb1af0afb8d67c81798 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9efea4678e0e4dee8fe588222a3a6a67" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37931 } } peers { permanent_uuid: "1cfec8b74a2342cca5723d915046f117" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 43833 } } peers { permanent_uuid: "8df3306d4bbe4fb1af0afb8d67c81798" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37673 } }
I20240502 21:25:26.854267 16506 leader_election.cc:302] T 1173ce275c4f40f0a0a31729310fbefd P 9efea4678e0e4dee8fe588222a3a6a67 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cfec8b74a2342cca5723d915046f117, 9efea4678e0e4dee8fe588222a3a6a67; no voters:
I20240502 21:25:26.854693 16913 leader_election.cc:288] T 38999367825249e1a0f5f156bd3c3b25 P 8df3306d4bbe4fb1af0afb8d67c81798 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9efea4678e0e4dee8fe588222a3a6a67 (127.15.255.129:37931), 1cfec8b74a2342cca5723d915046f117 (127.15.255.130:43833)
I20240502 21:25:26.857267 16702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62a4fb5953dc4fa7badc5e2ee1ed4261" candidate_uuid: "9efea4678e0e4dee8fe588222a3a6a67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cfec8b74a2342cca5723d915046f117" is_pre_election: true
W20240502 21:25:26.857770 16506 leader_election.cc:341] T 62a4fb5953dc4fa7badc5e2ee1ed4261 P 9efea4678e0e4dee8fe588222a3a6a67 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cfec8b74a2342cca5723d915046f117 (127.15.255.130:43833): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:25:26.861539 16838 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62a4fb5953dc4fa7badc5e2ee1ed4261" candidate_uuid: "9efea4678e0e4dee8fe588222a3a6a67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8df3306d4bbe4fb1af0afb8d67c81798" is_pre_election: true
I20240502 21:25:26.861742 16838 raft_consensus.cc:2454] T 62a4fb5953dc4fa7badc5e2ee1ed4261 P 8df3306d4bbe4fb1af0afb8d67c81798 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9efea4678e0e4dee8fe588222a3a6a67 in term 0.
I20240502 21:25:26.862218 16504 leader_election.cc:302] T 62a4fb5953dc4fa7badc5e2ee1ed4261 P 9efea4678e0e4dee8fe588222a3a6a67 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8df3306d4bbe4fb1af0afb8d67c81798, 9efea4678e0e4dee8fe588222a3a6a67; no voters: 1cfec8b74a2342cca5723d915046f117
I20240502 21:25:26.863517 16964 raft_consensus.cc:2781] T 1173ce275c4f40f0a0a31729310fbefd P 9efea4678e0e4dee8fe588222a3a6a67 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:25:26.863768 16964 raft_consensus.cc:482] T 1173ce275c4f40f0a0a31729310fbefd P 9efea4678e0e4dee8fe588222a3a6a67 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:25:26.863974 16964 raft_consensus.cc:3037] T 1173ce275c4f40f0a0a31729310fbefd P 9efea4678e0e4dee8fe588222a3a6a67 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:26.867130 16964 raft_consensus.cc:504] T 1173ce275c4f40f0a0a31729310fbefd P 9efea4678e0e4dee8fe588222a3a6a67 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9efea4678e0e4dee8fe588222a3a6a67" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37931 } } peers { permanent_uuid: "1cfec8b74a2342cca5723d915046f117" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 43833 } } peers { permanent_uuid: "8df3306d4bbe4fb1af0afb8d67c81798" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37673 } }
I20240502 21:25:26.869197 16702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1173ce275c4f40f0a0a31729310fbefd" candidate_uuid: "9efea4678e0e4dee8fe588222a3a6a67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cfec8b74a2342cca5723d915046f117"
I20240502 21:25:26.869372 16702 raft_consensus.cc:3037] T 1173ce275c4f40f0a0a31729310fbefd P 1cfec8b74a2342cca5723d915046f117 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:26.869706 16964 leader_election.cc:288] T 1173ce275c4f40f0a0a31729310fbefd P 9efea4678e0e4dee8fe588222a3a6a67 [CANDIDATE]: Term 1 election: Requested vote from peers 1cfec8b74a2342cca5723d915046f117 (127.15.255.130:43833), 8df3306d4bbe4fb1af0afb8d67c81798 (127.15.255.131:37673)
I20240502 21:25:26.870196 16964 raft_consensus.cc:2781] T 62a4fb5953dc4fa7badc5e2ee1ed4261 P 9efea4678e0e4dee8fe588222a3a6a67 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:25:26.870496 16964 raft_consensus.cc:482] T 62a4fb5953dc4fa7badc5e2ee1ed4261 P 9efea4678e0e4dee8fe588222a3a6a67 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:25:26.870731 16964 raft_consensus.cc:3037] T 62a4fb5953dc4fa7badc5e2ee1ed4261 P 9efea4678e0e4dee8fe588222a3a6a67 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:26.872123 16702 raft_consensus.cc:2454] T 1173ce275c4f40f0a0a31729310fbefd P 1cfec8b74a2342cca5723d915046f117 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9efea4678e0e4dee8fe588222a3a6a67 in term 1.
I20240502 21:25:26.873633 16506 leader_election.cc:302] T 1173ce275c4f40f0a0a31729310fbefd P 9efea4678e0e4dee8fe588222a3a6a67 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cfec8b74a2342cca5723d915046f117, 9efea4678e0e4dee8fe588222a3a6a67; no voters:
I20240502 21:25:26.875895 16963 leader_election.cc:288] T 1173ce275c4f40f0a0a31729310fbefd P 9efea4678e0e4dee8fe588222a3a6a67 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cfec8b74a2342cca5723d915046f117 (127.15.255.130:43833), 8df3306d4bbe4fb1af0afb8d67c81798 (127.15.255.131:37673)
I20240502 21:25:26.876061 16963 raft_consensus.cc:2781] T 1173ce275c4f40f0a0a31729310fbefd P 9efea4678e0e4dee8fe588222a3a6a67 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:25:26.876196 16963 raft_consensus.cc:686] T 1173ce275c4f40f0a0a31729310fbefd P 9efea4678e0e4dee8fe588222a3a6a67 [term 1 LEADER]: Becoming Leader. State: Replica: 9efea4678e0e4dee8fe588222a3a6a67, State: Running, Role: LEADER
I20240502 21:25:26.876487 16963 consensus_queue.cc:237] T 1173ce275c4f40f0a0a31729310fbefd P 9efea4678e0e4dee8fe588222a3a6a67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9efea4678e0e4dee8fe588222a3a6a67" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37931 } } peers { permanent_uuid: "1cfec8b74a2342cca5723d915046f117" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 43833 } } peers { permanent_uuid: "8df3306d4bbe4fb1af0afb8d67c81798" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37673 } }
I20240502 21:25:26.877035 16925 leader_election.cc:288] T 62a4fb5953dc4fa7badc5e2ee1ed4261 P 9efea4678e0e4dee8fe588222a3a6a67 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cfec8b74a2342cca5723d915046f117 (127.15.255.130:43833), 8df3306d4bbe4fb1af0afb8d67c81798 (127.15.255.131:37673)
I20240502 21:25:26.876518 16964 raft_consensus.cc:504] T 62a4fb5953dc4fa7badc5e2ee1ed4261 P 9efea4678e0e4dee8fe588222a3a6a67 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9efea4678e0e4dee8fe588222a3a6a67" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37931 } } peers { permanent_uuid: "1cfec8b74a2342cca5723d915046f117" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 43833 } } peers { permanent_uuid: "8df3306d4bbe4fb1af0afb8d67c81798" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37673 } }
I20240502 21:25:26.878022 16702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62a4fb5953dc4fa7badc5e2ee1ed4261" candidate_uuid: "9efea4678e0e4dee8fe588222a3a6a67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cfec8b74a2342cca5723d915046f117"
W20240502 21:25:26.878631 16506 leader_election.cc:341] T 62a4fb5953dc4fa7badc5e2ee1ed4261 P 9efea4678e0e4dee8fe588222a3a6a67 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1cfec8b74a2342cca5723d915046f117 (127.15.255.130:43833): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:25:26.879365 16964 leader_election.cc:288] T 62a4fb5953dc4fa7badc5e2ee1ed4261 P 9efea4678e0e4dee8fe588222a3a6a67 [CANDIDATE]: Term 1 election: Requested vote from peers 1cfec8b74a2342cca5723d915046f117 (127.15.255.130:43833), 8df3306d4bbe4fb1af0afb8d67c81798 (127.15.255.131:37673)
I20240502 21:25:26.879750 16838 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "610bb5f6731047b3b86883445a9e05bb" candidate_uuid: "1cfec8b74a2342cca5723d915046f117" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8df3306d4bbe4fb1af0afb8d67c81798" is_pre_election: true
I20240502 21:25:26.879961 16838 raft_consensus.cc:2454] T 610bb5f6731047b3b86883445a9e05bb P 8df3306d4bbe4fb1af0afb8d67c81798 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cfec8b74a2342cca5723d915046f117 in term 0.
I20240502 21:25:26.880470 16838 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "610bb5f6731047b3b86883445a9e05bb" candidate_uuid: "1cfec8b74a2342cca5723d915046f117" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8df3306d4bbe4fb1af0afb8d67c81798"
I20240502 21:25:26.880713 16838 raft_consensus.cc:3037] T 610bb5f6731047b3b86883445a9e05bb P 8df3306d4bbe4fb1af0afb8d67c81798 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:26.880751 16905 tablet_bootstrap.cc:492] T 72f6a167c03d4ee6bd72d6e5793f784d P 9efea4678e0e4dee8fe588222a3a6a67: No bootstrap required, opened a new log
I20240502 21:25:26.880924 16905 ts_tablet_manager.cc:1367] T 72f6a167c03d4ee6bd72d6e5793f784d P 9efea4678e0e4dee8fe588222a3a6a67: Time spent bootstrapping tablet: real 0.089s user 0.013s sys 0.026s
I20240502 21:25:26.881686 16948 raft_consensus.cc:482] T 2b1fbeb84317494595f7262fe7857ebd P 1cfec8b74a2342cca5723d915046f117 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:25:26.881889 16948 raft_consensus.cc:504] T 2b1fbeb84317494595f7262fe7857ebd P 1cfec8b74a2342cca5723d915046f117 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cfec8b74a2342cca5723d915046f117" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 43833 } } peers { permanent_uuid: "9efea4678e0e4dee8fe588222a3a6a67" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37931 } } peers { permanent_uuid: "8df3306d4bbe4fb1af0afb8d67c81798" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37673 } }
I20240502 21:25:26.882536 16948 leader_election.cc:288] T 2b1fbeb84317494595f7262fe7857ebd P 1cfec8b74a2342cca5723d915046f117 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9efea4678e0e4dee8fe588222a3a6a67 (127.15.255.129:37931), 8df3306d4bbe4fb1af0afb8d67c81798 (127.15.255.131:37673)
I20240502 21:25:26.883291 16948 raft_consensus.cc:482] T 71d05c4f1e6f4f7186ed7b8e157b7d2f P 1cfec8b74a2342cca5723d915046f117 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:25:26.883327 16570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b1fbeb84317494595f7262fe7857ebd" candidate_uuid: "1cfec8b74a2342cca5723d915046f117" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9efea4678e0e4dee8fe588222a3a6a67" is_pre_election: true
I20240502 21:25:26.883421 16948 raft_consensus.cc:504] T 71d05c4f1e6f4f7186ed7b8e157b7d2f P 1cfec8b74a2342cca5723d915046f117 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9efea4678e0e4dee8fe588222a3a6a67" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37931 } } peers { permanent_uuid: "1cfec8b74a2342cca5723d915046f117" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 43833 } } peers { permanent_uuid: "8df3306d4bbe4fb1af0afb8d67c81798" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37673 } }
I20240502 21:25:26.883543 16570 raft_consensus.cc:2454] T 2b1fbeb84317494595f7262fe7857ebd P 9efea4678e0e4dee8fe588222a3a6a67 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cfec8b74a2342cca5723d915046f117 in term 0.
I20240502 21:25:26.883939 16838 raft_consensus.cc:2454] T 610bb5f6731047b3b86883445a9e05bb P 8df3306d4bbe4fb1af0afb8d67c81798 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cfec8b74a2342cca5723d915046f117 in term 1.
I20240502 21:25:26.884022 16948 leader_election.cc:288] T 71d05c4f1e6f4f7186ed7b8e157b7d2f P 1cfec8b74a2342cca5723d915046f117 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9efea4678e0e4dee8fe588222a3a6a67 (127.15.255.129:37931), 8df3306d4bbe4fb1af0afb8d67c81798 (127.15.255.131:37673)
I20240502 21:25:26.884112 16635 leader_election.cc:302] T 2b1fbeb84317494595f7262fe7857ebd P 1cfec8b74a2342cca5723d915046f117 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cfec8b74a2342cca5723d915046f117, 9efea4678e0e4dee8fe588222a3a6a67; no voters:
I20240502 21:25:26.884532 16913 raft_consensus.cc:482] T 71d05c4f1e6f4f7186ed7b8e157b7d2f P 8df3306d4bbe4fb1af0afb8d67c81798 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:25:26.884683 16913 raft_consensus.cc:504] T 71d05c4f1e6f4f7186ed7b8e157b7d2f P 8df3306d4bbe4fb1af0afb8d67c81798 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9efea4678e0e4dee8fe588222a3a6a67" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37931 } } peers { permanent_uuid: "1cfec8b74a2342cca5723d915046f117" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 43833 } } peers { permanent_uuid: "8df3306d4bbe4fb1af0afb8d67c81798" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37673 } }
I20240502 21:25:26.884822 16948 raft_consensus.cc:2781] T 2b1fbeb84317494595f7262fe7857ebd P 1cfec8b74a2342cca5723d915046f117 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:25:26.884914 16948 raft_consensus.cc:482] T 2b1fbeb84317494595f7262fe7857ebd P 1cfec8b74a2342cca5723d915046f117 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:25:26.884992 16948 raft_consensus.cc:3037] T 2b1fbeb84317494595f7262fe7857ebd P 1cfec8b74a2342cca5723d915046f117 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:26.885064 16570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71d05c4f1e6f4f7186ed7b8e157b7d2f" candidate_uuid: "1cfec8b74a2342cca5723d915046f117" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9efea4678e0e4dee8fe588222a3a6a67" is_pre_election: true
I20240502 21:25:26.885249 16570 raft_consensus.cc:2454] T 71d05c4f1e6f4f7186ed7b8e157b7d2f P 9efea4678e0e4dee8fe588222a3a6a67 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cfec8b74a2342cca5723d915046f117 in term 0.
I20240502 21:25:26.885291 16913 leader_election.cc:288] T 71d05c4f1e6f4f7186ed7b8e157b7d2f P 8df3306d4bbe4fb1af0afb8d67c81798 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9efea4678e0e4dee8fe588222a3a6a67 (127.15.255.129:37931), 1cfec8b74a2342cca5723d915046f117 (127.15.255.130:43833)
I20240502 21:25:26.885658 16635 leader_election.cc:302] T 71d05c4f1e6f4f7186ed7b8e157b7d2f P 1cfec8b74a2342cca5723d915046f117 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cfec8b74a2342cca5723d915046f117, 9efea4678e0e4dee8fe588222a3a6a67; no voters:
I20240502 21:25:26.885752 16838 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b1fbeb84317494595f7262fe7857ebd" candidate_uuid: "1cfec8b74a2342cca5723d915046f117" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8df3306d4bbe4fb1af0afb8d67c81798" is_pre_election: true
I20240502 21:25:26.885989 16929 raft_consensus.cc:2781] T 71d05c4f1e6f4f7186ed7b8e157b7d2f P 1cfec8b74a2342cca5723d915046f117 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:25:26.886112 16929 raft_consensus.cc:482] T 71d05c4f1e6f4f7186ed7b8e157b7d2f P 1cfec8b74a2342cca5723d915046f117 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:25:26.886207 16929 raft_consensus.cc:3037] T 71d05c4f1e6f4f7186ed7b8e157b7d2f P 1cfec8b74a2342cca5723d915046f117 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:26.886601 16838 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71d05c4f1e6f4f7186ed7b8e157b7d2f" candidate_uuid: "1cfec8b74a2342cca5723d915046f117" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8df3306d4bbe4fb1af0afb8d67c81798" is_pre_election: true
I20240502 21:25:26.886844 16838 raft_consensus.cc:2454] T 71d05c4f1e6f4f7186ed7b8e157b7d2f P 8df3306d4bbe4fb1af0afb8d67c81798 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cfec8b74a2342cca5723d915046f117 in term 0.
I20240502 21:25:26.887758 16948 raft_consensus.cc:504] T 2b1fbeb84317494595f7262fe7857ebd P 1cfec8b74a2342cca5723d915046f117 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cfec8b74a2342cca5723d915046f117" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 43833 } } peers { permanent_uuid: "9efea4678e0e4dee8fe588222a3a6a67" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37931 } } peers { permanent_uuid: "8df3306d4bbe4fb1af0afb8d67c81798" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37673 } }
W20240502 21:25:26.888518 16636 leader_election.cc:341] T 2b1fbeb84317494595f7262fe7857ebd P 1cfec8b74a2342cca5723d915046f117 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8df3306d4bbe4fb1af0afb8d67c81798 (127.15.255.131:37673): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:25:26.889156 16929 raft_consensus.cc:504] T 71d05c4f1e6f4f7186ed7b8e157b7d2f P 1cfec8b74a2342cca5723d915046f117 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9efea4678e0e4dee8fe588222a3a6a67" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37931 } } peers { permanent_uuid: "1cfec8b74a2342cca5723d915046f117" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 43833 } } peers { permanent_uuid: "8df3306d4bbe4fb1af0afb8d67c81798" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37673 } }
I20240502 21:25:26.889420 16570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b1fbeb84317494595f7262fe7857ebd" candidate_uuid: "1cfec8b74a2342cca5723d915046f117" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9efea4678e0e4dee8fe588222a3a6a67"
I20240502 21:25:26.889583 16570 raft_consensus.cc:3037] T 2b1fbeb84317494595f7262fe7857ebd P 9efea4678e0e4dee8fe588222a3a6a67 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:26.889772 16929 leader_election.cc:288] T 71d05c4f1e6f4f7186ed7b8e157b7d2f P 1cfec8b74a2342cca5723d915046f117 [CANDIDATE]: Term 1 election: Requested vote from peers 9efea4678e0e4dee8fe588222a3a6a67 (127.15.255.129:37931), 8df3306d4bbe4fb1af0afb8d67c81798 (127.15.255.131:37673)
I20240502 21:25:26.890411 16838 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71d05c4f1e6f4f7186ed7b8e157b7d2f" candidate_uuid: "1cfec8b74a2342cca5723d915046f117" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8df3306d4bbe4fb1af0afb8d67c81798"
I20240502 21:25:26.890558 16838 raft_consensus.cc:3037] T 71d05c4f1e6f4f7186ed7b8e157b7d2f P 8df3306d4bbe4fb1af0afb8d67c81798 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:26.890525 16569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71d05c4f1e6f4f7186ed7b8e157b7d2f" candidate_uuid: "1cfec8b74a2342cca5723d915046f117" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9efea4678e0e4dee8fe588222a3a6a67"
I20240502 21:25:26.890590 16905 raft_consensus.cc:348] T 72f6a167c03d4ee6bd72d6e5793f784d P 9efea4678e0e4dee8fe588222a3a6a67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9efea4678e0e4dee8fe588222a3a6a67" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37931 } } peers { permanent_uuid: "1cfec8b74a2342cca5723d915046f117" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 43833 } } peers { permanent_uuid: "8df3306d4bbe4fb1af0afb8d67c81798" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37673 } }
I20240502 21:25:26.890874 16569 raft_consensus.cc:3037] T 71d05c4f1e6f4f7186ed7b8e157b7d2f P 9efea4678e0e4dee8fe588222a3a6a67 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:26.890877 16905 raft_consensus.cc:374] T 72f6a167c03d4ee6bd72d6e5793f784d P 9efea4678e0e4dee8fe588222a3a6a67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:26.891009 16905 raft_consensus.cc:729] T 72f6a167c03d4ee6bd72d6e5793f784d P 9efea4678e0e4dee8fe588222a3a6a67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9efea4678e0e4dee8fe588222a3a6a67, State: Initialized, Role: FOLLOWER
I20240502 21:25:26.891323 16905 consensus_queue.cc:260] T 72f6a167c03d4ee6bd72d6e5793f784d P 9efea4678e0e4dee8fe588222a3a6a67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9efea4678e0e4dee8fe588222a3a6a67" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37931 } } peers { permanent_uuid: "1cfec8b74a2342cca5723d915046f117" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 43833 } } peers { permanent_uuid: "8df3306d4bbe4fb1af0afb8d67c81798" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37673 } }
I20240502 21:25:26.891896 16905 ts_tablet_manager.cc:1398] T 72f6a167c03d4ee6bd72d6e5793f784d P 9efea4678e0e4dee8fe588222a3a6a67: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20240502 21:25:26.892329 16905 tablet_bootstrap.cc:492] T 4cfc1f2146dd449c87a46196e25280c2 P 9efea4678e0e4dee8fe588222a3a6a67: Bootstrap starting.
I20240502 21:25:26.892824 16948 leader_election.cc:288] T 2b1fbeb84317494595f7262fe7857ebd P 1cfec8b74a2342cca5723d915046f117 [CANDIDATE]: Term 1 election: Requested vote from peers 9efea4678e0e4dee8fe588222a3a6a67 (127.15.255.129:37931), 8df3306d4bbe4fb1af0afb8d67c81798 (127.15.255.131:37673)
I20240502 21:25:26.893045 16838 raft_consensus.cc:2454] T 71d05c4f1e6f4f7186ed7b8e157b7d2f P 8df3306d4bbe4fb1af0afb8d67c81798 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cfec8b74a2342cca5723d915046f117 in term 1.
I20240502 21:25:26.893069 16570 raft_consensus.cc:2454] T 2b1fbeb84317494595f7262fe7857ebd P 9efea4678e0e4dee8fe588222a3a6a67 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cfec8b74a2342cca5723d915046f117 in term 1.
I20240502 21:25:26.893364 16636 leader_election.cc:302] T 71d05c4f1e6f4f7186ed7b8e157b7d2f P 1cfec8b74a2342cca5723d915046f117 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cfec8b74a2342cca5723d915046f117, 8df3306d4bbe4fb1af0afb8d67c81798; no voters:
I20240502 21:25:26.893586 16635 leader_election.cc:302] T 2b1fbeb84317494595f7262fe7857ebd P 1cfec8b74a2342cca5723d915046f117 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cfec8b74a2342cca5723d915046f117, 9efea4678e0e4dee8fe588222a3a6a67; no voters:
I20240502 21:25:26.893987 16838 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b1fbeb84317494595f7262fe7857ebd" candidate_uuid: "1cfec8b74a2342cca5723d915046f117" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8df3306d4bbe4fb1af0afb8d67c81798"
W20240502 21:25:26.894466 16636 leader_election.cc:341] T 2b1fbeb84317494595f7262fe7857ebd P 1cfec8b74a2342cca5723d915046f117 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8df3306d4bbe4fb1af0afb8d67c81798 (127.15.255.131:37673): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:25:26.894737 16929 raft_consensus.cc:2781] T 71d05c4f1e6f4f7186ed7b8e157b7d2f P 1cfec8b74a2342cca5723d915046f117 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:25:26.894870 16929 raft_consensus.cc:686] T 71d05c4f1e6f4f7186ed7b8e157b7d2f P 1cfec8b74a2342cca5723d915046f117 [term 1 LEADER]: Becoming Leader. State: Replica: 1cfec8b74a2342cca5723d915046f117, State: Running, Role: LEADER
I20240502 21:25:26.895198 16907 raft_consensus.cc:2781] T 2b1fbeb84317494595f7262fe7857ebd P 1cfec8b74a2342cca5723d915046f117 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:25:26.895145 16929 consensus_queue.cc:237] T 71d05c4f1e6f4f7186ed7b8e157b7d2f P 1cfec8b74a2342cca5723d915046f117 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9efea4678e0e4dee8fe588222a3a6a67" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37931 } } peers { permanent_uuid: "1cfec8b74a2342cca5723d915046f117" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 43833 } } peers { permanent_uuid: "8df3306d4bbe4fb1af0afb8d67c81798" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37673 } }
I20240502 21:25:26.898196 16907 raft_consensus.cc:686] T 2b1fbeb84317494595f7262fe7857ebd P 1cfec8b74a2342cca5723d915046f117 [term 1 LEADER]: Becoming Leader. State: Replica: 1cfec8b74a2342cca5723d915046f117, State: Running, Role: LEADER
I20240502 21:25:26.898507 16907 consensus_queue.cc:237] T 2b1fbeb84317494595f7262fe7857ebd P 1cfec8b74a2342cca5723d915046f117 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cfec8b74a2342cca5723d915046f117" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 43833 } } peers { permanent_uuid: "9efea4678e0e4dee8fe588222a3a6a67" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37931 } } peers { permanent_uuid: "8df3306d4bbe4fb1af0afb8d67c81798" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37673 } }
I20240502 21:25:26.900990 16903 tablet_bootstrap.cc:492] T c06e83b5216e4138817cc58a61c02e79 P 1cfec8b74a2342cca5723d915046f117: No bootstrap required, opened a new log
I20240502 21:25:26.901196 16903 ts_tablet_manager.cc:1367] T c06e83b5216e4138817cc58a61c02e79 P 1cfec8b74a2342cca5723d915046f117: Time spent bootstrapping tablet: real 0.072s user 0.006s sys 0.026s
I20240502 21:25:26.901998 16903 raft_consensus.cc:348] T c06e83b5216e4138817cc58a61c02e79 P 1cfec8b74a2342cca5723d915046f117 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cfec8b74a2342cca5723d915046f117" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 43833 } } peers { permanent_uuid: "9efea4678e0e4dee8fe588222a3a6a67" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37931 } } peers { permanent_uuid: "8df3306d4bbe4fb1af0afb8d67c81798" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37673 } }
I20240502 21:25:26.902261 16903 raft_consensus.cc:374] T c06e83b5216e4138817cc58a61c02e79 P 1cfec8b74a2342cca5723d915046f117 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:26.902400 16903 raft_consensus.cc:729] T c06e83b5216e4138817cc58a61c02e79 P 1cfec8b74a2342cca5723d915046f117 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cfec8b74a2342cca5723d915046f117, State: Initialized, Role: FOLLOWER
I20240502 21:25:26.902722 16903 consensus_queue.cc:260] T c06e83b5216e4138817cc58a61c02e79 P 1cfec8b74a2342cca5723d915046f117 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cfec8b74a2342cca5723d915046f117" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 43833 } } peers { permanent_uuid: "9efea4678e0e4dee8fe588222a3a6a67" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37931 } } peers { permanent_uuid: "8df3306d4bbe4fb1af0afb8d67c81798" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37673 } }
I20240502 21:25:26.903416 16903 ts_tablet_manager.cc:1398] T c06e83b5216e4138817cc58a61c02e79 P 1cfec8b74a2342cca5723d915046f117: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 21:25:26.903730 16903 tablet_bootstrap.cc:492] T 62a4fb5953dc4fa7badc5e2ee1ed4261 P 1cfec8b74a2342cca5723d915046f117: Bootstrap starting.
I20240502 21:25:26.903929 16569 raft_consensus.cc:2454] T 71d05c4f1e6f4f7186ed7b8e157b7d2f P 9efea4678e0e4dee8fe588222a3a6a67 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cfec8b74a2342cca5723d915046f117 in term 1.
I20240502 21:25:26.905498 16903 tablet_bootstrap.cc:654] T 62a4fb5953dc4fa7badc5e2ee1ed4261 P 1cfec8b74a2342cca5723d915046f117: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:26.905812 16701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb358d933789476898f74949ffa81e49" candidate_uuid: "8df3306d4bbe4fb1af0afb8d67c81798" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cfec8b74a2342cca5723d915046f117" is_pre_election: true
I20240502 21:25:26.906134 16701 raft_consensus.cc:2454] T cb358d933789476898f74949ffa81e49 P 1cfec8b74a2342cca5723d915046f117 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8df3306d4bbe4fb1af0afb8d67c81798 in term 0.
I20240502 21:25:26.906466 16700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38999367825249e1a0f5f156bd3c3b25" candidate_uuid: "8df3306d4bbe4fb1af0afb8d67c81798" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cfec8b74a2342cca5723d915046f117" is_pre_election: true
I20240502 21:25:26.906466 16699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71d05c4f1e6f4f7186ed7b8e157b7d2f" candidate_uuid: "8df3306d4bbe4fb1af0afb8d67c81798" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cfec8b74a2342cca5723d915046f117" is_pre_election: true
I20240502 21:25:26.906762 16771 leader_election.cc:302] T cb358d933789476898f74949ffa81e49 P 8df3306d4bbe4fb1af0afb8d67c81798 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cfec8b74a2342cca5723d915046f117, 8df3306d4bbe4fb1af0afb8d67c81798; no voters:
I20240502 21:25:26.906140 16702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e882da2f771249f8a3626d1a7a35de34" candidate_uuid: "8df3306d4bbe4fb1af0afb8d67c81798" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cfec8b74a2342cca5723d915046f117" is_pre_election: true
I20240502 21:25:26.907320 16702 raft_consensus.cc:2379] T e882da2f771249f8a3626d1a7a35de34 P 1cfec8b74a2342cca5723d915046f117 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8df3306d4bbe4fb1af0afb8d67c81798 in current term 1: Already voted for candidate 9efea4678e0e4dee8fe588222a3a6a67 in this term.
W20240502 21:25:26.907317 16771 leader_election.cc:341] T 38999367825249e1a0f5f156bd3c3b25 P 8df3306d4bbe4fb1af0afb8d67c81798 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cfec8b74a2342cca5723d915046f117 (127.15.255.130:43833): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:25:26.907897 16913 raft_consensus.cc:2781] T cb358d933789476898f74949ffa81e49 P 8df3306d4bbe4fb1af0afb8d67c81798 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:25:26.908010 16913 raft_consensus.cc:482] T cb358d933789476898f74949ffa81e49 P 8df3306d4bbe4fb1af0afb8d67c81798 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:25:26.908133 16913 raft_consensus.cc:3037] T cb358d933789476898f74949ffa81e49 P 8df3306d4bbe4fb1af0afb8d67c81798 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:26.908651 16905 tablet_bootstrap.cc:654] T 4cfc1f2146dd449c87a46196e25280c2 P 9efea4678e0e4dee8fe588222a3a6a67: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:26.911162 16913 raft_consensus.cc:504] T cb358d933789476898f74949ffa81e49 P 8df3306d4bbe4fb1af0afb8d67c81798 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9efea4678e0e4dee8fe588222a3a6a67" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37931 } } peers { permanent_uuid: "1cfec8b74a2342cca5723d915046f117" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 43833 } } peers { permanent_uuid: "8df3306d4bbe4fb1af0afb8d67c81798" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37673 } }
I20240502 21:25:26.912066 16913 leader_election.cc:288] T cb358d933789476898f74949ffa81e49 P 8df3306d4bbe4fb1af0afb8d67c81798 [CANDIDATE]: Term 1 election: Requested vote from peers 9efea4678e0e4dee8fe588222a3a6a67 (127.15.255.129:37931), 1cfec8b74a2342cca5723d915046f117 (127.15.255.130:43833)
I20240502 21:25:26.912243 16913 raft_consensus.cc:482] T 72f6a167c03d4ee6bd72d6e5793f784d P 8df3306d4bbe4fb1af0afb8d67c81798 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:25:26.912389 16913 raft_consensus.cc:504] T 72f6a167c03d4ee6bd72d6e5793f784d P 8df3306d4bbe4fb1af0afb8d67c81798 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9efea4678e0e4dee8fe588222a3a6a67" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37931 } } peers { permanent_uuid: "1cfec8b74a2342cca5723d915046f117" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 43833 } } peers { permanent_uuid: "8df3306d4bbe4fb1af0afb8d67c81798" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37673 } }
I20240502 21:25:26.913353 16913 leader_election.cc:288] T 72f6a167c03d4ee6bd72d6e5793f784d P 8df3306d4bbe4fb1af0afb8d67c81798 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9efea4678e0e4dee8fe588222a3a6a67 (127.15.255.129:37931), 1cfec8b74a2342cca5723d915046f117 (127.15.255.130:43833)
I20240502 21:25:26.913717 16702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb358d933789476898f74949ffa81e49" candidate_uuid: "8df3306d4bbe4fb1af0afb8d67c81798" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cfec8b74a2342cca5723d915046f117"
I20240502 21:25:26.913892 16702 raft_consensus.cc:3037] T cb358d933789476898f74949ffa81e49 P 1cfec8b74a2342cca5723d915046f117 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:26.916395 16702 raft_consensus.cc:2454] T cb358d933789476898f74949ffa81e49 P 1cfec8b74a2342cca5723d915046f117 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8df3306d4bbe4fb1af0afb8d67c81798 in term 1.
I20240502 21:25:26.916713 16699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72f6a167c03d4ee6bd72d6e5793f784d" candidate_uuid: "8df3306d4bbe4fb1af0afb8d67c81798" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cfec8b74a2342cca5723d915046f117" is_pre_election: true
I20240502 21:25:26.916931 16699 raft_consensus.cc:2454] T 72f6a167c03d4ee6bd72d6e5793f784d P 1cfec8b74a2342cca5723d915046f117 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8df3306d4bbe4fb1af0afb8d67c81798 in term 0.
W20240502 21:25:26.916177 16504 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.255.131:37673, user_credentials={real_user=slave}} blocked reactor thread for 53944.1us
I20240502 21:25:26.917759 16838 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1173ce275c4f40f0a0a31729310fbefd" candidate_uuid: "9efea4678e0e4dee8fe588222a3a6a67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8df3306d4bbe4fb1af0afb8d67c81798"
I20240502 21:25:26.917965 16838 raft_consensus.cc:3037] T 1173ce275c4f40f0a0a31729310fbefd P 8df3306d4bbe4fb1af0afb8d67c81798 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:26.918045 16771 leader_election.cc:302] T cb358d933789476898f74949ffa81e49 P 8df3306d4bbe4fb1af0afb8d67c81798 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cfec8b74a2342cca5723d915046f117, 8df3306d4bbe4fb1af0afb8d67c81798; no voters:
I20240502 21:25:26.918298 16771 leader_election.cc:302] T 72f6a167c03d4ee6bd72d6e5793f784d P 8df3306d4bbe4fb1af0afb8d67c81798 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cfec8b74a2342cca5723d915046f117, 8df3306d4bbe4fb1af0afb8d67c81798; no voters:
I20240502 21:25:26.918718 16837 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1173ce275c4f40f0a0a31729310fbefd" candidate_uuid: "9efea4678e0e4dee8fe588222a3a6a67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8df3306d4bbe4fb1af0afb8d67c81798" is_pre_election: true
I20240502 21:25:26.919435 16913 raft_consensus.cc:2781] T cb358d933789476898f74949ffa81e49 P 8df3306d4bbe4fb1af0afb8d67c81798 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:25:26.919147 16426 catalog_manager.cc:5484] T e882da2f771249f8a3626d1a7a35de34 P 9efea4678e0e4dee8fe588222a3a6a67 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9efea4678e0e4dee8fe588222a3a6a67 (127.15.255.129). New cstate: current_term: 1 leader_uuid: "9efea4678e0e4dee8fe588222a3a6a67" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cfec8b74a2342cca5723d915046f117" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 43833 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9efea4678e0e4dee8fe588222a3a6a67" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37931 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8df3306d4bbe4fb1af0afb8d67c81798" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37673 } health_report { overall_health: UNKNOWN } } }
I20240502 21:25:26.920774 16838 raft_consensus.cc:2454] T 1173ce275c4f40f0a0a31729310fbefd P 8df3306d4bbe4fb1af0afb8d67c81798 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9efea4678e0e4dee8fe588222a3a6a67 in term 1.
I20240502 21:25:26.923439 16838 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62a4fb5953dc4fa7badc5e2ee1ed4261" candidate_uuid: "9efea4678e0e4dee8fe588222a3a6a67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8df3306d4bbe4fb1af0afb8d67c81798"
I20240502 21:25:26.923645 16838 raft_consensus.cc:3037] T 62a4fb5953dc4fa7badc5e2ee1ed4261 P 8df3306d4bbe4fb1af0afb8d67c81798 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:26.923869 16905 tablet_bootstrap.cc:492] T 4cfc1f2146dd449c87a46196e25280c2 P 9efea4678e0e4dee8fe588222a3a6a67: No bootstrap required, opened a new log
I20240502 21:25:26.925267 16426 catalog_manager.cc:5484] T 1173ce275c4f40f0a0a31729310fbefd P 9efea4678e0e4dee8fe588222a3a6a67 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9efea4678e0e4dee8fe588222a3a6a67 (127.15.255.129). New cstate: current_term: 1 leader_uuid: "9efea4678e0e4dee8fe588222a3a6a67" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9efea4678e0e4dee8fe588222a3a6a67" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37931 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1cfec8b74a2342cca5723d915046f117" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 43833 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8df3306d4bbe4fb1af0afb8d67c81798" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37673 } health_report { overall_health: UNKNOWN } } }
I20240502 21:25:26.926568 16905 ts_tablet_manager.cc:1367] T 4cfc1f2146dd449c87a46196e25280c2 P 9efea4678e0e4dee8fe588222a3a6a67: Time spent bootstrapping tablet: real 0.034s user 0.004s sys 0.000s
I20240502 21:25:26.926702 16838 raft_consensus.cc:2454] T 62a4fb5953dc4fa7badc5e2ee1ed4261 P 8df3306d4bbe4fb1af0afb8d67c81798 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9efea4678e0e4dee8fe588222a3a6a67 in term 1.
I20240502 21:25:26.927148 16504 leader_election.cc:302] T 62a4fb5953dc4fa7badc5e2ee1ed4261 P 9efea4678e0e4dee8fe588222a3a6a67 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8df3306d4bbe4fb1af0afb8d67c81798, 9efea4678e0e4dee8fe588222a3a6a67; no voters: 1cfec8b74a2342cca5723d915046f117
I20240502 21:25:26.927948 16905 raft_consensus.cc:348] T 4cfc1f2146dd449c87a46196e25280c2 P 9efea4678e0e4dee8fe588222a3a6a67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cfec8b74a2342cca5723d915046f117" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 43833 } } peers { permanent_uuid: "9efea4678e0e4dee8fe588222a3a6a67" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37931 } } peers { permanent_uuid: "8df3306d4bbe4fb1af0afb8d67c81798" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37673 } }
I20240502 21:25:26.928602 16905 raft_consensus.cc:374] T 4cfc1f2146dd449c87a46196e25280c2 P 9efea4678e0e4dee8fe588222a3a6a67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:26.928759 16905 raft_consensus.cc:729] T 4cfc1f2146dd449c87a46196e25280c2 P 9efea4678e0e4dee8fe588222a3a6a67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9efea4678e0e4dee8fe588222a3a6a67, State: Initialized, Role: FOLLOWER
I20240502 21:25:26.928500 16963 raft_consensus.cc:2781] T 62a4fb5953dc4fa7badc5e2ee1ed4261 P 9efea4678e0e4dee8fe588222a3a6a67 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:25:26.929093 16963 raft_consensus.cc:686] T 62a4fb5953dc4fa7badc5e2ee1ed4261 P 9efea4678e0e4dee8fe588222a3a6a67 [term 1 LEADER]: Becoming Leader. State: Replica: 9efea4678e0e4dee8fe588222a3a6a67, State: Running, Role: LEADER
I20240502 21:25:26.929391 16963 consensus_queue.cc:237] T 62a4fb5953dc4fa7badc5e2ee1ed4261 P 9efea4678e0e4dee8fe588222a3a6a67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9efea4678e0e4dee8fe588222a3a6a67" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37931 } } peers { permanent_uuid: "1cfec8b74a2342cca5723d915046f117" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 43833 } } peers { permanent_uuid: "8df3306d4bbe4fb1af0afb8d67c81798" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37673 } }
I20240502 21:25:26.930444 16913 raft_consensus.cc:686] T cb358d933789476898f74949ffa81e49 P 8df3306d4bbe4fb1af0afb8d67c81798 [term 1 LEADER]: Becoming Leader. State: Replica: 8df3306d4bbe4fb1af0afb8d67c81798, State: Running, Role: LEADER
I20240502 21:25:26.930583 16974 raft_consensus.cc:2781] T 72f6a167c03d4ee6bd72d6e5793f784d P 8df3306d4bbe4fb1af0afb8d67c81798 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:25:26.930678 16974 raft_consensus.cc:482] T 72f6a167c03d4ee6bd72d6e5793f784d P 8df3306d4bbe4fb1af0afb8d67c81798 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:25:26.930804 16974 raft_consensus.cc:3037] T 72f6a167c03d4ee6bd72d6e5793f784d P 8df3306d4bbe4fb1af0afb8d67c81798 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:26.930806 16569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e882da2f771249f8a3626d1a7a35de34" candidate_uuid: "8df3306d4bbe4fb1af0afb8d67c81798" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9efea4678e0e4dee8fe588222a3a6a67" is_pre_election: true
I20240502 21:25:26.931177 16913 consensus_queue.cc:237] T cb358d933789476898f74949ffa81e49 P 8df3306d4bbe4fb1af0afb8d67c81798 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9efea4678e0e4dee8fe588222a3a6a67" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37931 } } peers { permanent_uuid: "1cfec8b74a2342cca5723d915046f117" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 43833 } } peers { permanent_uuid: "8df3306d4bbe4fb1af0afb8d67c81798" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37673 } }
I20240502 21:25:26.931700 16569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb358d933789476898f74949ffa81e49" candidate_uuid: "8df3306d4bbe4fb1af0afb8d67c81798" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9efea4678e0e4dee8fe588222a3a6a67" is_pre_election: true
I20240502 21:25:26.931893 16569 raft_consensus.cc:2454] T cb358d933789476898f74949ffa81e49 P 9efea4678e0e4dee8fe588222a3a6a67 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8df3306d4bbe4fb1af0afb8d67c81798 in term 0.
I20240502 21:25:26.932358 16569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38999367825249e1a0f5f156bd3c3b25" candidate_uuid: "8df3306d4bbe4fb1af0afb8d67c81798" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9efea4678e0e4dee8fe588222a3a6a67" is_pre_election: true
I20240502 21:25:26.932559 16569 raft_consensus.cc:2454] T 38999367825249e1a0f5f156bd3c3b25 P 9efea4678e0e4dee8fe588222a3a6a67 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8df3306d4bbe4fb1af0afb8d67c81798 in term 0.
I20240502 21:25:26.933033 16569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71d05c4f1e6f4f7186ed7b8e157b7d2f" candidate_uuid: "8df3306d4bbe4fb1af0afb8d67c81798" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9efea4678e0e4dee8fe588222a3a6a67" is_pre_election: true
I20240502 21:25:26.933257 16569 raft_consensus.cc:2379] T 71d05c4f1e6f4f7186ed7b8e157b7d2f P 9efea4678e0e4dee8fe588222a3a6a67 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8df3306d4bbe4fb1af0afb8d67c81798 in current term 1: Already voted for candidate 1cfec8b74a2342cca5723d915046f117 in this term.
I20240502 21:25:26.933738 16569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb358d933789476898f74949ffa81e49" candidate_uuid: "8df3306d4bbe4fb1af0afb8d67c81798" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9efea4678e0e4dee8fe588222a3a6a67"
I20240502 21:25:26.933912 16569 raft_consensus.cc:3037] T cb358d933789476898f74949ffa81e49 P 9efea4678e0e4dee8fe588222a3a6a67 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:26.933882 16570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72f6a167c03d4ee6bd72d6e5793f784d" candidate_uuid: "8df3306d4bbe4fb1af0afb8d67c81798" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9efea4678e0e4dee8fe588222a3a6a67" is_pre_election: true
I20240502 21:25:26.934063 16570 raft_consensus.cc:2454] T 72f6a167c03d4ee6bd72d6e5793f784d P 9efea4678e0e4dee8fe588222a3a6a67 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8df3306d4bbe4fb1af0afb8d67c81798 in term 0.
I20240502 21:25:26.934484 16767 leader_election.cc:302] T e882da2f771249f8a3626d1a7a35de34 P 8df3306d4bbe4fb1af0afb8d67c81798 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8df3306d4bbe4fb1af0afb8d67c81798; no voters: 1cfec8b74a2342cca5723d915046f117, 9efea4678e0e4dee8fe588222a3a6a67
I20240502 21:25:26.934754 16973 raft_consensus.cc:2726] T e882da2f771249f8a3626d1a7a35de34 P 8df3306d4bbe4fb1af0afb8d67c81798 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 21:25:26.935729 16767 leader_election.cc:302] T 38999367825249e1a0f5f156bd3c3b25 P 8df3306d4bbe4fb1af0afb8d67c81798 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8df3306d4bbe4fb1af0afb8d67c81798, 9efea4678e0e4dee8fe588222a3a6a67; no voters: 1cfec8b74a2342cca5723d915046f117
I20240502 21:25:26.935858 16974 raft_consensus.cc:504] T 72f6a167c03d4ee6bd72d6e5793f784d P 8df3306d4bbe4fb1af0afb8d67c81798 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9efea4678e0e4dee8fe588222a3a6a67" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37931 } } peers { permanent_uuid: "1cfec8b74a2342cca5723d915046f117" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 43833 } } peers { permanent_uuid: "8df3306d4bbe4fb1af0afb8d67c81798" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37673 } }
I20240502 21:25:26.936154 16973 raft_consensus.cc:2781] T 38999367825249e1a0f5f156bd3c3b25 P 8df3306d4bbe4fb1af0afb8d67c81798 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:25:26.936261 16973 raft_consensus.cc:482] T 38999367825249e1a0f5f156bd3c3b25 P 8df3306d4bbe4fb1af0afb8d67c81798 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:25:26.936367 16973 raft_consensus.cc:3037] T 38999367825249e1a0f5f156bd3c3b25 P 8df3306d4bbe4fb1af0afb8d67c81798 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:26.936558 16767 leader_election.cc:302] T 71d05c4f1e6f4f7186ed7b8e157b7d2f P 8df3306d4bbe4fb1af0afb8d67c81798 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8df3306d4bbe4fb1af0afb8d67c81798; no voters: 1cfec8b74a2342cca5723d915046f117, 9efea4678e0e4dee8fe588222a3a6a67
I20240502 21:25:26.936400 16569 raft_consensus.cc:2454] T cb358d933789476898f74949ffa81e49 P 9efea4678e0e4dee8fe588222a3a6a67 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8df3306d4bbe4fb1af0afb8d67c81798 in term 1.
I20240502 21:25:26.937009 16699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72f6a167c03d4ee6bd72d6e5793f784d" candidate_uuid: "8df3306d4bbe4fb1af0afb8d67c81798" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cfec8b74a2342cca5723d915046f117"
I20240502 21:25:26.937202 16699 raft_consensus.cc:3037] T 72f6a167c03d4ee6bd72d6e5793f784d P 1cfec8b74a2342cca5723d915046f117 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:26.937528 16904 tablet_bootstrap.cc:492] T 05a7626e52ae43a6829feacaf1837ea5 P 8df3306d4bbe4fb1af0afb8d67c81798: No bootstrap required, opened a new log
I20240502 21:25:26.937681 16904 ts_tablet_manager.cc:1367] T 05a7626e52ae43a6829feacaf1837ea5 P 8df3306d4bbe4fb1af0afb8d67c81798: Time spent bootstrapping tablet: real 0.146s user 0.046s sys 0.077s
I20240502 21:25:26.938484 16904 raft_consensus.cc:348] T 05a7626e52ae43a6829feacaf1837ea5 P 8df3306d4bbe4fb1af0afb8d67c81798 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9efea4678e0e4dee8fe588222a3a6a67" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37931 } } peers { permanent_uuid: "8df3306d4bbe4fb1af0afb8d67c81798" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37673 } } peers { permanent_uuid: "1cfec8b74a2342cca5723d915046f117" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 43833 } }
I20240502 21:25:26.938696 16904 raft_consensus.cc:374] T 05a7626e52ae43a6829feacaf1837ea5 P 8df3306d4bbe4fb1af0afb8d67c81798 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:26.938808 16904 raft_consensus.cc:729] T 05a7626e52ae43a6829feacaf1837ea5 P 8df3306d4bbe4fb1af0afb8d67c81798 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8df3306d4bbe4fb1af0afb8d67c81798, State: Initialized, Role: FOLLOWER
I20240502 21:25:26.939101 16904 consensus_queue.cc:260] T 05a7626e52ae43a6829feacaf1837ea5 P 8df3306d4bbe4fb1af0afb8d67c81798 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9efea4678e0e4dee8fe588222a3a6a67" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37931 } } peers { permanent_uuid: "8df3306d4bbe4fb1af0afb8d67c81798" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37673 } } peers { permanent_uuid: "1cfec8b74a2342cca5723d915046f117" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 43833 } }
I20240502 21:25:26.939719 16699 raft_consensus.cc:2454] T 72f6a167c03d4ee6bd72d6e5793f784d P 1cfec8b74a2342cca5723d915046f117 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8df3306d4bbe4fb1af0afb8d67c81798 in term 1.
I20240502 21:25:26.940157 16771 leader_election.cc:302] T 72f6a167c03d4ee6bd72d6e5793f784d P 8df3306d4bbe4fb1af0afb8d67c81798 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cfec8b74a2342cca5723d915046f117, 8df3306d4bbe4fb1af0afb8d67c81798; no voters:
I20240502 21:25:26.940480 16426 catalog_manager.cc:5484] T cb358d933789476898f74949ffa81e49 P 8df3306d4bbe4fb1af0afb8d67c81798 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8df3306d4bbe4fb1af0afb8d67c81798 (127.15.255.131). New cstate: current_term: 1 leader_uuid: "8df3306d4bbe4fb1af0afb8d67c81798" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9efea4678e0e4dee8fe588222a3a6a67" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37931 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1cfec8b74a2342cca5723d915046f117" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 43833 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8df3306d4bbe4fb1af0afb8d67c81798" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37673 } health_report { overall_health: HEALTHY } } }
I20240502 21:25:26.941015 16978 raft_consensus.cc:2781] T 72f6a167c03d4ee6bd72d6e5793f784d P 8df3306d4bbe4fb1af0afb8d67c81798 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:25:26.941190 16978 raft_consensus.cc:686] T 72f6a167c03d4ee6bd72d6e5793f784d P 8df3306d4bbe4fb1af0afb8d67c81798 [term 1 LEADER]: Becoming Leader. State: Replica: 8df3306d4bbe4fb1af0afb8d67c81798, State: Running, Role: LEADER
I20240502 21:25:26.941247 16905 consensus_queue.cc:260] T 4cfc1f2146dd449c87a46196e25280c2 P 9efea4678e0e4dee8fe588222a3a6a67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cfec8b74a2342cca5723d915046f117" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 43833 } } peers { permanent_uuid: "9efea4678e0e4dee8fe588222a3a6a67" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37931 } } peers { permanent_uuid: "8df3306d4bbe4fb1af0afb8d67c81798" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37673 } }
I20240502 21:25:26.942008 16905 ts_tablet_manager.cc:1398] T 4cfc1f2146dd449c87a46196e25280c2 P 9efea4678e0e4dee8fe588222a3a6a67: Time spent starting tablet: real 0.015s user 0.001s sys 0.000s
I20240502 21:25:26.942312 16905 tablet_bootstrap.cc:492] T c06e83b5216e4138817cc58a61c02e79 P 9efea4678e0e4dee8fe588222a3a6a67: Bootstrap starting.
I20240502 21:25:26.939502 16913 raft_consensus.cc:2726] T 71d05c4f1e6f4f7186ed7b8e157b7d2f P 8df3306d4bbe4fb1af0afb8d67c81798 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 21:25:26.943945 16913 raft_consensus.cc:482] T 05a7626e52ae43a6829feacaf1837ea5 P 8df3306d4bbe4fb1af0afb8d67c81798 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:25:26.944020 16905 tablet_bootstrap.cc:654] T c06e83b5216e4138817cc58a61c02e79 P 9efea4678e0e4dee8fe588222a3a6a67: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:26.944083 16913 raft_consensus.cc:504] T 05a7626e52ae43a6829feacaf1837ea5 P 8df3306d4bbe4fb1af0afb8d67c81798 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9efea4678e0e4dee8fe588222a3a6a67" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37931 } } peers { permanent_uuid: "8df3306d4bbe4fb1af0afb8d67c81798" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37673 } } peers { permanent_uuid: "1cfec8b74a2342cca5723d915046f117" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 43833 } }
I20240502 21:25:26.945233 16913 leader_election.cc:288] T 05a7626e52ae43a6829feacaf1837ea5 P 8df3306d4bbe4fb1af0afb8d67c81798 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9efea4678e0e4dee8fe588222a3a6a67 (127.15.255.129:37931), 1cfec8b74a2342cca5723d915046f117 (127.15.255.130:43833)
I20240502 21:25:26.946015 16699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05a7626e52ae43a6829feacaf1837ea5" candidate_uuid: "8df3306d4bbe4fb1af0afb8d67c81798" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cfec8b74a2342cca5723d915046f117" is_pre_election: true
I20240502 21:25:26.946238 16967 raft_consensus.cc:482] T 61268934b6b74fad90db0a52dd48ac82 P 1cfec8b74a2342cca5723d915046f117 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:25:26.946247 16699 raft_consensus.cc:2379] T 05a7626e52ae43a6829feacaf1837ea5 P 1cfec8b74a2342cca5723d915046f117 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8df3306d4bbe4fb1af0afb8d67c81798 in current term 1: Already voted for candidate 9efea4678e0e4dee8fe588222a3a6a67 in this term.
I20240502 21:25:26.946399 16967 raft_consensus.cc:504] T 61268934b6b74fad90db0a52dd48ac82 P 1cfec8b74a2342cca5723d915046f117 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cfec8b74a2342cca5723d915046f117" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 43833 } } peers { permanent_uuid: "9efea4678e0e4dee8fe588222a3a6a67" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37931 } } peers { permanent_uuid: "8df3306d4bbe4fb1af0afb8d67c81798" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37673 } }
I20240502 21:25:26.947294 16967 leader_election.cc:288] T 61268934b6b74fad90db0a52dd48ac82 P 1cfec8b74a2342cca5723d915046f117 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9efea4678e0e4dee8fe588222a3a6a67 (127.15.255.129:37931), 8df3306d4bbe4fb1af0afb8d67c81798 (127.15.255.131:37673)
I20240502 21:25:26.947611 16570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61268934b6b74fad90db0a52dd48ac82" candidate_uuid: "1cfec8b74a2342cca5723d915046f117" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9efea4678e0e4dee8fe588222a3a6a67" is_pre_election: true
I20240502 21:25:26.947816 16838 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61268934b6b74fad90db0a52dd48ac82" candidate_uuid: "1cfec8b74a2342cca5723d915046f117" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8df3306d4bbe4fb1af0afb8d67c81798" is_pre_election: true
I20240502 21:25:26.948001 16838 raft_consensus.cc:2454] T 61268934b6b74fad90db0a52dd48ac82 P 8df3306d4bbe4fb1af0afb8d67c81798 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cfec8b74a2342cca5723d915046f117 in term 0.
W20240502 21:25:26.948261 16635 leader_election.cc:341] T 61268934b6b74fad90db0a52dd48ac82 P 1cfec8b74a2342cca5723d915046f117 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9efea4678e0e4dee8fe588222a3a6a67 (127.15.255.129:37931): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:25:26.948482 16636 leader_election.cc:302] T 61268934b6b74fad90db0a52dd48ac82 P 1cfec8b74a2342cca5723d915046f117 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cfec8b74a2342cca5723d915046f117, 8df3306d4bbe4fb1af0afb8d67c81798; no voters: 9efea4678e0e4dee8fe588222a3a6a67
I20240502 21:25:26.948765 16967 raft_consensus.cc:2781] T 61268934b6b74fad90db0a52dd48ac82 P 1cfec8b74a2342cca5723d915046f117 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:25:26.948869 16967 raft_consensus.cc:482] T 61268934b6b74fad90db0a52dd48ac82 P 1cfec8b74a2342cca5723d915046f117 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:25:26.948976 16967 raft_consensus.cc:3037] T 61268934b6b74fad90db0a52dd48ac82 P 1cfec8b74a2342cca5723d915046f117 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:26.948956 16973 raft_consensus.cc:504] T 38999367825249e1a0f5f156bd3c3b25 P 8df3306d4bbe4fb1af0afb8d67c81798 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9efea4678e0e4dee8fe588222a3a6a67" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37931 } } peers { permanent_uuid: "1cfec8b74a2342cca5723d915046f117" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 43833 } } peers { permanent_uuid: "8df3306d4bbe4fb1af0afb8d67c81798" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37673 } }
I20240502 21:25:26.949200 16426 catalog_manager.cc:5484] T 71d05c4f1e6f4f7186ed7b8e157b7d2f P 1cfec8b74a2342cca5723d915046f117 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1cfec8b74a2342cca5723d915046f117 (127.15.255.130). New cstate: current_term: 1 leader_uuid: "1cfec8b74a2342cca5723d915046f117" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9efea4678e0e4dee8fe588222a3a6a67" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37931 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1cfec8b74a2342cca5723d915046f117" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 43833 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8df3306d4bbe4fb1af0afb8d67c81798" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37673 } health_report { overall_health: UNKNOWN } } }
I20240502 21:25:26.949882 16973 leader_election.cc:288] T 38999367825249e1a0f5f156bd3c3b25 P 8df3306d4bbe4fb1af0afb8d67c81798 [CANDIDATE]: Term 1 election: Requested vote from peers 9efea4678e0e4dee8fe588222a3a6a67 (127.15.255.129:37931), 1cfec8b74a2342cca5723d915046f117 (127.15.255.130:43833)
I20240502 21:25:26.949733 16426 catalog_manager.cc:5484] T 2b1fbeb84317494595f7262fe7857ebd P 1cfec8b74a2342cca5723d915046f117 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1cfec8b74a2342cca5723d915046f117 (127.15.255.130). New cstate: current_term: 1 leader_uuid: "1cfec8b74a2342cca5723d915046f117" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cfec8b74a2342cca5723d915046f117" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 43833 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9efea4678e0e4dee8fe588222a3a6a67" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37931 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8df3306d4bbe4fb1af0afb8d67c81798" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37673 } health_report { overall_health: UNKNOWN } } }
I20240502 21:25:26.950142 16699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38999367825249e1a0f5f156bd3c3b25" candidate_uuid: "8df3306d4bbe4fb1af0afb8d67c81798" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cfec8b74a2342cca5723d915046f117"
I20240502 21:25:26.950657 16903 tablet_bootstrap.cc:492] T 62a4fb5953dc4fa7badc5e2ee1ed4261 P 1cfec8b74a2342cca5723d915046f117: No bootstrap required, opened a new log
I20240502 21:25:26.950747 16569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72f6a167c03d4ee6bd72d6e5793f784d" candidate_uuid: "8df3306d4bbe4fb1af0afb8d67c81798" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9efea4678e0e4dee8fe588222a3a6a67"
I20240502 21:25:26.950919 16903 ts_tablet_manager.cc:1367] T 62a4fb5953dc4fa7badc5e2ee1ed4261 P 1cfec8b74a2342cca5723d915046f117: Time spent bootstrapping tablet: real 0.047s user 0.005s sys 0.006s
I20240502 21:25:26.950940 16569 raft_consensus.cc:3037] T 72f6a167c03d4ee6bd72d6e5793f784d P 9efea4678e0e4dee8fe588222a3a6a67 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:26.951722 16903 raft_consensus.cc:348] T 62a4fb5953dc4fa7badc5e2ee1ed4261 P 1cfec8b74a2342cca5723d915046f117 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9efea4678e0e4dee8fe588222a3a6a67" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37931 } } peers { permanent_uuid: "1cfec8b74a2342cca5723d915046f117" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 43833 } } peers { permanent_uuid: "8df3306d4bbe4fb1af0afb8d67c81798" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37673 } }
I20240502 21:25:26.951869 16967 raft_consensus.cc:504] T 61268934b6b74fad90db0a52dd48ac82 P 1cfec8b74a2342cca5723d915046f117 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cfec8b74a2342cca5723d915046f117" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 43833 } } peers { permanent_uuid: "9efea4678e0e4dee8fe588222a3a6a67" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37931 } } peers { permanent_uuid: "8df3306d4bbe4fb1af0afb8d67c81798" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37673 } }
I20240502 21:25:26.952018 16903 raft_consensus.cc:374] T 62a4fb5953dc4fa7badc5e2ee1ed4261 P 1cfec8b74a2342cca5723d915046f117 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:26.952203 16903 raft_consensus.cc:729] T 62a4fb5953dc4fa7badc5e2ee1ed4261 P 1cfec8b74a2342cca5723d915046f117 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cfec8b74a2342cca5723d915046f117, State: Initialized, Role: FOLLOWER
W20240502 21:25:26.952250 16771 leader_election.cc:341] T 38999367825249e1a0f5f156bd3c3b25 P 8df3306d4bbe4fb1af0afb8d67c81798 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1cfec8b74a2342cca5723d915046f117 (127.15.255.130:43833): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:25:26.952570 16903 consensus_queue.cc:260] T 62a4fb5953dc4fa7badc5e2ee1ed4261 P 1cfec8b74a2342cca5723d915046f117 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9efea4678e0e4dee8fe588222a3a6a67" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37931 } } peers { permanent_uuid: "1cfec8b74a2342cca5723d915046f117" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 43833 } } peers { permanent_uuid: "8df3306d4bbe4fb1af0afb8d67c81798" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37673 } }
I20240502 21:25:26.952828 16570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61268934b6b74fad90db0a52dd48ac82" candidate_uuid: "1cfec8b74a2342cca5723d915046f117" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9efea4678e0e4dee8fe588222a3a6a67"
I20240502 21:25:26.953301 16903 ts_tablet_manager.cc:1398] T 62a4fb5953dc4fa7badc5e2ee1ed4261 P 1cfec8b74a2342cca5723d915046f117: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 21:25:26.953291 16838 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61268934b6b74fad90db0a52dd48ac82" candidate_uuid: "1cfec8b74a2342cca5723d915046f117" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8df3306d4bbe4fb1af0afb8d67c81798"
W20240502 21:25:26.953374 16635 leader_election.cc:341] T 61268934b6b74fad90db0a52dd48ac82 P 1cfec8b74a2342cca5723d915046f117 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 9efea4678e0e4dee8fe588222a3a6a67 (127.15.255.129:37931): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:25:26.953418 16569 raft_consensus.cc:2454] T 72f6a167c03d4ee6bd72d6e5793f784d P 9efea4678e0e4dee8fe588222a3a6a67 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8df3306d4bbe4fb1af0afb8d67c81798 in term 1.
I20240502 21:25:26.953464 16838 raft_consensus.cc:3037] T 61268934b6b74fad90db0a52dd48ac82 P 8df3306d4bbe4fb1af0afb8d67c81798 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:26.953538 16967 leader_election.cc:288] T 61268934b6b74fad90db0a52dd48ac82 P 1cfec8b74a2342cca5723d915046f117 [CANDIDATE]: Term 1 election: Requested vote from peers 9efea4678e0e4dee8fe588222a3a6a67 (127.15.255.129:37931), 8df3306d4bbe4fb1af0afb8d67c81798 (127.15.255.131:37673)
I20240502 21:25:26.953826 16903 tablet_bootstrap.cc:492] T 38999367825249e1a0f5f156bd3c3b25 P 1cfec8b74a2342cca5723d915046f117: Bootstrap starting.
I20240502 21:25:26.954239 16904 ts_tablet_manager.cc:1398] T 05a7626e52ae43a6829feacaf1837ea5 P 8df3306d4bbe4fb1af0afb8d67c81798: Time spent starting tablet: real 0.016s user 0.007s sys 0.003s
I20240502 21:25:26.947151 16978 consensus_queue.cc:237] T 72f6a167c03d4ee6bd72d6e5793f784d P 8df3306d4bbe4fb1af0afb8d67c81798 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9efea4678e0e4dee8fe588222a3a6a67" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37931 } } peers { permanent_uuid: "1cfec8b74a2342cca5723d915046f117" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 43833 } } peers { permanent_uuid: "8df3306d4bbe4fb1af0afb8d67c81798" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37673 } }
I20240502 21:25:26.955360 16569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05a7626e52ae43a6829feacaf1837ea5" candidate_uuid: "8df3306d4bbe4fb1af0afb8d67c81798" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9efea4678e0e4dee8fe588222a3a6a67" is_pre_election: true
I20240502 21:25:26.955971 16838 raft_consensus.cc:2454] T 61268934b6b74fad90db0a52dd48ac82 P 8df3306d4bbe4fb1af0afb8d67c81798 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cfec8b74a2342cca5723d915046f117 in term 1.
I20240502 21:25:26.954560 16904 tablet_bootstrap.cc:492] T 2b1fbeb84317494595f7262fe7857ebd P 8df3306d4bbe4fb1af0afb8d67c81798: Bootstrap starting.
I20240502 21:25:26.956445 16636 leader_election.cc:302] T 61268934b6b74fad90db0a52dd48ac82 P 1cfec8b74a2342cca5723d915046f117 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cfec8b74a2342cca5723d915046f117, 8df3306d4bbe4fb1af0afb8d67c81798; no voters: 9efea4678e0e4dee8fe588222a3a6a67
I20240502 21:25:26.954689 16974 leader_election.cc:288] T 72f6a167c03d4ee6bd72d6e5793f784d P 8df3306d4bbe4fb1af0afb8d67c81798 [CANDIDATE]: Term 1 election: Requested vote from peers 9efea4678e0e4dee8fe588222a3a6a67 (127.15.255.129:37931), 1cfec8b74a2342cca5723d915046f117 (127.15.255.130:43833)
I20240502 21:25:26.956838 16569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38999367825249e1a0f5f156bd3c3b25" candidate_uuid: "8df3306d4bbe4fb1af0afb8d67c81798" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9efea4678e0e4dee8fe588222a3a6a67"
I20240502 21:25:26.956969 16967 raft_consensus.cc:2781] T 61268934b6b74fad90db0a52dd48ac82 P 1cfec8b74a2342cca5723d915046f117 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:25:26.957109 16569 raft_consensus.cc:3037] T 38999367825249e1a0f5f156bd3c3b25 P 9efea4678e0e4dee8fe588222a3a6a67 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:26.957113 16967 raft_consensus.cc:686] T 61268934b6b74fad90db0a52dd48ac82 P 1cfec8b74a2342cca5723d915046f117 [term 1 LEADER]: Becoming Leader. State: Replica: 1cfec8b74a2342cca5723d915046f117, State: Running, Role: LEADER
I20240502 21:25:26.957367 16967 consensus_queue.cc:237] T 61268934b6b74fad90db0a52dd48ac82 P 1cfec8b74a2342cca5723d915046f117 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cfec8b74a2342cca5723d915046f117" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 43833 } } peers { permanent_uuid: "9efea4678e0e4dee8fe588222a3a6a67" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37931 } } peers { permanent_uuid: "8df3306d4bbe4fb1af0afb8d67c81798" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37673 } }
I20240502 21:25:26.957774 16904 tablet_bootstrap.cc:654] T 2b1fbeb84317494595f7262fe7857ebd P 8df3306d4bbe4fb1af0afb8d67c81798: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:26.958961 16903 tablet_bootstrap.cc:654] T 38999367825249e1a0f5f156bd3c3b25 P 1cfec8b74a2342cca5723d915046f117: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:26.959885 16569 raft_consensus.cc:2454] T 38999367825249e1a0f5f156bd3c3b25 P 9efea4678e0e4dee8fe588222a3a6a67 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8df3306d4bbe4fb1af0afb8d67c81798 in term 1.
I20240502 21:25:26.960671 16767 leader_election.cc:302] T 05a7626e52ae43a6829feacaf1837ea5 P 8df3306d4bbe4fb1af0afb8d67c81798 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8df3306d4bbe4fb1af0afb8d67c81798; no voters: 1cfec8b74a2342cca5723d915046f117, 9efea4678e0e4dee8fe588222a3a6a67
I20240502 21:25:26.961099 16974 raft_consensus.cc:3037] T 05a7626e52ae43a6829feacaf1837ea5 P 8df3306d4bbe4fb1af0afb8d67c81798 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:26.963907 16974 raft_consensus.cc:2726] T 05a7626e52ae43a6829feacaf1837ea5 P 8df3306d4bbe4fb1af0afb8d67c81798 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 21:25:26.964273 16767 leader_election.cc:302] T 38999367825249e1a0f5f156bd3c3b25 P 8df3306d4bbe4fb1af0afb8d67c81798 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8df3306d4bbe4fb1af0afb8d67c81798, 9efea4678e0e4dee8fe588222a3a6a67; no voters: 1cfec8b74a2342cca5723d915046f117
I20240502 21:25:26.964886 16974 raft_consensus.cc:2781] T 38999367825249e1a0f5f156bd3c3b25 P 8df3306d4bbe4fb1af0afb8d67c81798 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:25:26.965062 16974 raft_consensus.cc:686] T 38999367825249e1a0f5f156bd3c3b25 P 8df3306d4bbe4fb1af0afb8d67c81798 [term 1 LEADER]: Becoming Leader. State: Replica: 8df3306d4bbe4fb1af0afb8d67c81798, State: Running, Role: LEADER
I20240502 21:25:26.965355 16974 consensus_queue.cc:237] T 38999367825249e1a0f5f156bd3c3b25 P 8df3306d4bbe4fb1af0afb8d67c81798 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9efea4678e0e4dee8fe588222a3a6a67" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37931 } } peers { permanent_uuid: "1cfec8b74a2342cca5723d915046f117" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 43833 } } peers { permanent_uuid: "8df3306d4bbe4fb1af0afb8d67c81798" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37673 } }
I20240502 21:25:26.966681 16904 tablet_bootstrap.cc:492] T 2b1fbeb84317494595f7262fe7857ebd P 8df3306d4bbe4fb1af0afb8d67c81798: No bootstrap required, opened a new log
I20240502 21:25:26.966845 16904 ts_tablet_manager.cc:1367] T 2b1fbeb84317494595f7262fe7857ebd P 8df3306d4bbe4fb1af0afb8d67c81798: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.000s
I20240502 21:25:26.966964 16425 catalog_manager.cc:5484] T 72f6a167c03d4ee6bd72d6e5793f784d P 8df3306d4bbe4fb1af0afb8d67c81798 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8df3306d4bbe4fb1af0afb8d67c81798 (127.15.255.131). New cstate: current_term: 1 leader_uuid: "8df3306d4bbe4fb1af0afb8d67c81798" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9efea4678e0e4dee8fe588222a3a6a67" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37931 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1cfec8b74a2342cca5723d915046f117" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 43833 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8df3306d4bbe4fb1af0afb8d67c81798" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37673 } health_report { overall_health: HEALTHY } } }
I20240502 21:25:26.967579 16904 raft_consensus.cc:348] T 2b1fbeb84317494595f7262fe7857ebd P 8df3306d4bbe4fb1af0afb8d67c81798 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cfec8b74a2342cca5723d915046f117" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 43833 } } peers { permanent_uuid: "9efea4678e0e4dee8fe588222a3a6a67" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37931 } } peers { permanent_uuid: "8df3306d4bbe4fb1af0afb8d67c81798" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37673 } }
I20240502 21:25:26.968153 16904 raft_consensus.cc:374] T 2b1fbeb84317494595f7262fe7857ebd P 8df3306d4bbe4fb1af0afb8d67c81798 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:26.968230 16904 raft_consensus.cc:729] T 2b1fbeb84317494595f7262fe7857ebd P 8df3306d4bbe4fb1af0afb8d67c81798 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8df3306d4bbe4fb1af0afb8d67c81798, State: Initialized, Role: FOLLOWER
I20240502 21:25:26.968537 16903 tablet_bootstrap.cc:492] T 38999367825249e1a0f5f156bd3c3b25 P 1cfec8b74a2342cca5723d915046f117: No bootstrap required, opened a new log
I20240502 21:25:26.968469 16904 consensus_queue.cc:260] T 2b1fbeb84317494595f7262fe7857ebd P 8df3306d4bbe4fb1af0afb8d67c81798 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cfec8b74a2342cca5723d915046f117" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 43833 } } peers { permanent_uuid: "9efea4678e0e4dee8fe588222a3a6a67" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37931 } } peers { permanent_uuid: "8df3306d4bbe4fb1af0afb8d67c81798" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37673 } }
I20240502 21:25:26.968696 16903 ts_tablet_manager.cc:1367] T 38999367825249e1a0f5f156bd3c3b25 P 1cfec8b74a2342cca5723d915046f117: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.002s
I20240502 21:25:26.969041 16904 ts_tablet_manager.cc:1398] T 2b1fbeb84317494595f7262fe7857ebd P 8df3306d4bbe4fb1af0afb8d67c81798: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:25:26.969343 16904 tablet_bootstrap.cc:492] T 1b9f6d5e769e48f394fe59a835c22fc0 P 8df3306d4bbe4fb1af0afb8d67c81798: Bootstrap starting.
I20240502 21:25:26.969451 16903 raft_consensus.cc:348] T 38999367825249e1a0f5f156bd3c3b25 P 1cfec8b74a2342cca5723d915046f117 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9efea4678e0e4dee8fe588222a3a6a67" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37931 } } peers { permanent_uuid: "1cfec8b74a2342cca5723d915046f117" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 43833 } } peers { permanent_uuid: "8df3306d4bbe4fb1af0afb8d67c81798" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37673 } }
I20240502 21:25:26.969664 16903 raft_consensus.cc:374] T 38999367825249e1a0f5f156bd3c3b25 P 1cfec8b74a2342cca5723d915046f117 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:26.969756 16903 raft_consensus.cc:729] T 38999367825249e1a0f5f156bd3c3b25 P 1cfec8b74a2342cca5723d915046f117 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cfec8b74a2342cca5723d915046f117, State: Initialized, Role: FOLLOWER
I20240502 21:25:26.970003 16903 consensus_queue.cc:260] T 38999367825249e1a0f5f156bd3c3b25 P 1cfec8b74a2342cca5723d915046f117 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9efea4678e0e4dee8fe588222a3a6a67" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37931 } } peers { permanent_uuid: "1cfec8b74a2342cca5723d915046f117" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 43833 } } peers { permanent_uuid: "8df3306d4bbe4fb1af0afb8d67c81798" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37673 } }
I20240502 21:25:26.970207 16424 catalog_manager.cc:5484] T 61268934b6b74fad90db0a52dd48ac82 P 1cfec8b74a2342cca5723d915046f117 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1cfec8b74a2342cca5723d915046f117 (127.15.255.130). New cstate: current_term: 1 leader_uuid: "1cfec8b74a2342cca5723d915046f117" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cfec8b74a2342cca5723d915046f117" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 43833 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9efea4678e0e4dee8fe588222a3a6a67" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37931 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8df3306d4bbe4fb1af0afb8d67c81798" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37673 } health_report { overall_health: UNKNOWN } } }
I20240502 21:25:26.970500 16903 ts_tablet_manager.cc:1398] T 38999367825249e1a0f5f156bd3c3b25 P 1cfec8b74a2342cca5723d915046f117: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:25:26.970991 16904 tablet_bootstrap.cc:654] T 1b9f6d5e769e48f394fe59a835c22fc0 P 8df3306d4bbe4fb1af0afb8d67c81798: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:26.972728 16426 catalog_manager.cc:5484] T 62a4fb5953dc4fa7badc5e2ee1ed4261 P 9efea4678e0e4dee8fe588222a3a6a67 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9efea4678e0e4dee8fe588222a3a6a67 (127.15.255.129). New cstate: current_term: 1 leader_uuid: "9efea4678e0e4dee8fe588222a3a6a67" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9efea4678e0e4dee8fe588222a3a6a67" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37931 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1cfec8b74a2342cca5723d915046f117" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 43833 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8df3306d4bbe4fb1af0afb8d67c81798" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37673 } health_report { overall_health: UNKNOWN } } }
I20240502 21:25:26.979084 16904 tablet_bootstrap.cc:492] T 1b9f6d5e769e48f394fe59a835c22fc0 P 8df3306d4bbe4fb1af0afb8d67c81798: No bootstrap required, opened a new log
I20240502 21:25:26.979233 16904 ts_tablet_manager.cc:1367] T 1b9f6d5e769e48f394fe59a835c22fc0 P 8df3306d4bbe4fb1af0afb8d67c81798: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20240502 21:25:26.980027 16904 raft_consensus.cc:348] T 1b9f6d5e769e48f394fe59a835c22fc0 P 8df3306d4bbe4fb1af0afb8d67c81798 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9efea4678e0e4dee8fe588222a3a6a67" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37931 } } peers { permanent_uuid: "1cfec8b74a2342cca5723d915046f117" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 43833 } } peers { permanent_uuid: "8df3306d4bbe4fb1af0afb8d67c81798" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37673 } }
I20240502 21:25:26.980208 16904 raft_consensus.cc:374] T 1b9f6d5e769e48f394fe59a835c22fc0 P 8df3306d4bbe4fb1af0afb8d67c81798 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:26.980340 16904 raft_consensus.cc:729] T 1b9f6d5e769e48f394fe59a835c22fc0 P 8df3306d4bbe4fb1af0afb8d67c81798 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8df3306d4bbe4fb1af0afb8d67c81798, State: Initialized, Role: FOLLOWER
I20240502 21:25:26.980603 16904 consensus_queue.cc:260] T 1b9f6d5e769e48f394fe59a835c22fc0 P 8df3306d4bbe4fb1af0afb8d67c81798 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9efea4678e0e4dee8fe588222a3a6a67" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37931 } } peers { permanent_uuid: "1cfec8b74a2342cca5723d915046f117" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 43833 } } peers { permanent_uuid: "8df3306d4bbe4fb1af0afb8d67c81798" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37673 } }
I20240502 21:25:26.980854 16963 raft_consensus.cc:482] T ed40aed15ca3447aa01283771ac257ed P 9efea4678e0e4dee8fe588222a3a6a67 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:25:26.981040 16963 raft_consensus.cc:504] T ed40aed15ca3447aa01283771ac257ed P 9efea4678e0e4dee8fe588222a3a6a67 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9efea4678e0e4dee8fe588222a3a6a67" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37931 } } peers { permanent_uuid: "1cfec8b74a2342cca5723d915046f117" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 43833 } } peers { permanent_uuid: "8df3306d4bbe4fb1af0afb8d67c81798" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37673 } }
I20240502 21:25:26.981140 16904 ts_tablet_manager.cc:1398] T 1b9f6d5e769e48f394fe59a835c22fc0 P 8df3306d4bbe4fb1af0afb8d67c81798: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:25:26.981245 16905 tablet_bootstrap.cc:492] T c06e83b5216e4138817cc58a61c02e79 P 9efea4678e0e4dee8fe588222a3a6a67: No bootstrap required, opened a new log
I20240502 21:25:26.981390 16905 ts_tablet_manager.cc:1367] T c06e83b5216e4138817cc58a61c02e79 P 9efea4678e0e4dee8fe588222a3a6a67: Time spent bootstrapping tablet: real 0.039s user 0.004s sys 0.002s
I20240502 21:25:26.981429 16904 tablet_bootstrap.cc:492] T 4cfc1f2146dd449c87a46196e25280c2 P 8df3306d4bbe4fb1af0afb8d67c81798: Bootstrap starting.
I20240502 21:25:26.981315 16426 catalog_manager.cc:5484] T 38999367825249e1a0f5f156bd3c3b25 P 8df3306d4bbe4fb1af0afb8d67c81798 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8df3306d4bbe4fb1af0afb8d67c81798 (127.15.255.131). New cstate: current_term: 1 leader_uuid: "8df3306d4bbe4fb1af0afb8d67c81798" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9efea4678e0e4dee8fe588222a3a6a67" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37931 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1cfec8b74a2342cca5723d915046f117" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 43833 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8df3306d4bbe4fb1af0afb8d67c81798" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37673 } health_report { overall_health: HEALTHY } } }
I20240502 21:25:26.981689 16963 leader_election.cc:288] T ed40aed15ca3447aa01283771ac257ed P 9efea4678e0e4dee8fe588222a3a6a67 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cfec8b74a2342cca5723d915046f117 (127.15.255.130:43833), 8df3306d4bbe4fb1af0afb8d67c81798 (127.15.255.131:37673)
I20240502 21:25:26.982100 16905 raft_consensus.cc:348] T c06e83b5216e4138817cc58a61c02e79 P 9efea4678e0e4dee8fe588222a3a6a67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cfec8b74a2342cca5723d915046f117" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 43833 } } peers { permanent_uuid: "9efea4678e0e4dee8fe588222a3a6a67" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37931 } } peers { permanent_uuid: "8df3306d4bbe4fb1af0afb8d67c81798" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37673 } }
I20240502 21:25:26.982224 16699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed40aed15ca3447aa01283771ac257ed" candidate_uuid: "9efea4678e0e4dee8fe588222a3a6a67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cfec8b74a2342cca5723d915046f117" is_pre_election: true
I20240502 21:25:26.982306 16905 raft_consensus.cc:374] T c06e83b5216e4138817cc58a61c02e79 P 9efea4678e0e4dee8fe588222a3a6a67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:26.982369 16905 raft_consensus.cc:729] T c06e83b5216e4138817cc58a61c02e79 P 9efea4678e0e4dee8fe588222a3a6a67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9efea4678e0e4dee8fe588222a3a6a67, State: Initialized, Role: FOLLOWER
I20240502 21:25:26.982425 16699 raft_consensus.cc:2454] T ed40aed15ca3447aa01283771ac257ed P 1cfec8b74a2342cca5723d915046f117 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9efea4678e0e4dee8fe588222a3a6a67 in term 0.
I20240502 21:25:26.982861 16904 tablet_bootstrap.cc:654] T 4cfc1f2146dd449c87a46196e25280c2 P 8df3306d4bbe4fb1af0afb8d67c81798: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:26.982837 16506 leader_election.cc:302] T ed40aed15ca3447aa01283771ac257ed P 9efea4678e0e4dee8fe588222a3a6a67 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cfec8b74a2342cca5723d915046f117, 9efea4678e0e4dee8fe588222a3a6a67; no voters:
I20240502 21:25:26.983137 16963 raft_consensus.cc:2781] T ed40aed15ca3447aa01283771ac257ed P 9efea4678e0e4dee8fe588222a3a6a67 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:25:26.983222 16963 raft_consensus.cc:482] T ed40aed15ca3447aa01283771ac257ed P 9efea4678e0e4dee8fe588222a3a6a67 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:25:26.983319 16963 raft_consensus.cc:3037] T ed40aed15ca3447aa01283771ac257ed P 9efea4678e0e4dee8fe588222a3a6a67 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:26.983419 16838 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed40aed15ca3447aa01283771ac257ed" candidate_uuid: "9efea4678e0e4dee8fe588222a3a6a67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8df3306d4bbe4fb1af0afb8d67c81798" is_pre_election: true
I20240502 21:25:26.983628 16838 raft_consensus.cc:2454] T ed40aed15ca3447aa01283771ac257ed P 8df3306d4bbe4fb1af0afb8d67c81798 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9efea4678e0e4dee8fe588222a3a6a67 in term 0.
I20240502 21:25:26.984023 16905 consensus_queue.cc:260] T c06e83b5216e4138817cc58a61c02e79 P 9efea4678e0e4dee8fe588222a3a6a67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cfec8b74a2342cca5723d915046f117" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 43833 } } peers { permanent_uuid: "9efea4678e0e4dee8fe588222a3a6a67" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37931 } } peers { permanent_uuid: "8df3306d4bbe4fb1af0afb8d67c81798" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37673 } }
I20240502 21:25:26.985283 16905 ts_tablet_manager.cc:1398] T c06e83b5216e4138817cc58a61c02e79 P 9efea4678e0e4dee8fe588222a3a6a67: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240502 21:25:26.985688 16905 tablet_bootstrap.cc:492] T 61268934b6b74fad90db0a52dd48ac82 P 9efea4678e0e4dee8fe588222a3a6a67: Bootstrap starting.
I20240502 21:25:26.985872 16963 raft_consensus.cc:504] T ed40aed15ca3447aa01283771ac257ed P 9efea4678e0e4dee8fe588222a3a6a67 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9efea4678e0e4dee8fe588222a3a6a67" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37931 } } peers { permanent_uuid: "1cfec8b74a2342cca5723d915046f117" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 43833 } } peers { permanent_uuid: "8df3306d4bbe4fb1af0afb8d67c81798" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37673 } }
I20240502 21:25:26.986793 16699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed40aed15ca3447aa01283771ac257ed" candidate_uuid: "9efea4678e0e4dee8fe588222a3a6a67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cfec8b74a2342cca5723d915046f117"
I20240502 21:25:26.986980 16699 raft_consensus.cc:3037] T ed40aed15ca3447aa01283771ac257ed P 1cfec8b74a2342cca5723d915046f117 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:26.987792 16904 tablet_bootstrap.cc:492] T 4cfc1f2146dd449c87a46196e25280c2 P 8df3306d4bbe4fb1af0afb8d67c81798: No bootstrap required, opened a new log
I20240502 21:25:26.987923 16904 ts_tablet_manager.cc:1367] T 4cfc1f2146dd449c87a46196e25280c2 P 8df3306d4bbe4fb1af0afb8d67c81798: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.001s
I20240502 21:25:26.987934 16905 tablet_bootstrap.cc:654] T 61268934b6b74fad90db0a52dd48ac82 P 9efea4678e0e4dee8fe588222a3a6a67: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:26.988549 16904 raft_consensus.cc:348] T 4cfc1f2146dd449c87a46196e25280c2 P 8df3306d4bbe4fb1af0afb8d67c81798 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cfec8b74a2342cca5723d915046f117" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 43833 } } peers { permanent_uuid: "9efea4678e0e4dee8fe588222a3a6a67" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37931 } } peers { permanent_uuid: "8df3306d4bbe4fb1af0afb8d67c81798" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37673 } }
I20240502 21:25:26.988759 16904 raft_consensus.cc:374] T 4cfc1f2146dd449c87a46196e25280c2 P 8df3306d4bbe4fb1af0afb8d67c81798 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:26.988829 16904 raft_consensus.cc:729] T 4cfc1f2146dd449c87a46196e25280c2 P 8df3306d4bbe4fb1af0afb8d67c81798 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8df3306d4bbe4fb1af0afb8d67c81798, State: Initialized, Role: FOLLOWER
I20240502 21:25:26.989033 16904 consensus_queue.cc:260] T 4cfc1f2146dd449c87a46196e25280c2 P 8df3306d4bbe4fb1af0afb8d67c81798 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cfec8b74a2342cca5723d915046f117" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 43833 } } peers { permanent_uuid: "9efea4678e0e4dee8fe588222a3a6a67" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37931 } } peers { permanent_uuid: "8df3306d4bbe4fb1af0afb8d67c81798" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37673 } }
I20240502 21:25:26.989482 16699 raft_consensus.cc:2454] T ed40aed15ca3447aa01283771ac257ed P 1cfec8b74a2342cca5723d915046f117 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9efea4678e0e4dee8fe588222a3a6a67 in term 1.
I20240502 21:25:26.989609 16904 ts_tablet_manager.cc:1398] T 4cfc1f2146dd449c87a46196e25280c2 P 8df3306d4bbe4fb1af0afb8d67c81798: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:25:26.989885 16904 tablet_bootstrap.cc:492] T c06e83b5216e4138817cc58a61c02e79 P 8df3306d4bbe4fb1af0afb8d67c81798: Bootstrap starting.
I20240502 21:25:26.990043 16506 leader_election.cc:302] T ed40aed15ca3447aa01283771ac257ed P 9efea4678e0e4dee8fe588222a3a6a67 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cfec8b74a2342cca5723d915046f117, 9efea4678e0e4dee8fe588222a3a6a67; no voters:
I20240502 21:25:26.990233 16905 tablet_bootstrap.cc:492] T 61268934b6b74fad90db0a52dd48ac82 P 9efea4678e0e4dee8fe588222a3a6a67: No bootstrap required, opened a new log
I20240502 21:25:26.990271 16838 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed40aed15ca3447aa01283771ac257ed" candidate_uuid: "9efea4678e0e4dee8fe588222a3a6a67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8df3306d4bbe4fb1af0afb8d67c81798"
I20240502 21:25:26.990347 16905 ts_tablet_manager.cc:1367] T 61268934b6b74fad90db0a52dd48ac82 P 9efea4678e0e4dee8fe588222a3a6a67: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.002s
I20240502 21:25:26.990461 16838 raft_consensus.cc:3037] T ed40aed15ca3447aa01283771ac257ed P 8df3306d4bbe4fb1af0afb8d67c81798 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:26.991042 16905 raft_consensus.cc:348] T 61268934b6b74fad90db0a52dd48ac82 P 9efea4678e0e4dee8fe588222a3a6a67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cfec8b74a2342cca5723d915046f117" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 43833 } } peers { permanent_uuid: "9efea4678e0e4dee8fe588222a3a6a67" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37931 } } peers { permanent_uuid: "8df3306d4bbe4fb1af0afb8d67c81798" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37673 } }
I20240502 21:25:26.991273 16905 raft_consensus.cc:374] T 61268934b6b74fad90db0a52dd48ac82 P 9efea4678e0e4dee8fe588222a3a6a67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:26.991361 16905 raft_consensus.cc:729] T 61268934b6b74fad90db0a52dd48ac82 P 9efea4678e0e4dee8fe588222a3a6a67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9efea4678e0e4dee8fe588222a3a6a67, State: Initialized, Role: FOLLOWER
I20240502 21:25:26.991389 16904 tablet_bootstrap.cc:654] T c06e83b5216e4138817cc58a61c02e79 P 8df3306d4bbe4fb1af0afb8d67c81798: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:26.991606 16905 consensus_queue.cc:260] T 61268934b6b74fad90db0a52dd48ac82 P 9efea4678e0e4dee8fe588222a3a6a67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cfec8b74a2342cca5723d915046f117" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 43833 } } peers { permanent_uuid: "9efea4678e0e4dee8fe588222a3a6a67" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37931 } } peers { permanent_uuid: "8df3306d4bbe4fb1af0afb8d67c81798" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37673 } }
I20240502 21:25:26.991901 16966 raft_consensus.cc:2781] T ed40aed15ca3447aa01283771ac257ed P 9efea4678e0e4dee8fe588222a3a6a67 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:25:26.992034 16966 raft_consensus.cc:686] T ed40aed15ca3447aa01283771ac257ed P 9efea4678e0e4dee8fe588222a3a6a67 [term 1 LEADER]: Becoming Leader. State: Replica: 9efea4678e0e4dee8fe588222a3a6a67, State: Running, Role: LEADER
I20240502 21:25:26.992348 16966 consensus_queue.cc:237] T ed40aed15ca3447aa01283771ac257ed P 9efea4678e0e4dee8fe588222a3a6a67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9efea4678e0e4dee8fe588222a3a6a67" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37931 } } peers { permanent_uuid: "1cfec8b74a2342cca5723d915046f117" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 43833 } } peers { permanent_uuid: "8df3306d4bbe4fb1af0afb8d67c81798" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37673 } }
I20240502 21:25:26.993206 16838 raft_consensus.cc:2454] T ed40aed15ca3447aa01283771ac257ed P 8df3306d4bbe4fb1af0afb8d67c81798 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9efea4678e0e4dee8fe588222a3a6a67 in term 1.
I20240502 21:25:26.994761 16905 ts_tablet_manager.cc:1398] T 61268934b6b74fad90db0a52dd48ac82 P 9efea4678e0e4dee8fe588222a3a6a67: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240502 21:25:26.993736 16963 leader_election.cc:288] T ed40aed15ca3447aa01283771ac257ed P 9efea4678e0e4dee8fe588222a3a6a67 [CANDIDATE]: Term 1 election: Requested vote from peers 1cfec8b74a2342cca5723d915046f117 (127.15.255.130:43833), 8df3306d4bbe4fb1af0afb8d67c81798 (127.15.255.131:37673)
I20240502 21:25:26.995527 16426 catalog_manager.cc:5484] T ed40aed15ca3447aa01283771ac257ed P 9efea4678e0e4dee8fe588222a3a6a67 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9efea4678e0e4dee8fe588222a3a6a67 (127.15.255.129). New cstate: current_term: 1 leader_uuid: "9efea4678e0e4dee8fe588222a3a6a67" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9efea4678e0e4dee8fe588222a3a6a67" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37931 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1cfec8b74a2342cca5723d915046f117" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 43833 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8df3306d4bbe4fb1af0afb8d67c81798" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37673 } health_report { overall_health: UNKNOWN } } }
I20240502 21:25:26.995905 16904 tablet_bootstrap.cc:492] T c06e83b5216e4138817cc58a61c02e79 P 8df3306d4bbe4fb1af0afb8d67c81798: No bootstrap required, opened a new log
I20240502 21:25:26.995995 16904 ts_tablet_manager.cc:1367] T c06e83b5216e4138817cc58a61c02e79 P 8df3306d4bbe4fb1af0afb8d67c81798: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.001s
I20240502 21:25:26.996533 16904 raft_consensus.cc:348] T c06e83b5216e4138817cc58a61c02e79 P 8df3306d4bbe4fb1af0afb8d67c81798 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cfec8b74a2342cca5723d915046f117" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 43833 } } peers { permanent_uuid: "9efea4678e0e4dee8fe588222a3a6a67" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37931 } } peers { permanent_uuid: "8df3306d4bbe4fb1af0afb8d67c81798" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37673 } }
I20240502 21:25:26.996704 16904 raft_consensus.cc:374] T c06e83b5216e4138817cc58a61c02e79 P 8df3306d4bbe4fb1af0afb8d67c81798 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:26.996805 16904 raft_consensus.cc:729] T c06e83b5216e4138817cc58a61c02e79 P 8df3306d4bbe4fb1af0afb8d67c81798 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8df3306d4bbe4fb1af0afb8d67c81798, State: Initialized, Role: FOLLOWER
I20240502 21:25:26.997026 16904 consensus_queue.cc:260] T c06e83b5216e4138817cc58a61c02e79 P 8df3306d4bbe4fb1af0afb8d67c81798 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cfec8b74a2342cca5723d915046f117" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 43833 } } peers { permanent_uuid: "9efea4678e0e4dee8fe588222a3a6a67" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37931 } } peers { permanent_uuid: "8df3306d4bbe4fb1af0afb8d67c81798" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37673 } }
I20240502 21:25:26.997467 16904 ts_tablet_manager.cc:1398] T c06e83b5216e4138817cc58a61c02e79 P 8df3306d4bbe4fb1af0afb8d67c81798: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 21:25:26.999699 16974 raft_consensus.cc:482] T 2b1fbeb84317494595f7262fe7857ebd P 8df3306d4bbe4fb1af0afb8d67c81798 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:25:26.999833 16974 raft_consensus.cc:504] T 2b1fbeb84317494595f7262fe7857ebd P 8df3306d4bbe4fb1af0afb8d67c81798 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cfec8b74a2342cca5723d915046f117" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 43833 } } peers { permanent_uuid: "9efea4678e0e4dee8fe588222a3a6a67" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37931 } } peers { permanent_uuid: "8df3306d4bbe4fb1af0afb8d67c81798" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37673 } }
I20240502 21:25:27.000488 16974 leader_election.cc:288] T 2b1fbeb84317494595f7262fe7857ebd P 8df3306d4bbe4fb1af0afb8d67c81798 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cfec8b74a2342cca5723d915046f117 (127.15.255.130:43833), 9efea4678e0e4dee8fe588222a3a6a67 (127.15.255.129:37931)
I20240502 21:25:27.000733 16699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b1fbeb84317494595f7262fe7857ebd" candidate_uuid: "8df3306d4bbe4fb1af0afb8d67c81798" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cfec8b74a2342cca5723d915046f117" is_pre_election: true
I20240502 21:25:27.001035 16569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b1fbeb84317494595f7262fe7857ebd" candidate_uuid: "8df3306d4bbe4fb1af0afb8d67c81798" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9efea4678e0e4dee8fe588222a3a6a67" is_pre_election: true
I20240502 21:25:27.001255 16569 raft_consensus.cc:2379] T 2b1fbeb84317494595f7262fe7857ebd P 9efea4678e0e4dee8fe588222a3a6a67 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8df3306d4bbe4fb1af0afb8d67c81798 in current term 1: Already voted for candidate 1cfec8b74a2342cca5723d915046f117 in this term.
I20240502 21:25:27.001574 16767 leader_election.cc:302] T 2b1fbeb84317494595f7262fe7857ebd P 8df3306d4bbe4fb1af0afb8d67c81798 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8df3306d4bbe4fb1af0afb8d67c81798; no voters: 1cfec8b74a2342cca5723d915046f117, 9efea4678e0e4dee8fe588222a3a6a67
I20240502 21:25:27.001799 16974 raft_consensus.cc:3037] T 2b1fbeb84317494595f7262fe7857ebd P 8df3306d4bbe4fb1af0afb8d67c81798 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:27.003386 16974 raft_consensus.cc:2726] T 2b1fbeb84317494595f7262fe7857ebd P 8df3306d4bbe4fb1af0afb8d67c81798 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 21:25:27.052155 16963 raft_consensus.cc:482] T c06e83b5216e4138817cc58a61c02e79 P 9efea4678e0e4dee8fe588222a3a6a67 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:25:27.052326 16963 raft_consensus.cc:504] T c06e83b5216e4138817cc58a61c02e79 P 9efea4678e0e4dee8fe588222a3a6a67 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cfec8b74a2342cca5723d915046f117" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 43833 } } peers { permanent_uuid: "9efea4678e0e4dee8fe588222a3a6a67" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37931 } } peers { permanent_uuid: "8df3306d4bbe4fb1af0afb8d67c81798" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37673 } }
I20240502 21:25:27.052958 16963 leader_election.cc:288] T c06e83b5216e4138817cc58a61c02e79 P 9efea4678e0e4dee8fe588222a3a6a67 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cfec8b74a2342cca5723d915046f117 (127.15.255.130:43833), 8df3306d4bbe4fb1af0afb8d67c81798 (127.15.255.131:37673)
I20240502 21:25:27.053303 16699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c06e83b5216e4138817cc58a61c02e79" candidate_uuid: "9efea4678e0e4dee8fe588222a3a6a67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cfec8b74a2342cca5723d915046f117" is_pre_election: true
I20240502 21:25:27.053457 16838 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c06e83b5216e4138817cc58a61c02e79" candidate_uuid: "9efea4678e0e4dee8fe588222a3a6a67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8df3306d4bbe4fb1af0afb8d67c81798" is_pre_election: true
I20240502 21:25:27.053506 16699 raft_consensus.cc:2454] T c06e83b5216e4138817cc58a61c02e79 P 1cfec8b74a2342cca5723d915046f117 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9efea4678e0e4dee8fe588222a3a6a67 in term 0.
I20240502 21:25:27.053673 16838 raft_consensus.cc:2454] T c06e83b5216e4138817cc58a61c02e79 P 8df3306d4bbe4fb1af0afb8d67c81798 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9efea4678e0e4dee8fe588222a3a6a67 in term 0.
I20240502 21:25:27.053951 16506 leader_election.cc:302] T c06e83b5216e4138817cc58a61c02e79 P 9efea4678e0e4dee8fe588222a3a6a67 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cfec8b74a2342cca5723d915046f117, 9efea4678e0e4dee8fe588222a3a6a67; no voters:
I20240502 21:25:27.054188 16963 raft_consensus.cc:2781] T c06e83b5216e4138817cc58a61c02e79 P 9efea4678e0e4dee8fe588222a3a6a67 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:25:27.054265 16963 raft_consensus.cc:482] T c06e83b5216e4138817cc58a61c02e79 P 9efea4678e0e4dee8fe588222a3a6a67 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:25:27.054337 16963 raft_consensus.cc:3037] T c06e83b5216e4138817cc58a61c02e79 P 9efea4678e0e4dee8fe588222a3a6a67 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:27.056021 16963 raft_consensus.cc:504] T c06e83b5216e4138817cc58a61c02e79 P 9efea4678e0e4dee8fe588222a3a6a67 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cfec8b74a2342cca5723d915046f117" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 43833 } } peers { permanent_uuid: "9efea4678e0e4dee8fe588222a3a6a67" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37931 } } peers { permanent_uuid: "8df3306d4bbe4fb1af0afb8d67c81798" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37673 } }
I20240502 21:25:27.056527 16963 leader_election.cc:288] T c06e83b5216e4138817cc58a61c02e79 P 9efea4678e0e4dee8fe588222a3a6a67 [CANDIDATE]: Term 1 election: Requested vote from peers 1cfec8b74a2342cca5723d915046f117 (127.15.255.130:43833), 8df3306d4bbe4fb1af0afb8d67c81798 (127.15.255.131:37673)
I20240502 21:25:27.056860 16699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c06e83b5216e4138817cc58a61c02e79" candidate_uuid: "9efea4678e0e4dee8fe588222a3a6a67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cfec8b74a2342cca5723d915046f117"
I20240502 21:25:27.056901 16838 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c06e83b5216e4138817cc58a61c02e79" candidate_uuid: "9efea4678e0e4dee8fe588222a3a6a67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8df3306d4bbe4fb1af0afb8d67c81798"
I20240502 21:25:27.057042 16699 raft_consensus.cc:3037] T c06e83b5216e4138817cc58a61c02e79 P 1cfec8b74a2342cca5723d915046f117 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:27.057039 16838 raft_consensus.cc:3037] T c06e83b5216e4138817cc58a61c02e79 P 8df3306d4bbe4fb1af0afb8d67c81798 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:27.058723 16699 raft_consensus.cc:2454] T c06e83b5216e4138817cc58a61c02e79 P 1cfec8b74a2342cca5723d915046f117 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9efea4678e0e4dee8fe588222a3a6a67 in term 1.
I20240502 21:25:27.058746 16838 raft_consensus.cc:2454] T c06e83b5216e4138817cc58a61c02e79 P 8df3306d4bbe4fb1af0afb8d67c81798 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9efea4678e0e4dee8fe588222a3a6a67 in term 1.
I20240502 21:25:27.059083 16506 leader_election.cc:302] T c06e83b5216e4138817cc58a61c02e79 P 9efea4678e0e4dee8fe588222a3a6a67 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1cfec8b74a2342cca5723d915046f117, 8df3306d4bbe4fb1af0afb8d67c81798, 9efea4678e0e4dee8fe588222a3a6a67; no voters:
I20240502 21:25:27.059353 16963 raft_consensus.cc:2781] T c06e83b5216e4138817cc58a61c02e79 P 9efea4678e0e4dee8fe588222a3a6a67 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:25:27.059513 16963 raft_consensus.cc:686] T c06e83b5216e4138817cc58a61c02e79 P 9efea4678e0e4dee8fe588222a3a6a67 [term 1 LEADER]: Becoming Leader. State: Replica: 9efea4678e0e4dee8fe588222a3a6a67, State: Running, Role: LEADER
I20240502 21:25:27.059754 16963 consensus_queue.cc:237] T c06e83b5216e4138817cc58a61c02e79 P 9efea4678e0e4dee8fe588222a3a6a67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cfec8b74a2342cca5723d915046f117" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 43833 } } peers { permanent_uuid: "9efea4678e0e4dee8fe588222a3a6a67" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37931 } } peers { permanent_uuid: "8df3306d4bbe4fb1af0afb8d67c81798" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37673 } }
I20240502 21:25:27.062136 16426 catalog_manager.cc:5484] T c06e83b5216e4138817cc58a61c02e79 P 9efea4678e0e4dee8fe588222a3a6a67 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9efea4678e0e4dee8fe588222a3a6a67 (127.15.255.129). New cstate: current_term: 1 leader_uuid: "9efea4678e0e4dee8fe588222a3a6a67" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cfec8b74a2342cca5723d915046f117" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 43833 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9efea4678e0e4dee8fe588222a3a6a67" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37931 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8df3306d4bbe4fb1af0afb8d67c81798" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37673 } health_report { overall_health: UNKNOWN } } }
I20240502 21:25:27.115293 16974 raft_consensus.cc:482] T 1b9f6d5e769e48f394fe59a835c22fc0 P 8df3306d4bbe4fb1af0afb8d67c81798 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:25:27.115474 16974 raft_consensus.cc:504] T 1b9f6d5e769e48f394fe59a835c22fc0 P 8df3306d4bbe4fb1af0afb8d67c81798 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9efea4678e0e4dee8fe588222a3a6a67" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37931 } } peers { permanent_uuid: "1cfec8b74a2342cca5723d915046f117" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 43833 } } peers { permanent_uuid: "8df3306d4bbe4fb1af0afb8d67c81798" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37673 } }
I20240502 21:25:27.116055 16974 leader_election.cc:288] T 1b9f6d5e769e48f394fe59a835c22fc0 P 8df3306d4bbe4fb1af0afb8d67c81798 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9efea4678e0e4dee8fe588222a3a6a67 (127.15.255.129:37931), 1cfec8b74a2342cca5723d915046f117 (127.15.255.130:43833)
I20240502 21:25:27.116503 16569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b9f6d5e769e48f394fe59a835c22fc0" candidate_uuid: "8df3306d4bbe4fb1af0afb8d67c81798" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9efea4678e0e4dee8fe588222a3a6a67" is_pre_election: true
I20240502 21:25:27.116561 16699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b9f6d5e769e48f394fe59a835c22fc0" candidate_uuid: "8df3306d4bbe4fb1af0afb8d67c81798" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cfec8b74a2342cca5723d915046f117" is_pre_election: true
I20240502 21:25:27.116740 16569 raft_consensus.cc:2379] T 1b9f6d5e769e48f394fe59a835c22fc0 P 9efea4678e0e4dee8fe588222a3a6a67 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8df3306d4bbe4fb1af0afb8d67c81798 in current term 1: Already voted for candidate 1cfec8b74a2342cca5723d915046f117 in this term.
I20240502 21:25:27.117146 16767 leader_election.cc:302] T 1b9f6d5e769e48f394fe59a835c22fc0 P 8df3306d4bbe4fb1af0afb8d67c81798 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8df3306d4bbe4fb1af0afb8d67c81798; no voters: 1cfec8b74a2342cca5723d915046f117, 9efea4678e0e4dee8fe588222a3a6a67
I20240502 21:25:27.117413 16974 raft_consensus.cc:3037] T 1b9f6d5e769e48f394fe59a835c22fc0 P 8df3306d4bbe4fb1af0afb8d67c81798 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:27.119196 16974 raft_consensus.cc:2726] T 1b9f6d5e769e48f394fe59a835c22fc0 P 8df3306d4bbe4fb1af0afb8d67c81798 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 21:25:27.125655 16967 raft_consensus.cc:482] T 4cfc1f2146dd449c87a46196e25280c2 P 1cfec8b74a2342cca5723d915046f117 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:25:27.125806 16967 raft_consensus.cc:504] T 4cfc1f2146dd449c87a46196e25280c2 P 1cfec8b74a2342cca5723d915046f117 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cfec8b74a2342cca5723d915046f117" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 43833 } } peers { permanent_uuid: "9efea4678e0e4dee8fe588222a3a6a67" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37931 } } peers { permanent_uuid: "8df3306d4bbe4fb1af0afb8d67c81798" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37673 } }
I20240502 21:25:27.126298 16967 leader_election.cc:288] T 4cfc1f2146dd449c87a46196e25280c2 P 1cfec8b74a2342cca5723d915046f117 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9efea4678e0e4dee8fe588222a3a6a67 (127.15.255.129:37931), 8df3306d4bbe4fb1af0afb8d67c81798 (127.15.255.131:37673)
I20240502 21:25:27.126726 16838 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cfc1f2146dd449c87a46196e25280c2" candidate_uuid: "1cfec8b74a2342cca5723d915046f117" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8df3306d4bbe4fb1af0afb8d67c81798" is_pre_election: true
I20240502 21:25:27.126742 16569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cfc1f2146dd449c87a46196e25280c2" candidate_uuid: "1cfec8b74a2342cca5723d915046f117" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9efea4678e0e4dee8fe588222a3a6a67" is_pre_election: true
I20240502 21:25:27.126901 16838 raft_consensus.cc:2454] T 4cfc1f2146dd449c87a46196e25280c2 P 8df3306d4bbe4fb1af0afb8d67c81798 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cfec8b74a2342cca5723d915046f117 in term 0.
I20240502 21:25:27.126906 16569 raft_consensus.cc:2454] T 4cfc1f2146dd449c87a46196e25280c2 P 9efea4678e0e4dee8fe588222a3a6a67 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cfec8b74a2342cca5723d915046f117 in term 0.
I20240502 21:25:27.127249 16635 leader_election.cc:302] T 4cfc1f2146dd449c87a46196e25280c2 P 1cfec8b74a2342cca5723d915046f117 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cfec8b74a2342cca5723d915046f117, 9efea4678e0e4dee8fe588222a3a6a67; no voters:
I20240502 21:25:27.127525 16967 raft_consensus.cc:2781] T 4cfc1f2146dd449c87a46196e25280c2 P 1cfec8b74a2342cca5723d915046f117 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:25:27.127604 16967 raft_consensus.cc:482] T 4cfc1f2146dd449c87a46196e25280c2 P 1cfec8b74a2342cca5723d915046f117 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:25:27.127674 16967 raft_consensus.cc:3037] T 4cfc1f2146dd449c87a46196e25280c2 P 1cfec8b74a2342cca5723d915046f117 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:27.129480 16967 raft_consensus.cc:504] T 4cfc1f2146dd449c87a46196e25280c2 P 1cfec8b74a2342cca5723d915046f117 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cfec8b74a2342cca5723d915046f117" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 43833 } } peers { permanent_uuid: "9efea4678e0e4dee8fe588222a3a6a67" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37931 } } peers { permanent_uuid: "8df3306d4bbe4fb1af0afb8d67c81798" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37673 } }
I20240502 21:25:27.129992 16967 leader_election.cc:288] T 4cfc1f2146dd449c87a46196e25280c2 P 1cfec8b74a2342cca5723d915046f117 [CANDIDATE]: Term 1 election: Requested vote from peers 9efea4678e0e4dee8fe588222a3a6a67 (127.15.255.129:37931), 8df3306d4bbe4fb1af0afb8d67c81798 (127.15.255.131:37673)
I20240502 21:25:27.130337 16569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cfc1f2146dd449c87a46196e25280c2" candidate_uuid: "1cfec8b74a2342cca5723d915046f117" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9efea4678e0e4dee8fe588222a3a6a67"
I20240502 21:25:27.130402 16838 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cfc1f2146dd449c87a46196e25280c2" candidate_uuid: "1cfec8b74a2342cca5723d915046f117" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8df3306d4bbe4fb1af0afb8d67c81798"
I20240502 21:25:27.130544 16569 raft_consensus.cc:3037] T 4cfc1f2146dd449c87a46196e25280c2 P 9efea4678e0e4dee8fe588222a3a6a67 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:27.130568 16838 raft_consensus.cc:3037] T 4cfc1f2146dd449c87a46196e25280c2 P 8df3306d4bbe4fb1af0afb8d67c81798 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:27.133220 16838 raft_consensus.cc:2454] T 4cfc1f2146dd449c87a46196e25280c2 P 8df3306d4bbe4fb1af0afb8d67c81798 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cfec8b74a2342cca5723d915046f117 in term 1.
I20240502 21:25:27.133235 16569 raft_consensus.cc:2454] T 4cfc1f2146dd449c87a46196e25280c2 P 9efea4678e0e4dee8fe588222a3a6a67 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cfec8b74a2342cca5723d915046f117 in term 1.
I20240502 21:25:27.133623 16635 leader_election.cc:302] T 4cfc1f2146dd449c87a46196e25280c2 P 1cfec8b74a2342cca5723d915046f117 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cfec8b74a2342cca5723d915046f117, 9efea4678e0e4dee8fe588222a3a6a67; no voters:
I20240502 21:25:27.133859 16967 raft_consensus.cc:2781] T 4cfc1f2146dd449c87a46196e25280c2 P 1cfec8b74a2342cca5723d915046f117 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:25:27.134037 16967 raft_consensus.cc:686] T 4cfc1f2146dd449c87a46196e25280c2 P 1cfec8b74a2342cca5723d915046f117 [term 1 LEADER]: Becoming Leader. State: Replica: 1cfec8b74a2342cca5723d915046f117, State: Running, Role: LEADER
I20240502 21:25:27.134294 16967 consensus_queue.cc:237] T 4cfc1f2146dd449c87a46196e25280c2 P 1cfec8b74a2342cca5723d915046f117 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cfec8b74a2342cca5723d915046f117" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 43833 } } peers { permanent_uuid: "9efea4678e0e4dee8fe588222a3a6a67" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37931 } } peers { permanent_uuid: "8df3306d4bbe4fb1af0afb8d67c81798" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37673 } }
I20240502 21:25:27.136749 16426 catalog_manager.cc:5484] T 4cfc1f2146dd449c87a46196e25280c2 P 1cfec8b74a2342cca5723d915046f117 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1cfec8b74a2342cca5723d915046f117 (127.15.255.130). New cstate: current_term: 1 leader_uuid: "1cfec8b74a2342cca5723d915046f117" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cfec8b74a2342cca5723d915046f117" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 43833 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9efea4678e0e4dee8fe588222a3a6a67" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37931 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8df3306d4bbe4fb1af0afb8d67c81798" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37673 } health_report { overall_health: UNKNOWN } } }
I20240502 21:25:27.152980 16967 consensus_queue.cc:1035] T 1b9f6d5e769e48f394fe59a835c22fc0 P 1cfec8b74a2342cca5723d915046f117 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9efea4678e0e4dee8fe588222a3a6a67" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:25:27.159022 16967 consensus_queue.cc:1035] T 1b9f6d5e769e48f394fe59a835c22fc0 P 1cfec8b74a2342cca5723d915046f117 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8df3306d4bbe4fb1af0afb8d67c81798" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:25:27.214148 16967 raft_consensus.cc:482] T 62a4fb5953dc4fa7badc5e2ee1ed4261 P 1cfec8b74a2342cca5723d915046f117 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:25:27.214318 16967 raft_consensus.cc:504] T 62a4fb5953dc4fa7badc5e2ee1ed4261 P 1cfec8b74a2342cca5723d915046f117 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9efea4678e0e4dee8fe588222a3a6a67" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37931 } } peers { permanent_uuid: "1cfec8b74a2342cca5723d915046f117" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 43833 } } peers { permanent_uuid: "8df3306d4bbe4fb1af0afb8d67c81798" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37673 } }
I20240502 21:25:27.214921 16967 leader_election.cc:288] T 62a4fb5953dc4fa7badc5e2ee1ed4261 P 1cfec8b74a2342cca5723d915046f117 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9efea4678e0e4dee8fe588222a3a6a67 (127.15.255.129:37931), 8df3306d4bbe4fb1af0afb8d67c81798 (127.15.255.131:37673)
I20240502 21:25:27.215378 16569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62a4fb5953dc4fa7badc5e2ee1ed4261" candidate_uuid: "1cfec8b74a2342cca5723d915046f117" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9efea4678e0e4dee8fe588222a3a6a67" is_pre_election: true
I20240502 21:25:27.215421 16838 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62a4fb5953dc4fa7badc5e2ee1ed4261" candidate_uuid: "1cfec8b74a2342cca5723d915046f117" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8df3306d4bbe4fb1af0afb8d67c81798" is_pre_election: true
I20240502 21:25:27.215696 16838 raft_consensus.cc:2379] T 62a4fb5953dc4fa7badc5e2ee1ed4261 P 8df3306d4bbe4fb1af0afb8d67c81798 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1cfec8b74a2342cca5723d915046f117 in current term 1: Already voted for candidate 9efea4678e0e4dee8fe588222a3a6a67 in this term.
I20240502 21:25:27.216115 16636 leader_election.cc:302] T 62a4fb5953dc4fa7badc5e2ee1ed4261 P 1cfec8b74a2342cca5723d915046f117 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cfec8b74a2342cca5723d915046f117; no voters: 8df3306d4bbe4fb1af0afb8d67c81798, 9efea4678e0e4dee8fe588222a3a6a67
I20240502 21:25:27.216380 16967 raft_consensus.cc:3037] T 62a4fb5953dc4fa7badc5e2ee1ed4261 P 1cfec8b74a2342cca5723d915046f117 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:27.218240 16967 raft_consensus.cc:2726] T 62a4fb5953dc4fa7badc5e2ee1ed4261 P 1cfec8b74a2342cca5723d915046f117 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 21:25:27.233902 16963 consensus_queue.cc:1035] T 05a7626e52ae43a6829feacaf1837ea5 P 9efea4678e0e4dee8fe588222a3a6a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cfec8b74a2342cca5723d915046f117" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 43833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:25:27.236822 16966 consensus_queue.cc:1035] T 05a7626e52ae43a6829feacaf1837ea5 P 9efea4678e0e4dee8fe588222a3a6a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8df3306d4bbe4fb1af0afb8d67c81798" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:25:27.244578 16967 consensus_queue.cc:1035] T 610bb5f6731047b3b86883445a9e05bb P 1cfec8b74a2342cca5723d915046f117 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8df3306d4bbe4fb1af0afb8d67c81798" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:25:27.248855 16929 consensus_queue.cc:1035] T 610bb5f6731047b3b86883445a9e05bb P 1cfec8b74a2342cca5723d915046f117 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9efea4678e0e4dee8fe588222a3a6a67" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:25:27.252924 16929 consensus_queue.cc:1035] T c5ea5b6dcf934533ba8df41e52711337 P 1cfec8b74a2342cca5723d915046f117 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8df3306d4bbe4fb1af0afb8d67c81798" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:25:27.256992 16929 consensus_queue.cc:1035] T c5ea5b6dcf934533ba8df41e52711337 P 1cfec8b74a2342cca5723d915046f117 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9efea4678e0e4dee8fe588222a3a6a67" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:25:27.284734 16963 consensus_queue.cc:1035] T e882da2f771249f8a3626d1a7a35de34 P 9efea4678e0e4dee8fe588222a3a6a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cfec8b74a2342cca5723d915046f117" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 43833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:25:27.288383 16916 consensus_queue.cc:1035] T e882da2f771249f8a3626d1a7a35de34 P 9efea4678e0e4dee8fe588222a3a6a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8df3306d4bbe4fb1af0afb8d67c81798" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:25:27.323908 16964 consensus_queue.cc:1035] T 1173ce275c4f40f0a0a31729310fbefd P 9efea4678e0e4dee8fe588222a3a6a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8df3306d4bbe4fb1af0afb8d67c81798" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:25:27.325541 16907 consensus_queue.cc:1035] T 71d05c4f1e6f4f7186ed7b8e157b7d2f P 1cfec8b74a2342cca5723d915046f117 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8df3306d4bbe4fb1af0afb8d67c81798" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:25:27.329039 16929 consensus_queue.cc:1035] T 2b1fbeb84317494595f7262fe7857ebd P 1cfec8b74a2342cca5723d915046f117 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9efea4678e0e4dee8fe588222a3a6a67" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:25:27.329176 16964 consensus_queue.cc:1035] T 1173ce275c4f40f0a0a31729310fbefd P 9efea4678e0e4dee8fe588222a3a6a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cfec8b74a2342cca5723d915046f117" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 43833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:25:27.333549 16948 raft_consensus.cc:482] T 38999367825249e1a0f5f156bd3c3b25 P 1cfec8b74a2342cca5723d915046f117 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:25:27.333717 16948 raft_consensus.cc:504] T 38999367825249e1a0f5f156bd3c3b25 P 1cfec8b74a2342cca5723d915046f117 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9efea4678e0e4dee8fe588222a3a6a67" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37931 } } peers { permanent_uuid: "1cfec8b74a2342cca5723d915046f117" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 43833 } } peers { permanent_uuid: "8df3306d4bbe4fb1af0afb8d67c81798" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37673 } }
I20240502 21:25:27.334374 16948 leader_election.cc:288] T 38999367825249e1a0f5f156bd3c3b25 P 1cfec8b74a2342cca5723d915046f117 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9efea4678e0e4dee8fe588222a3a6a67 (127.15.255.129:37931), 8df3306d4bbe4fb1af0afb8d67c81798 (127.15.255.131:37673)
I20240502 21:25:27.335296 16568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38999367825249e1a0f5f156bd3c3b25" candidate_uuid: "1cfec8b74a2342cca5723d915046f117" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9efea4678e0e4dee8fe588222a3a6a67" is_pre_election: true
I20240502 21:25:27.335620 16568 raft_consensus.cc:2379] T 38999367825249e1a0f5f156bd3c3b25 P 9efea4678e0e4dee8fe588222a3a6a67 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1cfec8b74a2342cca5723d915046f117 in current term 1: Already voted for candidate 8df3306d4bbe4fb1af0afb8d67c81798 in this term.
I20240502 21:25:27.335908 16837 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38999367825249e1a0f5f156bd3c3b25" candidate_uuid: "1cfec8b74a2342cca5723d915046f117" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8df3306d4bbe4fb1af0afb8d67c81798" is_pre_election: true
I20240502 21:25:27.336673 16948 consensus_queue.cc:1035] T 2b1fbeb84317494595f7262fe7857ebd P 1cfec8b74a2342cca5723d915046f117 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8df3306d4bbe4fb1af0afb8d67c81798" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:25:27.337109 16948 consensus_queue.cc:1035] T 71d05c4f1e6f4f7186ed7b8e157b7d2f P 1cfec8b74a2342cca5723d915046f117 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9efea4678e0e4dee8fe588222a3a6a67" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:25:27.338557 16636 leader_election.cc:302] T 38999367825249e1a0f5f156bd3c3b25 P 1cfec8b74a2342cca5723d915046f117 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cfec8b74a2342cca5723d915046f117; no voters: 8df3306d4bbe4fb1af0afb8d67c81798, 9efea4678e0e4dee8fe588222a3a6a67
I20240502 21:25:27.339390 16966 consensus_queue.cc:1035] T 62a4fb5953dc4fa7badc5e2ee1ed4261 P 9efea4678e0e4dee8fe588222a3a6a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8df3306d4bbe4fb1af0afb8d67c81798" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:25:27.342125 16967 raft_consensus.cc:3037] T 38999367825249e1a0f5f156bd3c3b25 P 1cfec8b74a2342cca5723d915046f117 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:27.349241 16967 raft_consensus.cc:2726] T 38999367825249e1a0f5f156bd3c3b25 P 1cfec8b74a2342cca5723d915046f117 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 21:25:27.354487 16966 consensus_queue.cc:1035] T 62a4fb5953dc4fa7badc5e2ee1ed4261 P 9efea4678e0e4dee8fe588222a3a6a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cfec8b74a2342cca5723d915046f117" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 43833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:25:27.362145 16974 consensus_queue.cc:1035] T cb358d933789476898f74949ffa81e49 P 8df3306d4bbe4fb1af0afb8d67c81798 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9efea4678e0e4dee8fe588222a3a6a67" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:25:27.372009 16966 consensus_queue.cc:1035] T ed40aed15ca3447aa01283771ac257ed P 9efea4678e0e4dee8fe588222a3a6a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cfec8b74a2342cca5723d915046f117" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 43833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:25:27.374472 16925 raft_consensus.cc:482] T 61268934b6b74fad90db0a52dd48ac82 P 9efea4678e0e4dee8fe588222a3a6a67 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:25:27.374668 16925 raft_consensus.cc:504] T 61268934b6b74fad90db0a52dd48ac82 P 9efea4678e0e4dee8fe588222a3a6a67 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cfec8b74a2342cca5723d915046f117" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 43833 } } peers { permanent_uuid: "9efea4678e0e4dee8fe588222a3a6a67" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37931 } } peers { permanent_uuid: "8df3306d4bbe4fb1af0afb8d67c81798" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37673 } }
I20240502 21:25:27.375377 16925 leader_election.cc:288] T 61268934b6b74fad90db0a52dd48ac82 P 9efea4678e0e4dee8fe588222a3a6a67 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cfec8b74a2342cca5723d915046f117 (127.15.255.130:43833), 8df3306d4bbe4fb1af0afb8d67c81798 (127.15.255.131:37673)
I20240502 21:25:27.376524 16836 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61268934b6b74fad90db0a52dd48ac82" candidate_uuid: "9efea4678e0e4dee8fe588222a3a6a67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8df3306d4bbe4fb1af0afb8d67c81798" is_pre_election: true
I20240502 21:25:27.376802 16836 raft_consensus.cc:2379] T 61268934b6b74fad90db0a52dd48ac82 P 8df3306d4bbe4fb1af0afb8d67c81798 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9efea4678e0e4dee8fe588222a3a6a67 in current term 1: Already voted for candidate 1cfec8b74a2342cca5723d915046f117 in this term.
I20240502 21:25:27.377318 16964 consensus_queue.cc:1035] T ed40aed15ca3447aa01283771ac257ed P 9efea4678e0e4dee8fe588222a3a6a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8df3306d4bbe4fb1af0afb8d67c81798" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:25:27.378621 16702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61268934b6b74fad90db0a52dd48ac82" candidate_uuid: "9efea4678e0e4dee8fe588222a3a6a67" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cfec8b74a2342cca5723d915046f117" is_pre_election: true
I20240502 21:25:27.379281 16506 leader_election.cc:302] T 61268934b6b74fad90db0a52dd48ac82 P 9efea4678e0e4dee8fe588222a3a6a67 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9efea4678e0e4dee8fe588222a3a6a67; no voters: 1cfec8b74a2342cca5723d915046f117, 8df3306d4bbe4fb1af0afb8d67c81798
I20240502 21:25:27.379864 16925 raft_consensus.cc:3037] T 61268934b6b74fad90db0a52dd48ac82 P 9efea4678e0e4dee8fe588222a3a6a67 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:27.382162 16974 consensus_queue.cc:1035] T cb358d933789476898f74949ffa81e49 P 8df3306d4bbe4fb1af0afb8d67c81798 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cfec8b74a2342cca5723d915046f117" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 43833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:25:27.387391 16974 consensus_queue.cc:1035] T 38999367825249e1a0f5f156bd3c3b25 P 8df3306d4bbe4fb1af0afb8d67c81798 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cfec8b74a2342cca5723d915046f117" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 43833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:25:27.402979 16925 raft_consensus.cc:2726] T 61268934b6b74fad90db0a52dd48ac82 P 9efea4678e0e4dee8fe588222a3a6a67 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 21:25:27.414702 16974 consensus_queue.cc:1035] T 38999367825249e1a0f5f156bd3c3b25 P 8df3306d4bbe4fb1af0afb8d67c81798 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9efea4678e0e4dee8fe588222a3a6a67" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:25:27.464773 16929 consensus_queue.cc:1035] T 61268934b6b74fad90db0a52dd48ac82 P 1cfec8b74a2342cca5723d915046f117 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9efea4678e0e4dee8fe588222a3a6a67" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:25:27.465879 16929 consensus_queue.cc:1035] T 61268934b6b74fad90db0a52dd48ac82 P 1cfec8b74a2342cca5723d915046f117 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8df3306d4bbe4fb1af0afb8d67c81798" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:25:27.503742 16974 consensus_queue.cc:1035] T 72f6a167c03d4ee6bd72d6e5793f784d P 8df3306d4bbe4fb1af0afb8d67c81798 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9efea4678e0e4dee8fe588222a3a6a67" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:25:27.507972 16978 consensus_queue.cc:1035] T 72f6a167c03d4ee6bd72d6e5793f784d P 8df3306d4bbe4fb1af0afb8d67c81798 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cfec8b74a2342cca5723d915046f117" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 43833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:25:27.517969 16925 consensus_queue.cc:1035] T c06e83b5216e4138817cc58a61c02e79 P 9efea4678e0e4dee8fe588222a3a6a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8df3306d4bbe4fb1af0afb8d67c81798" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:25:27.521991 16966 consensus_queue.cc:1035] T c06e83b5216e4138817cc58a61c02e79 P 9efea4678e0e4dee8fe588222a3a6a67 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cfec8b74a2342cca5723d915046f117" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 43833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:25:27.576205 16948 consensus_queue.cc:1035] T 4cfc1f2146dd449c87a46196e25280c2 P 1cfec8b74a2342cca5723d915046f117 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9efea4678e0e4dee8fe588222a3a6a67" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:25:27.586298 16929 consensus_queue.cc:1035] T 4cfc1f2146dd449c87a46196e25280c2 P 1cfec8b74a2342cca5723d915046f117 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8df3306d4bbe4fb1af0afb8d67c81798" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 21:25:27.635609 17006 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:25:27.635902 17006 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:25:27.643278 17006 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_0'
INSERT report
rows total: 2000
time total: 321.875 ms
time per row: 0.160938 ms
I20240502 21:25:28.575388 16682 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 21:25:28.593211 16543 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:25:28.596550 16806 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:25:28.766862 16682 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 21:25:28.776593 16806 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:25:28.815131 16543 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:25:28.988468 16682 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 21:25:28.992044 16543 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:25:28.992254 16806 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:25:29.586002 16682 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 21:25:29.589246 16806 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:25:29.600225 16543 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:25:29.750804 16682 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 21:25:29.757498 16806 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:25:29.765206 16543 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:25:29.896322 16382 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvNciQZ/build/asan/bin/kudu pid 16486
I20240502 21:25:29.902357 16612 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 21:25:30.023414 16382 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvNciQZ/build/asan/bin/kudu with pid 16486
W20240502 21:25:30.041586 16637 connection.cc:535] server connection from 127.15.255.129:43797 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 21:25:30.041586 16771 connection.cc:535] server connection from 127.15.255.129:35723 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 21:25:30.041978 16382 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvNciQZ/build/asan/bin/kudu pid 16620
W20240502 21:25:30.042039 16767 connection.cc:535] client connection to 127.15.255.129:37931 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 21:25:30.042124 16409 connection.cc:535] server connection from 127.15.255.129:38673 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 21:25:30.042305 16767 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 21:25:30.044200 16767 consensus_peers.cc:485] T 38999367825249e1a0f5f156bd3c3b25 P 8df3306d4bbe4fb1af0afb8d67c81798 -> Peer 9efea4678e0e4dee8fe588222a3a6a67 (127.15.255.129:37931): Couldn't send request to peer 9efea4678e0e4dee8fe588222a3a6a67. Status: Network error: Client connection negotiation failed: client connection to 127.15.255.129:37931: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 21:25:30.047775 16744 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 21:25:30.166574 16382 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvNciQZ/build/asan/bin/kudu with pid 16620
W20240502 21:25:30.184222 16767 connection.cc:535] server connection from 127.15.255.130:54513 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 21:25:30.184576 16382 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvNciQZ/build/asan/bin/kudu pid 16752
I20240502 21:25:30.190114 16877 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 21:25:30.319337 16382 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvNciQZ/build/asan/bin/kudu with pid 16752
I20240502 21:25:30.336828 16382 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvNciQZ/build/asan/bin/kudu pid 16394
I20240502 21:25:30.337834 16457 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 21:25:30.453565 16382 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvNciQZ/build/asan/bin/kudu with pid 16394
2024-05-02T21:25:30Z chronyd exiting
[ OK ] TableRangeRebalancingTest.Basic (9737 ms)
[----------] 1 test from TableRangeRebalancingTest (9737 ms total)
[----------] 1 test from RebalanceStartSafetyTest
[ RUN ] RebalanceStartSafetyTest.TooManyIgnoredTservers/1
I20240502 21:25:30.501129 16382 test_util.cc:268] Using random seed: 1041951732
I20240502 21:25:30.502842 16382 ts_itest-base.cc:112] Starting cluster with:
I20240502 21:25:30.502902 16382 ts_itest-base.cc:113] --------------
I20240502 21:25:30.502943 16382 ts_itest-base.cc:114] 5 tablet servers
I20240502 21:25:30.502983 16382 ts_itest-base.cc:115] 3 replicas per TS
I20240502 21:25:30.503031 16382 ts_itest-base.cc:116] --------------
2024-05-02T21:25:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-02T21:25:30Z Disabled control of system clock
I20240502 21:25:30.524341 16382 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvNciQZ/build/asan/bin/kudu
/tmp/dist-test-taskvNciQZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714685120732928-16382-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714685120732928-16382-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714685120732928-16382-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714685120732928-16382-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.255.190:41421
--webserver_interface=127.15.255.190
--webserver_port=0
--builtin_ntp_servers=127.15.255.148:35553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.255.190:41421
--raft_prepare_replacement_before_eviction=true
--enable_range_replica_placement=false with env {}
W20240502 21:25:30.821415 17279 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:25:30.821694 17279 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:25:30.821825 17279 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:25:30.828819 17279 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 21:25:30.828933 17279 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240502 21:25:30.829000 17279 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:25:30.829080 17279 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240502 21:25:30.829140 17279 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240502 21:25:30.837934 17279 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.255.148:35553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714685120732928-16382-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714685120732928-16382-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.15.255.190:41421
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.255.190:41421
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714685120732928-16382-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.15.255.190
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714685120732928-16382-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:25:30.839043 17279 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:25:30.840662 17279 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:25:30.850800 17284 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:30.851215 17287 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:30.851848 17285 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:25:30.852468 17279 server_base.cc:1034] running on GCE node
I20240502 21:25:30.853124 17279 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:25:30.854308 17279 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:25:30.855507 17279 hybrid_clock.cc:648] HybridClock initialized: now 1714685130855471 us; error 28 us; skew 500 ppm
I20240502 21:25:30.855791 17279 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:25:30.857765 17279 webserver.cc:460] Webserver started at http://127.15.255.190:39231/ using document root <none> and password file <none>
I20240502 21:25:30.858369 17279 fs_manager.cc:359] Metadata directory not provided
I20240502 21:25:30.858459 17279 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:25:30.858719 17279 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:25:30.861065 17279 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714685120732928-16382-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "82c7193a4856498580c36a7dc896b35d"
format_stamp: "Formatted at 2024-05-02 21:25:30 on dist-test-slave-jdfx"
I20240502 21:25:30.861797 17279 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714685120732928-16382-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "82c7193a4856498580c36a7dc896b35d"
format_stamp: "Formatted at 2024-05-02 21:25:30 on dist-test-slave-jdfx"
I20240502 21:25:30.866328 17279 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20240502 21:25:30.869199 17293 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:30.870326 17279 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240502 21:25:30.870478 17279 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714685120732928-16382-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714685120732928-16382-0/raft_consensus-itest-cluster/master-0/wal
uuid: "82c7193a4856498580c36a7dc896b35d"
format_stamp: "Formatted at 2024-05-02 21:25:30 on dist-test-slave-jdfx"
I20240502 21:25:30.870651 17279 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714685120732928-16382-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714685120732928-16382-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714685120732928-16382-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:25:30.890574 17279 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:25:30.891244 17279 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:25:30.891482 17279 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:25:30.906483 17279 rpc_server.cc:307] RPC server started. Bound to: 127.15.255.190:41421
I20240502 21:25:30.906507 17346 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.255.190:41421 every 8 connection(s)
I20240502 21:25:30.907637 17279 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714685120732928-16382-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240502 21:25:30.910929 17347 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:30.916622 16382 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvNciQZ/build/asan/bin/kudu as pid 17279
I20240502 21:25:30.916827 16382 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714685120732928-16382-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240502 21:25:30.920140 17347 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 82c7193a4856498580c36a7dc896b35d: Bootstrap starting.
I20240502 21:25:30.922883 17347 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 82c7193a4856498580c36a7dc896b35d: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:30.923822 17347 log.cc:824] T 00000000000000000000000000000000 P 82c7193a4856498580c36a7dc896b35d: Log is configured to *not* fsync() on all Append() calls
I20240502 21:25:30.925985 17347 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 82c7193a4856498580c36a7dc896b35d: No bootstrap required, opened a new log
I20240502 21:25:30.932473 17347 raft_consensus.cc:348] T 00000000000000000000000000000000 P 82c7193a4856498580c36a7dc896b35d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82c7193a4856498580c36a7dc896b35d" member_type: VOTER last_known_addr { host: "127.15.255.190" port: 41421 } }
I20240502 21:25:30.932754 17347 raft_consensus.cc:374] T 00000000000000000000000000000000 P 82c7193a4856498580c36a7dc896b35d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:30.932843 17347 raft_consensus.cc:729] T 00000000000000000000000000000000 P 82c7193a4856498580c36a7dc896b35d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82c7193a4856498580c36a7dc896b35d, State: Initialized, Role: FOLLOWER
I20240502 21:25:30.933388 17347 consensus_queue.cc:260] T 00000000000000000000000000000000 P 82c7193a4856498580c36a7dc896b35d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82c7193a4856498580c36a7dc896b35d" member_type: VOTER last_known_addr { host: "127.15.255.190" port: 41421 } }
I20240502 21:25:30.933571 17347 raft_consensus.cc:388] T 00000000000000000000000000000000 P 82c7193a4856498580c36a7dc896b35d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 21:25:30.933688 17347 raft_consensus.cc:482] T 00000000000000000000000000000000 P 82c7193a4856498580c36a7dc896b35d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 21:25:30.933822 17347 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 82c7193a4856498580c36a7dc896b35d [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:30.935820 17347 raft_consensus.cc:504] T 00000000000000000000000000000000 P 82c7193a4856498580c36a7dc896b35d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82c7193a4856498580c36a7dc896b35d" member_type: VOTER last_known_addr { host: "127.15.255.190" port: 41421 } }
I20240502 21:25:30.936244 17347 leader_election.cc:302] T 00000000000000000000000000000000 P 82c7193a4856498580c36a7dc896b35d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 82c7193a4856498580c36a7dc896b35d; no voters:
I20240502 21:25:30.936764 17347 leader_election.cc:288] T 00000000000000000000000000000000 P 82c7193a4856498580c36a7dc896b35d [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 21:25:30.936880 17352 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 82c7193a4856498580c36a7dc896b35d [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:25:30.937436 17352 raft_consensus.cc:686] T 00000000000000000000000000000000 P 82c7193a4856498580c36a7dc896b35d [term 1 LEADER]: Becoming Leader. State: Replica: 82c7193a4856498580c36a7dc896b35d, State: Running, Role: LEADER
I20240502 21:25:30.937893 17352 consensus_queue.cc:237] T 00000000000000000000000000000000 P 82c7193a4856498580c36a7dc896b35d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82c7193a4856498580c36a7dc896b35d" member_type: VOTER last_known_addr { host: "127.15.255.190" port: 41421 } }
I20240502 21:25:30.938372 17347 sys_catalog.cc:564] T 00000000000000000000000000000000 P 82c7193a4856498580c36a7dc896b35d [sys.catalog]: configured and running, proceeding with master startup.
I20240502 21:25:30.940320 17354 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82c7193a4856498580c36a7dc896b35d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 82c7193a4856498580c36a7dc896b35d. Latest consensus state: current_term: 1 leader_uuid: "82c7193a4856498580c36a7dc896b35d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82c7193a4856498580c36a7dc896b35d" member_type: VOTER last_known_addr { host: "127.15.255.190" port: 41421 } } }
I20240502 21:25:30.940562 17353 sys_catalog.cc:455] T 00000000000000000000000000000000 P 82c7193a4856498580c36a7dc896b35d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "82c7193a4856498580c36a7dc896b35d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "82c7193a4856498580c36a7dc896b35d" member_type: VOTER last_known_addr { host: "127.15.255.190" port: 41421 } } }
I20240502 21:25:30.940785 17354 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82c7193a4856498580c36a7dc896b35d [sys.catalog]: This master's current role is: LEADER
I20240502 21:25:30.940872 17353 sys_catalog.cc:458] T 00000000000000000000000000000000 P 82c7193a4856498580c36a7dc896b35d [sys.catalog]: This master's current role is: LEADER
I20240502 21:25:30.942538 17361 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 21:25:30.947512 17361 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 21:25:30.955367 17361 catalog_manager.cc:1347] Generated new cluster ID: b79d7b388fcb45118c0e6b2be5ffd38f
I20240502 21:25:30.955461 17361 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 21:25:30.966265 17361 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 21:25:30.967561 17361 catalog_manager.cc:1504] Loading token signing keys...
I20240502 21:25:30.975960 17361 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 82c7193a4856498580c36a7dc896b35d: Generated new TSK 0
I20240502 21:25:30.976651 17361 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 21:25:30.980634 16382 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvNciQZ/build/asan/bin/kudu
/tmp/dist-test-taskvNciQZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.255.129:0
--local_ip_for_outbound_sockets=127.15.255.129
--webserver_interface=127.15.255.129
--webserver_port=0
--tserver_master_addrs=127.15.255.190:41421
--builtin_ntp_servers=127.15.255.148:35553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240502 21:25:31.281612 17371 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:25:31.281863 17371 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:25:31.281973 17371 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:25:31.289142 17371 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 21:25:31.289249 17371 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:25:31.289393 17371 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.255.129
I20240502 21:25:31.297894 17371 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.255.148:35553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.255.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.15.255.129
--webserver_port=0
--tserver_master_addrs=127.15.255.190:41421
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.255.129
--log_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:25:31.299041 17371 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:25:31.300704 17371 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:25:31.311913 17376 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:31.312397 17378 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:31.312927 17380 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:25:31.313469 17371 server_base.cc:1034] running on GCE node
I20240502 21:25:31.313951 17371 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:25:31.314733 17371 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:25:31.315929 17371 hybrid_clock.cc:648] HybridClock initialized: now 1714685131315883 us; error 29 us; skew 500 ppm
I20240502 21:25:31.316222 17371 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:25:31.318243 17371 webserver.cc:460] Webserver started at http://127.15.255.129:34139/ using document root <none> and password file <none>
I20240502 21:25:31.318887 17371 fs_manager.cc:359] Metadata directory not provided
I20240502 21:25:31.319001 17371 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:25:31.319290 17371 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:25:31.321731 17371 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "4b1b6f344c434a59b099fea2d10fada6"
format_stamp: "Formatted at 2024-05-02 21:25:31 on dist-test-slave-jdfx"
I20240502 21:25:31.322427 17371 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "4b1b6f344c434a59b099fea2d10fada6"
format_stamp: "Formatted at 2024-05-02 21:25:31 on dist-test-slave-jdfx"
I20240502 21:25:31.327041 17371 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20240502 21:25:31.330044 17386 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:31.331207 17371 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240502 21:25:31.331348 17371 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "4b1b6f344c434a59b099fea2d10fada6"
format_stamp: "Formatted at 2024-05-02 21:25:31 on dist-test-slave-jdfx"
I20240502 21:25:31.331521 17371 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:25:31.351272 17371 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:25:31.351951 17371 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:25:31.352186 17371 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:25:31.353035 17371 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:25:31.354326 17371 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:25:31.354416 17371 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:31.354519 17371 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:25:31.354590 17371 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:31.381791 17371 rpc_server.cc:307] RPC server started. Bound to: 127.15.255.129:42727
I20240502 21:25:31.381855 17499 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.255.129:42727 every 8 connection(s)
I20240502 21:25:31.382942 17371 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240502 21:25:31.385237 16382 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvNciQZ/build/asan/bin/kudu as pid 17371
I20240502 21:25:31.385462 16382 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240502 21:25:31.389050 16382 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvNciQZ/build/asan/bin/kudu
/tmp/dist-test-taskvNciQZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.255.130:0
--local_ip_for_outbound_sockets=127.15.255.130
--webserver_interface=127.15.255.130
--webserver_port=0
--tserver_master_addrs=127.15.255.190:41421
--builtin_ntp_servers=127.15.255.148:35553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 21:25:31.395995 17500 heartbeater.cc:347] Connected to a master server at 127.15.255.190:41421
I20240502 21:25:31.396247 17500 heartbeater.cc:464] Registering TS with master...
I20240502 21:25:31.397006 17500 heartbeater.cc:511] Master 127.15.255.190:41421 requested a full tablet report, sending...
I20240502 21:25:31.398831 17312 ts_manager.cc:194] Registered new tserver with Master: 4b1b6f344c434a59b099fea2d10fada6 (127.15.255.129:42727)
I20240502 21:25:31.400313 17312 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.255.129:59027
W20240502 21:25:31.697121 17503 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:25:31.697374 17503 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:25:31.697499 17503 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:25:31.704653 17503 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 21:25:31.704771 17503 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:25:31.704913 17503 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.255.130
I20240502 21:25:31.713291 17503 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.255.148:35553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.255.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.15.255.130
--webserver_port=0
--tserver_master_addrs=127.15.255.190:41421
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.255.130
--log_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:25:31.714455 17503 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:25:31.716094 17503 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:25:31.727452 17508 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:31.727903 17509 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:31.728358 17511 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:25:31.728879 17503 server_base.cc:1034] running on GCE node
I20240502 21:25:31.729415 17503 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:25:31.730247 17503 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:25:31.731443 17503 hybrid_clock.cc:648] HybridClock initialized: now 1714685131731397 us; error 39 us; skew 500 ppm
I20240502 21:25:31.731750 17503 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:25:31.733829 17503 webserver.cc:460] Webserver started at http://127.15.255.130:38235/ using document root <none> and password file <none>
I20240502 21:25:31.734476 17503 fs_manager.cc:359] Metadata directory not provided
I20240502 21:25:31.734580 17503 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:25:31.734882 17503 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:25:31.737424 17503 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "335a85cef17445549f8bcb9a5092366f"
format_stamp: "Formatted at 2024-05-02 21:25:31 on dist-test-slave-jdfx"
I20240502 21:25:31.738111 17503 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "335a85cef17445549f8bcb9a5092366f"
format_stamp: "Formatted at 2024-05-02 21:25:31 on dist-test-slave-jdfx"
I20240502 21:25:31.742870 17503 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20240502 21:25:31.746050 17517 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:31.747211 17503 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240502 21:25:31.747331 17503 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "335a85cef17445549f8bcb9a5092366f"
format_stamp: "Formatted at 2024-05-02 21:25:31 on dist-test-slave-jdfx"
I20240502 21:25:31.747484 17503 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:25:31.773833 17503 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:25:31.774614 17503 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:25:31.774878 17503 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:25:31.775760 17503 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:25:31.777153 17503 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:25:31.777246 17503 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:31.777350 17503 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:25:31.777417 17503 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:31.805424 17503 rpc_server.cc:307] RPC server started. Bound to: 127.15.255.130:33539
I20240502 21:25:31.805497 17635 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.255.130:33539 every 8 connection(s)
I20240502 21:25:31.806625 17503 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240502 21:25:31.814656 16382 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvNciQZ/build/asan/bin/kudu as pid 17503
I20240502 21:25:31.814798 16382 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240502 21:25:31.817857 16382 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvNciQZ/build/asan/bin/kudu
/tmp/dist-test-taskvNciQZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.255.131:0
--local_ip_for_outbound_sockets=127.15.255.131
--webserver_interface=127.15.255.131
--webserver_port=0
--tserver_master_addrs=127.15.255.190:41421
--builtin_ntp_servers=127.15.255.148:35553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 21:25:31.818025 17636 heartbeater.cc:347] Connected to a master server at 127.15.255.190:41421
I20240502 21:25:31.818296 17636 heartbeater.cc:464] Registering TS with master...
I20240502 21:25:31.819232 17636 heartbeater.cc:511] Master 127.15.255.190:41421 requested a full tablet report, sending...
I20240502 21:25:31.820601 17312 ts_manager.cc:194] Registered new tserver with Master: 335a85cef17445549f8bcb9a5092366f (127.15.255.130:33539)
I20240502 21:25:31.821307 17312 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.255.130:60847
W20240502 21:25:32.116986 17639 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:25:32.117230 17639 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:25:32.117352 17639 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:25:32.124751 17639 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 21:25:32.124871 17639 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:25:32.125021 17639 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.255.131
I20240502 21:25:32.133478 17639 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.255.148:35553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.255.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.15.255.131
--webserver_port=0
--tserver_master_addrs=127.15.255.190:41421
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.255.131
--log_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:25:32.134589 17639 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:25:32.136189 17639 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:25:32.147306 17647 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:32.147781 17644 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:32.148270 17645 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:25:32.148833 17639 server_base.cc:1034] running on GCE node
I20240502 21:25:32.149336 17639 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:25:32.150125 17639 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:25:32.151314 17639 hybrid_clock.cc:648] HybridClock initialized: now 1714685132151269 us; error 40 us; skew 500 ppm
I20240502 21:25:32.151597 17639 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:25:32.153576 17639 webserver.cc:460] Webserver started at http://127.15.255.131:36371/ using document root <none> and password file <none>
I20240502 21:25:32.154211 17639 fs_manager.cc:359] Metadata directory not provided
I20240502 21:25:32.154312 17639 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:25:32.154595 17639 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:25:32.156988 17639 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "d1278c0eeece4fe0b5027233f1c1528d"
format_stamp: "Formatted at 2024-05-02 21:25:32 on dist-test-slave-jdfx"
I20240502 21:25:32.157665 17639 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "d1278c0eeece4fe0b5027233f1c1528d"
format_stamp: "Formatted at 2024-05-02 21:25:32 on dist-test-slave-jdfx"
I20240502 21:25:32.162209 17639 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20240502 21:25:32.165235 17653 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:32.166422 17639 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240502 21:25:32.166572 17639 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "d1278c0eeece4fe0b5027233f1c1528d"
format_stamp: "Formatted at 2024-05-02 21:25:32 on dist-test-slave-jdfx"
I20240502 21:25:32.166756 17639 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:25:32.179867 17639 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:25:32.180613 17639 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:25:32.180860 17639 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:25:32.181677 17639 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:25:32.182993 17639 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:25:32.183084 17639 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:32.183184 17639 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:25:32.183250 17639 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:32.211194 17639 rpc_server.cc:307] RPC server started. Bound to: 127.15.255.131:33535
I20240502 21:25:32.211251 17769 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.255.131:33535 every 8 connection(s)
I20240502 21:25:32.212399 17639 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240502 21:25:32.222357 16382 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvNciQZ/build/asan/bin/kudu as pid 17639
I20240502 21:25:32.222554 16382 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240502 21:25:32.223429 17770 heartbeater.cc:347] Connected to a master server at 127.15.255.190:41421
I20240502 21:25:32.223686 17770 heartbeater.cc:464] Registering TS with master...
I20240502 21:25:32.224438 17770 heartbeater.cc:511] Master 127.15.255.190:41421 requested a full tablet report, sending...
I20240502 21:25:32.225585 16382 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvNciQZ/build/asan/bin/kudu
/tmp/dist-test-taskvNciQZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.255.132:0
--local_ip_for_outbound_sockets=127.15.255.132
--webserver_interface=127.15.255.132
--webserver_port=0
--tserver_master_addrs=127.15.255.190:41421
--builtin_ntp_servers=127.15.255.148:35553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 21:25:32.225780 17312 ts_manager.cc:194] Registered new tserver with Master: d1278c0eeece4fe0b5027233f1c1528d (127.15.255.131:33535)
I20240502 21:25:32.226742 17312 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.255.131:57711
I20240502 21:25:32.402887 17500 heartbeater.cc:503] Master 127.15.255.190:41421 was elected leader, sending a full tablet report...
W20240502 21:25:32.527484 17773 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:25:32.527757 17773 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:25:32.527889 17773 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:25:32.535159 17773 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 21:25:32.535272 17773 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:25:32.535420 17773 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.255.132
I20240502 21:25:32.544083 17773 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.255.148:35553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.255.132:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.15.255.132
--webserver_port=0
--tserver_master_addrs=127.15.255.190:41421
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.255.132
--log_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:25:32.545295 17773 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:25:32.546943 17773 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:25:32.558347 17779 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:32.558877 17778 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:32.559257 17781 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:25:32.559739 17773 server_base.cc:1034] running on GCE node
I20240502 21:25:32.560213 17773 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:25:32.561030 17773 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:25:32.562216 17773 hybrid_clock.cc:648] HybridClock initialized: now 1714685132562171 us; error 36 us; skew 500 ppm
I20240502 21:25:32.562494 17773 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:25:32.564517 17773 webserver.cc:460] Webserver started at http://127.15.255.132:46415/ using document root <none> and password file <none>
I20240502 21:25:32.565152 17773 fs_manager.cc:359] Metadata directory not provided
I20240502 21:25:32.565253 17773 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:25:32.565539 17773 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:25:32.567893 17773 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "7d4fdafe6e704a8889dd12240f102bda"
format_stamp: "Formatted at 2024-05-02 21:25:32 on dist-test-slave-jdfx"
I20240502 21:25:32.568637 17773 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "7d4fdafe6e704a8889dd12240f102bda"
format_stamp: "Formatted at 2024-05-02 21:25:32 on dist-test-slave-jdfx"
I20240502 21:25:32.574148 17773 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.001s
I20240502 21:25:32.577169 17788 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:32.578302 17773 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240502 21:25:32.578437 17773 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "7d4fdafe6e704a8889dd12240f102bda"
format_stamp: "Formatted at 2024-05-02 21:25:32 on dist-test-slave-jdfx"
I20240502 21:25:32.578609 17773 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:25:32.592525 17773 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:25:32.593217 17773 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:25:32.593461 17773 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:25:32.594264 17773 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:25:32.595556 17773 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:25:32.595651 17773 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:32.595759 17773 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:25:32.595826 17773 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:32.623848 17773 rpc_server.cc:307] RPC server started. Bound to: 127.15.255.132:43943
I20240502 21:25:32.623908 17900 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.255.132:43943 every 8 connection(s)
I20240502 21:25:32.625083 17773 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240502 21:25:32.629688 16382 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvNciQZ/build/asan/bin/kudu as pid 17773
I20240502 21:25:32.629819 16382 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240502 21:25:32.632638 16382 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvNciQZ/build/asan/bin/kudu
/tmp/dist-test-taskvNciQZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.255.133:0
--local_ip_for_outbound_sockets=127.15.255.133
--webserver_interface=127.15.255.133
--webserver_port=0
--tserver_master_addrs=127.15.255.190:41421
--builtin_ntp_servers=127.15.255.148:35553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 21:25:32.636876 17901 heartbeater.cc:347] Connected to a master server at 127.15.255.190:41421
I20240502 21:25:32.637140 17901 heartbeater.cc:464] Registering TS with master...
I20240502 21:25:32.637884 17901 heartbeater.cc:511] Master 127.15.255.190:41421 requested a full tablet report, sending...
I20240502 21:25:32.639251 17312 ts_manager.cc:194] Registered new tserver with Master: 7d4fdafe6e704a8889dd12240f102bda (127.15.255.132:43943)
I20240502 21:25:32.640282 17312 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.255.132:49121
I20240502 21:25:32.823668 17636 heartbeater.cc:503] Master 127.15.255.190:41421 was elected leader, sending a full tablet report...
W20240502 21:25:32.945678 17905 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:25:32.945955 17905 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:25:32.946084 17905 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:25:32.954155 17905 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 21:25:32.954285 17905 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:25:32.954429 17905 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.255.133
I20240502 21:25:32.963418 17905 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.255.148:35553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.255.133:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.15.255.133
--webserver_port=0
--tserver_master_addrs=127.15.255.190:41421
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.255.133
--log_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:25:32.965466 17905 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:25:32.967300 17905 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:25:32.979547 17910 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:32.979799 17911 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:32.980213 17913 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:25:32.980764 17905 server_base.cc:1034] running on GCE node
I20240502 21:25:32.981278 17905 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:25:32.982138 17905 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:25:32.983328 17905 hybrid_clock.cc:648] HybridClock initialized: now 1714685132983285 us; error 38 us; skew 500 ppm
I20240502 21:25:32.983628 17905 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:25:32.990440 17905 webserver.cc:460] Webserver started at http://127.15.255.133:44785/ using document root <none> and password file <none>
I20240502 21:25:32.991112 17905 fs_manager.cc:359] Metadata directory not provided
I20240502 21:25:32.991232 17905 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:25:32.991544 17905 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:25:32.994091 17905 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "16104a6d794c4d91a3bb8c6bd1dcfd24"
format_stamp: "Formatted at 2024-05-02 21:25:32 on dist-test-slave-jdfx"
I20240502 21:25:32.994843 17905 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "16104a6d794c4d91a3bb8c6bd1dcfd24"
format_stamp: "Formatted at 2024-05-02 21:25:32 on dist-test-slave-jdfx"
I20240502 21:25:32.999653 17905 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20240502 21:25:33.002811 17919 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:33.003988 17905 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 21:25:33.004127 17905 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "16104a6d794c4d91a3bb8c6bd1dcfd24"
format_stamp: "Formatted at 2024-05-02 21:25:32 on dist-test-slave-jdfx"
I20240502 21:25:33.004298 17905 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:25:33.018352 17905 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:25:33.019063 17905 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:25:33.019299 17905 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:25:33.020117 17905 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:25:33.021466 17905 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:25:33.021559 17905 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:33.021661 17905 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:25:33.021741 17905 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:33.052922 17905 rpc_server.cc:307] RPC server started. Bound to: 127.15.255.133:42713
I20240502 21:25:33.052961 18035 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.255.133:42713 every 8 connection(s)
I20240502 21:25:33.054215 17905 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240502 21:25:33.058481 16382 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvNciQZ/build/asan/bin/kudu as pid 17905
I20240502 21:25:33.058657 16382 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceStartSafetyTest.TooManyIgnoredTservers_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240502 21:25:33.066255 18036 heartbeater.cc:347] Connected to a master server at 127.15.255.190:41421
I20240502 21:25:33.066509 18036 heartbeater.cc:464] Registering TS with master...
I20240502 21:25:33.067253 18036 heartbeater.cc:511] Master 127.15.255.190:41421 requested a full tablet report, sending...
I20240502 21:25:33.068507 17312 ts_manager.cc:194] Registered new tserver with Master: 16104a6d794c4d91a3bb8c6bd1dcfd24 (127.15.255.133:42713)
I20240502 21:25:33.069243 17312 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.255.133:58417
I20240502 21:25:33.073639 16382 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240502 21:25:33.089082 17312 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51742:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20240502 21:25:33.118825 17704 tablet_service.cc:1467] Processing CreateTablet for tablet a588aedc5481457eb3a2ea846fa11093 (DEFAULT_TABLE table=TestTable [id=7553299ac8264178a3a66ca5564339e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:33.120529 17704 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a588aedc5481457eb3a2ea846fa11093. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:33.126487 17836 tablet_service.cc:1467] Processing CreateTablet for tablet a588aedc5481457eb3a2ea846fa11093 (DEFAULT_TABLE table=TestTable [id=7553299ac8264178a3a66ca5564339e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:33.126719 17567 tablet_service.cc:1467] Processing CreateTablet for tablet a588aedc5481457eb3a2ea846fa11093 (DEFAULT_TABLE table=TestTable [id=7553299ac8264178a3a66ca5564339e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:33.127818 17836 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a588aedc5481457eb3a2ea846fa11093. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:33.127965 17567 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a588aedc5481457eb3a2ea846fa11093. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:33.134516 18054 tablet_bootstrap.cc:492] T a588aedc5481457eb3a2ea846fa11093 P d1278c0eeece4fe0b5027233f1c1528d: Bootstrap starting.
I20240502 21:25:33.138175 18054 tablet_bootstrap.cc:654] T a588aedc5481457eb3a2ea846fa11093 P d1278c0eeece4fe0b5027233f1c1528d: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:33.138572 18055 tablet_bootstrap.cc:492] T a588aedc5481457eb3a2ea846fa11093 P 7d4fdafe6e704a8889dd12240f102bda: Bootstrap starting.
I20240502 21:25:33.139820 18056 tablet_bootstrap.cc:492] T a588aedc5481457eb3a2ea846fa11093 P 335a85cef17445549f8bcb9a5092366f: Bootstrap starting.
I20240502 21:25:33.140009 18054 log.cc:824] T a588aedc5481457eb3a2ea846fa11093 P d1278c0eeece4fe0b5027233f1c1528d: Log is configured to *not* fsync() on all Append() calls
I20240502 21:25:33.141616 18055 tablet_bootstrap.cc:654] T a588aedc5481457eb3a2ea846fa11093 P 7d4fdafe6e704a8889dd12240f102bda: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:33.142500 18055 log.cc:824] T a588aedc5481457eb3a2ea846fa11093 P 7d4fdafe6e704a8889dd12240f102bda: Log is configured to *not* fsync() on all Append() calls
I20240502 21:25:33.142855 18054 tablet_bootstrap.cc:492] T a588aedc5481457eb3a2ea846fa11093 P d1278c0eeece4fe0b5027233f1c1528d: No bootstrap required, opened a new log
I20240502 21:25:33.143003 18056 tablet_bootstrap.cc:654] T a588aedc5481457eb3a2ea846fa11093 P 335a85cef17445549f8bcb9a5092366f: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:33.143213 18054 ts_tablet_manager.cc:1367] T a588aedc5481457eb3a2ea846fa11093 P d1278c0eeece4fe0b5027233f1c1528d: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.003s
I20240502 21:25:33.144135 18056 log.cc:824] T a588aedc5481457eb3a2ea846fa11093 P 335a85cef17445549f8bcb9a5092366f: Log is configured to *not* fsync() on all Append() calls
I20240502 21:25:33.145509 18055 tablet_bootstrap.cc:492] T a588aedc5481457eb3a2ea846fa11093 P 7d4fdafe6e704a8889dd12240f102bda: No bootstrap required, opened a new log
I20240502 21:25:33.145785 18055 ts_tablet_manager.cc:1367] T a588aedc5481457eb3a2ea846fa11093 P 7d4fdafe6e704a8889dd12240f102bda: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20240502 21:25:33.147249 18056 tablet_bootstrap.cc:492] T a588aedc5481457eb3a2ea846fa11093 P 335a85cef17445549f8bcb9a5092366f: No bootstrap required, opened a new log
I20240502 21:25:33.147500 18056 ts_tablet_manager.cc:1367] T a588aedc5481457eb3a2ea846fa11093 P 335a85cef17445549f8bcb9a5092366f: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20240502 21:25:33.151510 18054 raft_consensus.cc:348] T a588aedc5481457eb3a2ea846fa11093 P d1278c0eeece4fe0b5027233f1c1528d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1278c0eeece4fe0b5027233f1c1528d" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 33535 } } peers { permanent_uuid: "7d4fdafe6e704a8889dd12240f102bda" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43943 } } peers { permanent_uuid: "335a85cef17445549f8bcb9a5092366f" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 33539 } }
I20240502 21:25:33.151916 18054 raft_consensus.cc:374] T a588aedc5481457eb3a2ea846fa11093 P d1278c0eeece4fe0b5027233f1c1528d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:33.152081 18054 raft_consensus.cc:729] T a588aedc5481457eb3a2ea846fa11093 P d1278c0eeece4fe0b5027233f1c1528d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1278c0eeece4fe0b5027233f1c1528d, State: Initialized, Role: FOLLOWER
I20240502 21:25:33.151930 18055 raft_consensus.cc:348] T a588aedc5481457eb3a2ea846fa11093 P 7d4fdafe6e704a8889dd12240f102bda [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1278c0eeece4fe0b5027233f1c1528d" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 33535 } } peers { permanent_uuid: "7d4fdafe6e704a8889dd12240f102bda" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43943 } } peers { permanent_uuid: "335a85cef17445549f8bcb9a5092366f" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 33539 } }
I20240502 21:25:33.152281 18055 raft_consensus.cc:374] T a588aedc5481457eb3a2ea846fa11093 P 7d4fdafe6e704a8889dd12240f102bda [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:33.152465 18055 raft_consensus.cc:729] T a588aedc5481457eb3a2ea846fa11093 P 7d4fdafe6e704a8889dd12240f102bda [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d4fdafe6e704a8889dd12240f102bda, State: Initialized, Role: FOLLOWER
I20240502 21:25:33.152762 18054 consensus_queue.cc:260] T a588aedc5481457eb3a2ea846fa11093 P d1278c0eeece4fe0b5027233f1c1528d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1278c0eeece4fe0b5027233f1c1528d" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 33535 } } peers { permanent_uuid: "7d4fdafe6e704a8889dd12240f102bda" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43943 } } peers { permanent_uuid: "335a85cef17445549f8bcb9a5092366f" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 33539 } }
I20240502 21:25:33.153072 18055 consensus_queue.cc:260] T a588aedc5481457eb3a2ea846fa11093 P 7d4fdafe6e704a8889dd12240f102bda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1278c0eeece4fe0b5027233f1c1528d" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 33535 } } peers { permanent_uuid: "7d4fdafe6e704a8889dd12240f102bda" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43943 } } peers { permanent_uuid: "335a85cef17445549f8bcb9a5092366f" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 33539 } }
I20240502 21:25:33.155895 18056 raft_consensus.cc:348] T a588aedc5481457eb3a2ea846fa11093 P 335a85cef17445549f8bcb9a5092366f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1278c0eeece4fe0b5027233f1c1528d" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 33535 } } peers { permanent_uuid: "7d4fdafe6e704a8889dd12240f102bda" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43943 } } peers { permanent_uuid: "335a85cef17445549f8bcb9a5092366f" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 33539 } }
I20240502 21:25:33.156299 18056 raft_consensus.cc:374] T a588aedc5481457eb3a2ea846fa11093 P 335a85cef17445549f8bcb9a5092366f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:33.156395 18056 raft_consensus.cc:729] T a588aedc5481457eb3a2ea846fa11093 P 335a85cef17445549f8bcb9a5092366f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 335a85cef17445549f8bcb9a5092366f, State: Initialized, Role: FOLLOWER
I20240502 21:25:33.157120 17770 heartbeater.cc:503] Master 127.15.255.190:41421 was elected leader, sending a full tablet report...
I20240502 21:25:33.157181 18056 consensus_queue.cc:260] T a588aedc5481457eb3a2ea846fa11093 P 335a85cef17445549f8bcb9a5092366f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1278c0eeece4fe0b5027233f1c1528d" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 33535 } } peers { permanent_uuid: "7d4fdafe6e704a8889dd12240f102bda" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43943 } } peers { permanent_uuid: "335a85cef17445549f8bcb9a5092366f" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 33539 } }
I20240502 21:25:33.158800 18056 ts_tablet_manager.cc:1398] T a588aedc5481457eb3a2ea846fa11093 P 335a85cef17445549f8bcb9a5092366f: Time spent starting tablet: real 0.011s user 0.012s sys 0.000s
I20240502 21:25:33.160168 18054 ts_tablet_manager.cc:1398] T a588aedc5481457eb3a2ea846fa11093 P d1278c0eeece4fe0b5027233f1c1528d: Time spent starting tablet: real 0.017s user 0.009s sys 0.005s
I20240502 21:25:33.161077 18055 ts_tablet_manager.cc:1398] T a588aedc5481457eb3a2ea846fa11093 P 7d4fdafe6e704a8889dd12240f102bda: Time spent starting tablet: real 0.015s user 0.010s sys 0.005s
I20240502 21:25:33.161513 17901 heartbeater.cc:503] Master 127.15.255.190:41421 was elected leader, sending a full tablet report...
I20240502 21:25:33.165208 18062 raft_consensus.cc:482] T a588aedc5481457eb3a2ea846fa11093 P 335a85cef17445549f8bcb9a5092366f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:25:33.165403 18062 raft_consensus.cc:504] T a588aedc5481457eb3a2ea846fa11093 P 335a85cef17445549f8bcb9a5092366f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1278c0eeece4fe0b5027233f1c1528d" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 33535 } } peers { permanent_uuid: "7d4fdafe6e704a8889dd12240f102bda" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43943 } } peers { permanent_uuid: "335a85cef17445549f8bcb9a5092366f" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 33539 } }
I20240502 21:25:33.166764 18062 leader_election.cc:288] T a588aedc5481457eb3a2ea846fa11093 P 335a85cef17445549f8bcb9a5092366f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1278c0eeece4fe0b5027233f1c1528d (127.15.255.131:33535), 7d4fdafe6e704a8889dd12240f102bda (127.15.255.132:43943)
I20240502 21:25:33.174845 17856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a588aedc5481457eb3a2ea846fa11093" candidate_uuid: "335a85cef17445549f8bcb9a5092366f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d4fdafe6e704a8889dd12240f102bda" is_pre_election: true
I20240502 21:25:33.174911 17724 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a588aedc5481457eb3a2ea846fa11093" candidate_uuid: "335a85cef17445549f8bcb9a5092366f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1278c0eeece4fe0b5027233f1c1528d" is_pre_election: true
I20240502 21:25:33.175294 17856 raft_consensus.cc:2454] T a588aedc5481457eb3a2ea846fa11093 P 7d4fdafe6e704a8889dd12240f102bda [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 335a85cef17445549f8bcb9a5092366f in term 0.
I20240502 21:25:33.175323 17724 raft_consensus.cc:2454] T a588aedc5481457eb3a2ea846fa11093 P d1278c0eeece4fe0b5027233f1c1528d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 335a85cef17445549f8bcb9a5092366f in term 0.
I20240502 21:25:33.175875 17522 leader_election.cc:302] T a588aedc5481457eb3a2ea846fa11093 P 335a85cef17445549f8bcb9a5092366f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 335a85cef17445549f8bcb9a5092366f, d1278c0eeece4fe0b5027233f1c1528d; no voters:
I20240502 21:25:33.176193 18062 raft_consensus.cc:2781] T a588aedc5481457eb3a2ea846fa11093 P 335a85cef17445549f8bcb9a5092366f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:25:33.176353 18062 raft_consensus.cc:482] T a588aedc5481457eb3a2ea846fa11093 P 335a85cef17445549f8bcb9a5092366f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:25:33.176484 18062 raft_consensus.cc:3037] T a588aedc5481457eb3a2ea846fa11093 P 335a85cef17445549f8bcb9a5092366f [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:33.181407 18062 raft_consensus.cc:504] T a588aedc5481457eb3a2ea846fa11093 P 335a85cef17445549f8bcb9a5092366f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1278c0eeece4fe0b5027233f1c1528d" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 33535 } } peers { permanent_uuid: "7d4fdafe6e704a8889dd12240f102bda" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43943 } } peers { permanent_uuid: "335a85cef17445549f8bcb9a5092366f" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 33539 } }
I20240502 21:25:33.181973 18062 leader_election.cc:288] T a588aedc5481457eb3a2ea846fa11093 P 335a85cef17445549f8bcb9a5092366f [CANDIDATE]: Term 1 election: Requested vote from peers d1278c0eeece4fe0b5027233f1c1528d (127.15.255.131:33535), 7d4fdafe6e704a8889dd12240f102bda (127.15.255.132:43943)
I20240502 21:25:33.182293 17724 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a588aedc5481457eb3a2ea846fa11093" candidate_uuid: "335a85cef17445549f8bcb9a5092366f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1278c0eeece4fe0b5027233f1c1528d"
I20240502 21:25:33.182471 17724 raft_consensus.cc:3037] T a588aedc5481457eb3a2ea846fa11093 P d1278c0eeece4fe0b5027233f1c1528d [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:33.182436 17856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a588aedc5481457eb3a2ea846fa11093" candidate_uuid: "335a85cef17445549f8bcb9a5092366f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d4fdafe6e704a8889dd12240f102bda"
I20240502 21:25:33.182616 17856 raft_consensus.cc:3037] T a588aedc5481457eb3a2ea846fa11093 P 7d4fdafe6e704a8889dd12240f102bda [term 0 FOLLOWER]: Advancing to term 1
W20240502 21:25:33.214305 17771 tablet.cc:2369] T a588aedc5481457eb3a2ea846fa11093 P d1278c0eeece4fe0b5027233f1c1528d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 21:25:33.278587 17724 raft_consensus.cc:2454] T a588aedc5481457eb3a2ea846fa11093 P d1278c0eeece4fe0b5027233f1c1528d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 335a85cef17445549f8bcb9a5092366f in term 1.
I20240502 21:25:33.278620 17856 raft_consensus.cc:2454] T a588aedc5481457eb3a2ea846fa11093 P 7d4fdafe6e704a8889dd12240f102bda [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 335a85cef17445549f8bcb9a5092366f in term 1.
I20240502 21:25:33.279083 17522 leader_election.cc:302] T a588aedc5481457eb3a2ea846fa11093 P 335a85cef17445549f8bcb9a5092366f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 335a85cef17445549f8bcb9a5092366f, d1278c0eeece4fe0b5027233f1c1528d; no voters:
I20240502 21:25:33.279421 18062 raft_consensus.cc:2781] T a588aedc5481457eb3a2ea846fa11093 P 335a85cef17445549f8bcb9a5092366f [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:25:33.279894 18062 raft_consensus.cc:686] T a588aedc5481457eb3a2ea846fa11093 P 335a85cef17445549f8bcb9a5092366f [term 1 LEADER]: Becoming Leader. State: Replica: 335a85cef17445549f8bcb9a5092366f, State: Running, Role: LEADER
I20240502 21:25:33.280459 18062 consensus_queue.cc:237] T a588aedc5481457eb3a2ea846fa11093 P 335a85cef17445549f8bcb9a5092366f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1278c0eeece4fe0b5027233f1c1528d" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 33535 } } peers { permanent_uuid: "7d4fdafe6e704a8889dd12240f102bda" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43943 } } peers { permanent_uuid: "335a85cef17445549f8bcb9a5092366f" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 33539 } }
I20240502 21:25:33.285526 17312 catalog_manager.cc:5484] T a588aedc5481457eb3a2ea846fa11093 P 335a85cef17445549f8bcb9a5092366f reported cstate change: term changed from 0 to 1, leader changed from <none> to 335a85cef17445549f8bcb9a5092366f (127.15.255.130). New cstate: current_term: 1 leader_uuid: "335a85cef17445549f8bcb9a5092366f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1278c0eeece4fe0b5027233f1c1528d" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 33535 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7d4fdafe6e704a8889dd12240f102bda" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43943 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "335a85cef17445549f8bcb9a5092366f" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 33539 } health_report { overall_health: HEALTHY } } }
I20240502 21:25:33.308658 16382 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
W20240502 21:25:33.308961 17637 tablet.cc:2369] T a588aedc5481457eb3a2ea846fa11093 P 335a85cef17445549f8bcb9a5092366f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 21:25:33.311393 16382 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 4b1b6f344c434a59b099fea2d10fada6 to finish bootstrapping
I20240502 21:25:33.320170 16382 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 335a85cef17445549f8bcb9a5092366f to finish bootstrapping
I20240502 21:25:33.327184 16382 ts_itest-base.cc:233] Waiting for 1 tablets on tserver d1278c0eeece4fe0b5027233f1c1528d to finish bootstrapping
I20240502 21:25:33.333503 16382 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 7d4fdafe6e704a8889dd12240f102bda to finish bootstrapping
I20240502 21:25:33.340391 16382 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 16104a6d794c4d91a3bb8c6bd1dcfd24 to finish bootstrapping
W20240502 21:25:33.376929 17902 tablet.cc:2369] T a588aedc5481457eb3a2ea846fa11093 P 7d4fdafe6e704a8889dd12240f102bda: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 21:25:33.660692 18062 consensus_queue.cc:1035] T a588aedc5481457eb3a2ea846fa11093 P 335a85cef17445549f8bcb9a5092366f [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d4fdafe6e704a8889dd12240f102bda" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:25:33.668735 18065 consensus_queue.cc:1035] T a588aedc5481457eb3a2ea846fa11093 P 335a85cef17445549f8bcb9a5092366f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1278c0eeece4fe0b5027233f1c1528d" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 33535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:25:33.787400 17435 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:25:33.788798 17967 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:25:33.791764 17567 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:25:33.798929 17704 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:25:33.799132 17836 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:25:33.956635 17967 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:25:33.963585 17567 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:25:33.966940 17435 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:25:33.972271 17704 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:25:33.981734 17836 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:25:34.071426 18036 heartbeater.cc:503] Master 127.15.255.190:41421 was elected leader, sending a full tablet report...
I20240502 21:25:34.131098 17967 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:25:34.134179 17567 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:25:34.143512 17435 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:25:34.145226 17836 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:25:34.145619 17704 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:25:34.734066 17967 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:25:34.754631 17435 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:25:34.755746 17567 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:25:34.765007 17704 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:25:34.769040 17836 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:25:34.905464 17967 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:25:34.918301 17567 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:25:34.922590 17435 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:25:34.925164 17836 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:25:34.941947 17704 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:25:35.089253 17967 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:25:35.096869 17435 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:25:35.097469 17567 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:25:35.103691 17704 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:25:35.113054 17836 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:25:35.257755 16382 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvNciQZ/build/asan/bin/kudu pid 17371
I20240502 21:25:35.259327 17495 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 21:25:35.348585 16382 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvNciQZ/build/asan/bin/kudu with pid 17371
I20240502 21:25:35.364440 16382 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvNciQZ/build/asan/bin/kudu pid 17503
I20240502 21:25:35.365408 17631 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 21:25:35.458786 16382 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvNciQZ/build/asan/bin/kudu with pid 17503
I20240502 21:25:35.474959 16382 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvNciQZ/build/asan/bin/kudu pid 17639
I20240502 21:25:35.475920 17764 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 21:25:35.575065 16382 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvNciQZ/build/asan/bin/kudu with pid 17639
I20240502 21:25:35.591259 16382 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvNciQZ/build/asan/bin/kudu pid 17773
I20240502 21:25:35.592249 17896 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 21:25:35.689677 16382 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvNciQZ/build/asan/bin/kudu with pid 17773
I20240502 21:25:35.704628 16382 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvNciQZ/build/asan/bin/kudu pid 17905
W20240502 21:25:35.704588 17296 connection.cc:535] server connection from 127.15.255.132:49121 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 21:25:35.705591 18030 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 21:25:35.797034 16382 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvNciQZ/build/asan/bin/kudu with pid 17905
I20240502 21:25:35.812135 16382 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvNciQZ/build/asan/bin/kudu pid 17279
I20240502 21:25:35.813344 17342 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 21:25:35.926776 16382 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvNciQZ/build/asan/bin/kudu with pid 17279
2024-05-02T21:25:35Z chronyd exiting
[ OK ] RebalanceStartSafetyTest.TooManyIgnoredTservers/1 (5464 ms)
[----------] 1 test from RebalanceStartSafetyTest (5464 ms total)
[----------] 1 test from RebalanceParamTest
[ RUN ] RebalanceParamTest.Rebalance/7
I20240502 21:25:35.965798 16382 test_util.cc:268] Using random seed: 1047416381
I20240502 21:25:35.967521 16382 ts_itest-base.cc:112] Starting cluster with:
I20240502 21:25:35.967610 16382 ts_itest-base.cc:113] --------------
I20240502 21:25:35.967661 16382 ts_itest-base.cc:114] 7 tablet servers
I20240502 21:25:35.967731 16382 ts_itest-base.cc:115] 5 replicas per TS
I20240502 21:25:35.967778 16382 ts_itest-base.cc:116] --------------
2024-05-02T21:25:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-02T21:25:35Z Disabled control of system clock
I20240502 21:25:35.988900 16382 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvNciQZ/build/asan/bin/kudu
/tmp/dist-test-taskvNciQZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714685120732928-16382-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714685120732928-16382-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714685120732928-16382-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714685120732928-16382-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.255.190:36947
--webserver_interface=127.15.255.190
--webserver_port=0
--builtin_ntp_servers=127.15.255.148:40185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.255.190:36947
--allow_unsafe_replication_factor
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=true
--tserver_unresponsive_timeout_ms=3000 with env {}
W20240502 21:25:36.294931 18297 flags.cc:425] Enabled unsafe flag: --allow_unsafe_replication_factor=true
W20240502 21:25:36.295235 18297 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:25:36.295344 18297 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:25:36.295444 18297 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:25:36.302453 18297 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 21:25:36.302557 18297 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240502 21:25:36.302649 18297 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:25:36.302709 18297 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240502 21:25:36.302795 18297 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240502 21:25:36.311305 18297 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.255.148:40185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714685120732928-16382-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714685120732928-16382-0/raft_consensus-itest-cluster/master-0/wal
--allow_unsafe_replication_factor=true
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.15.255.190:36947
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.255.190:36947
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714685120732928-16382-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.15.255.190
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714685120732928-16382-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:25:36.312470 18297 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:25:36.314110 18297 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:25:36.324754 18305 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:36.325249 18303 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:36.326085 18302 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:25:36.326546 18297 server_base.cc:1034] running on GCE node
I20240502 21:25:36.327196 18297 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:25:36.328402 18297 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:25:36.329736 18297 hybrid_clock.cc:648] HybridClock initialized: now 1714685136329676 us; error 38 us; skew 500 ppm
I20240502 21:25:36.330029 18297 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:25:36.332084 18297 webserver.cc:460] Webserver started at http://127.15.255.190:40335/ using document root <none> and password file <none>
I20240502 21:25:36.332782 18297 fs_manager.cc:359] Metadata directory not provided
I20240502 21:25:36.332909 18297 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:25:36.333184 18297 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:25:36.335556 18297 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714685120732928-16382-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "132c427694cc4f858dd86045029e179e"
format_stamp: "Formatted at 2024-05-02 21:25:36 on dist-test-slave-jdfx"
I20240502 21:25:36.336244 18297 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714685120732928-16382-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "132c427694cc4f858dd86045029e179e"
format_stamp: "Formatted at 2024-05-02 21:25:36 on dist-test-slave-jdfx"
I20240502 21:25:36.340853 18297 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20240502 21:25:36.343853 18311 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:36.345057 18297 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240502 21:25:36.345203 18297 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714685120732928-16382-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714685120732928-16382-0/raft_consensus-itest-cluster/master-0/wal
uuid: "132c427694cc4f858dd86045029e179e"
format_stamp: "Formatted at 2024-05-02 21:25:36 on dist-test-slave-jdfx"
I20240502 21:25:36.345358 18297 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714685120732928-16382-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714685120732928-16382-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714685120732928-16382-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:25:36.365613 18297 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:25:36.366278 18297 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:25:36.366551 18297 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:25:36.383157 18297 rpc_server.cc:307] RPC server started. Bound to: 127.15.255.190:36947
I20240502 21:25:36.383195 18364 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.255.190:36947 every 8 connection(s)
I20240502 21:25:36.384323 18297 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714685120732928-16382-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240502 21:25:36.387745 18365 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:36.390491 16382 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvNciQZ/build/asan/bin/kudu as pid 18297
I20240502 21:25:36.390728 16382 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714685120732928-16382-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240502 21:25:36.399008 18365 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 132c427694cc4f858dd86045029e179e: Bootstrap starting.
I20240502 21:25:36.402545 18365 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 132c427694cc4f858dd86045029e179e: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:36.403535 18365 log.cc:824] T 00000000000000000000000000000000 P 132c427694cc4f858dd86045029e179e: Log is configured to *not* fsync() on all Append() calls
I20240502 21:25:36.405850 18365 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 132c427694cc4f858dd86045029e179e: No bootstrap required, opened a new log
I20240502 21:25:36.411605 18365 raft_consensus.cc:348] T 00000000000000000000000000000000 P 132c427694cc4f858dd86045029e179e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "132c427694cc4f858dd86045029e179e" member_type: VOTER last_known_addr { host: "127.15.255.190" port: 36947 } }
I20240502 21:25:36.411921 18365 raft_consensus.cc:374] T 00000000000000000000000000000000 P 132c427694cc4f858dd86045029e179e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:36.412014 18365 raft_consensus.cc:729] T 00000000000000000000000000000000 P 132c427694cc4f858dd86045029e179e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 132c427694cc4f858dd86045029e179e, State: Initialized, Role: FOLLOWER
I20240502 21:25:36.412580 18365 consensus_queue.cc:260] T 00000000000000000000000000000000 P 132c427694cc4f858dd86045029e179e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "132c427694cc4f858dd86045029e179e" member_type: VOTER last_known_addr { host: "127.15.255.190" port: 36947 } }
I20240502 21:25:36.412791 18365 raft_consensus.cc:388] T 00000000000000000000000000000000 P 132c427694cc4f858dd86045029e179e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 21:25:36.412914 18365 raft_consensus.cc:482] T 00000000000000000000000000000000 P 132c427694cc4f858dd86045029e179e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 21:25:36.413048 18365 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 132c427694cc4f858dd86045029e179e [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:36.510943 18365 raft_consensus.cc:504] T 00000000000000000000000000000000 P 132c427694cc4f858dd86045029e179e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "132c427694cc4f858dd86045029e179e" member_type: VOTER last_known_addr { host: "127.15.255.190" port: 36947 } }
I20240502 21:25:36.511574 18365 leader_election.cc:302] T 00000000000000000000000000000000 P 132c427694cc4f858dd86045029e179e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 132c427694cc4f858dd86045029e179e; no voters:
I20240502 21:25:36.512223 18365 leader_election.cc:288] T 00000000000000000000000000000000 P 132c427694cc4f858dd86045029e179e [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 21:25:36.512383 18370 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 132c427694cc4f858dd86045029e179e [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:25:36.513417 18370 raft_consensus.cc:686] T 00000000000000000000000000000000 P 132c427694cc4f858dd86045029e179e [term 1 LEADER]: Becoming Leader. State: Replica: 132c427694cc4f858dd86045029e179e, State: Running, Role: LEADER
I20240502 21:25:36.514099 18370 consensus_queue.cc:237] T 00000000000000000000000000000000 P 132c427694cc4f858dd86045029e179e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "132c427694cc4f858dd86045029e179e" member_type: VOTER last_known_addr { host: "127.15.255.190" port: 36947 } }
I20240502 21:25:36.514426 18365 sys_catalog.cc:564] T 00000000000000000000000000000000 P 132c427694cc4f858dd86045029e179e [sys.catalog]: configured and running, proceeding with master startup.
I20240502 21:25:36.517022 18372 sys_catalog.cc:455] T 00000000000000000000000000000000 P 132c427694cc4f858dd86045029e179e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 132c427694cc4f858dd86045029e179e. Latest consensus state: current_term: 1 leader_uuid: "132c427694cc4f858dd86045029e179e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "132c427694cc4f858dd86045029e179e" member_type: VOTER last_known_addr { host: "127.15.255.190" port: 36947 } } }
I20240502 21:25:36.517318 18372 sys_catalog.cc:458] T 00000000000000000000000000000000 P 132c427694cc4f858dd86045029e179e [sys.catalog]: This master's current role is: LEADER
I20240502 21:25:36.517305 18371 sys_catalog.cc:455] T 00000000000000000000000000000000 P 132c427694cc4f858dd86045029e179e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "132c427694cc4f858dd86045029e179e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "132c427694cc4f858dd86045029e179e" member_type: VOTER last_known_addr { host: "127.15.255.190" port: 36947 } } }
I20240502 21:25:36.517603 18371 sys_catalog.cc:458] T 00000000000000000000000000000000 P 132c427694cc4f858dd86045029e179e [sys.catalog]: This master's current role is: LEADER
I20240502 21:25:36.520674 18376 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 21:25:36.525036 18376 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 21:25:36.532956 18376 catalog_manager.cc:1347] Generated new cluster ID: 5611db368787455a817cebd322dc4a71
I20240502 21:25:36.533046 18376 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 21:25:36.545475 18376 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 21:25:36.546485 18376 catalog_manager.cc:1504] Loading token signing keys...
I20240502 21:25:36.554901 18376 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 132c427694cc4f858dd86045029e179e: Generated new TSK 0
I20240502 21:25:36.555588 18376 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 21:25:36.571552 16382 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvNciQZ/build/asan/bin/kudu
/tmp/dist-test-taskvNciQZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.255.129:0
--local_ip_for_outbound_sockets=127.15.255.129
--webserver_interface=127.15.255.129
--webserver_port=0
--tserver_master_addrs=127.15.255.190:36947
--builtin_ntp_servers=127.15.255.148:40185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240502 21:25:36.911710 18389 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:25:36.911958 18389 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:25:36.912058 18389 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:25:36.919274 18389 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 21:25:36.919376 18389 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:25:36.919486 18389 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.255.129
I20240502 21:25:36.928486 18389 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.255.148:40185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.255.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.15.255.129
--webserver_port=0
--tserver_master_addrs=127.15.255.190:36947
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.255.129
--log_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:25:36.929728 18389 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:25:36.931430 18389 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:25:36.943578 18394 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:36.944193 18395 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:36.944787 18397 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:25:36.945333 18389 server_base.cc:1034] running on GCE node
I20240502 21:25:36.945874 18389 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:25:36.946722 18389 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:25:36.947912 18389 hybrid_clock.cc:648] HybridClock initialized: now 1714685136947866 us; error 36 us; skew 500 ppm
I20240502 21:25:36.948228 18389 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:25:36.950414 18389 webserver.cc:460] Webserver started at http://127.15.255.129:40873/ using document root <none> and password file <none>
I20240502 21:25:36.951067 18389 fs_manager.cc:359] Metadata directory not provided
I20240502 21:25:36.951185 18389 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:25:36.951460 18389 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:25:36.953886 18389 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "faadabfb099b4abf90a466395be585b5"
format_stamp: "Formatted at 2024-05-02 21:25:36 on dist-test-slave-jdfx"
I20240502 21:25:36.954586 18389 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "faadabfb099b4abf90a466395be585b5"
format_stamp: "Formatted at 2024-05-02 21:25:36 on dist-test-slave-jdfx"
I20240502 21:25:36.959131 18389 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20240502 21:25:36.962204 18403 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:36.963368 18389 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 21:25:36.963510 18389 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "faadabfb099b4abf90a466395be585b5"
format_stamp: "Formatted at 2024-05-02 21:25:36 on dist-test-slave-jdfx"
I20240502 21:25:36.963685 18389 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:25:36.993579 18389 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:25:36.994338 18389 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:25:36.994602 18389 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:25:36.995473 18389 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:25:36.996905 18389 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:25:36.997007 18389 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:36.997112 18389 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:25:36.997179 18389 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:37.028121 18389 rpc_server.cc:307] RPC server started. Bound to: 127.15.255.129:32983
I20240502 21:25:37.028179 18523 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.255.129:32983 every 8 connection(s)
I20240502 21:25:37.029417 18389 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240502 21:25:37.030709 16382 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvNciQZ/build/asan/bin/kudu as pid 18389
I20240502 21:25:37.030881 16382 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240502 21:25:37.034412 16382 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvNciQZ/build/asan/bin/kudu
/tmp/dist-test-taskvNciQZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.255.130:0
--local_ip_for_outbound_sockets=127.15.255.130
--webserver_interface=127.15.255.130
--webserver_port=0
--tserver_master_addrs=127.15.255.190:36947
--builtin_ntp_servers=127.15.255.148:40185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 21:25:37.041937 18524 heartbeater.cc:347] Connected to a master server at 127.15.255.190:36947
I20240502 21:25:37.042196 18524 heartbeater.cc:464] Registering TS with master...
I20240502 21:25:37.042959 18524 heartbeater.cc:511] Master 127.15.255.190:36947 requested a full tablet report, sending...
I20240502 21:25:37.044867 18329 ts_manager.cc:194] Registered new tserver with Master: faadabfb099b4abf90a466395be585b5 (127.15.255.129:32983)
I20240502 21:25:37.046375 18329 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.255.129:53125
W20240502 21:25:37.340752 18528 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:25:37.341014 18528 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:25:37.341135 18528 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:25:37.348287 18528 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 21:25:37.348404 18528 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:25:37.348585 18528 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.255.130
I20240502 21:25:37.357455 18528 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.255.148:40185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.255.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.15.255.130
--webserver_port=0
--tserver_master_addrs=127.15.255.190:36947
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.255.130
--log_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:25:37.358655 18528 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:25:37.360320 18528 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:25:37.371670 18533 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:37.372367 18536 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:37.372741 18534 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:25:37.373237 18528 server_base.cc:1034] running on GCE node
I20240502 21:25:37.373721 18528 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:25:37.374471 18528 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:25:37.375660 18528 hybrid_clock.cc:648] HybridClock initialized: now 1714685137375602 us; error 36 us; skew 500 ppm
I20240502 21:25:37.375936 18528 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:25:37.377952 18528 webserver.cc:460] Webserver started at http://127.15.255.130:33137/ using document root <none> and password file <none>
I20240502 21:25:37.378592 18528 fs_manager.cc:359] Metadata directory not provided
I20240502 21:25:37.378698 18528 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:25:37.378973 18528 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:25:37.381372 18528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "82f50f4b66f3450aa9f58a0b317f5cea"
format_stamp: "Formatted at 2024-05-02 21:25:37 on dist-test-slave-jdfx"
I20240502 21:25:37.382053 18528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "82f50f4b66f3450aa9f58a0b317f5cea"
format_stamp: "Formatted at 2024-05-02 21:25:37 on dist-test-slave-jdfx"
I20240502 21:25:37.386843 18528 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20240502 21:25:37.389930 18542 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:37.391191 18528 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20240502 21:25:37.391326 18528 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "82f50f4b66f3450aa9f58a0b317f5cea"
format_stamp: "Formatted at 2024-05-02 21:25:37 on dist-test-slave-jdfx"
I20240502 21:25:37.391490 18528 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:25:37.415846 18528 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:25:37.416535 18528 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:25:37.416790 18528 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:25:37.417601 18528 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:25:37.418952 18528 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:25:37.419046 18528 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:37.419126 18528 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:25:37.419181 18528 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:37.448437 18528 rpc_server.cc:307] RPC server started. Bound to: 127.15.255.130:45139
I20240502 21:25:37.448508 18657 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.255.130:45139 every 8 connection(s)
I20240502 21:25:37.449599 18528 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240502 21:25:37.450601 16382 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvNciQZ/build/asan/bin/kudu as pid 18528
I20240502 21:25:37.450778 16382 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240502 21:25:37.454706 16382 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvNciQZ/build/asan/bin/kudu
/tmp/dist-test-taskvNciQZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.255.131:0
--local_ip_for_outbound_sockets=127.15.255.131
--webserver_interface=127.15.255.131
--webserver_port=0
--tserver_master_addrs=127.15.255.190:36947
--builtin_ntp_servers=127.15.255.148:40185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 21:25:37.462138 18658 heartbeater.cc:347] Connected to a master server at 127.15.255.190:36947
I20240502 21:25:37.462437 18658 heartbeater.cc:464] Registering TS with master...
I20240502 21:25:37.463299 18658 heartbeater.cc:511] Master 127.15.255.190:36947 requested a full tablet report, sending...
I20240502 21:25:37.464708 18329 ts_manager.cc:194] Registered new tserver with Master: 82f50f4b66f3450aa9f58a0b317f5cea (127.15.255.130:45139)
I20240502 21:25:37.465438 18329 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.255.130:53075
W20240502 21:25:37.764089 18661 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:25:37.764317 18661 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:25:37.764454 18661 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:25:37.771692 18661 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 21:25:37.771797 18661 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:25:37.771942 18661 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.255.131
I20240502 21:25:37.780402 18661 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.255.148:40185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.255.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.15.255.131
--webserver_port=0
--tserver_master_addrs=127.15.255.190:36947
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.255.131
--log_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:25:37.781538 18661 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:25:37.783136 18661 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:25:37.794196 18667 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:37.794670 18668 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:37.795075 18670 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:25:37.795547 18661 server_base.cc:1034] running on GCE node
I20240502 21:25:37.796103 18661 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:25:37.796959 18661 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:25:37.798142 18661 hybrid_clock.cc:648] HybridClock initialized: now 1714685137798110 us; error 23 us; skew 500 ppm
I20240502 21:25:37.798418 18661 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:25:37.800449 18661 webserver.cc:460] Webserver started at http://127.15.255.131:42773/ using document root <none> and password file <none>
I20240502 21:25:37.801100 18661 fs_manager.cc:359] Metadata directory not provided
I20240502 21:25:37.801199 18661 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:25:37.801496 18661 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:25:37.805835 18661 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "501e3c0d44fe4d70af9ff926d9622417"
format_stamp: "Formatted at 2024-05-02 21:25:37 on dist-test-slave-jdfx"
I20240502 21:25:37.806535 18661 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "501e3c0d44fe4d70af9ff926d9622417"
format_stamp: "Formatted at 2024-05-02 21:25:37 on dist-test-slave-jdfx"
I20240502 21:25:37.811125 18661 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20240502 21:25:37.814159 18676 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:37.815359 18661 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240502 21:25:37.815510 18661 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "501e3c0d44fe4d70af9ff926d9622417"
format_stamp: "Formatted at 2024-05-02 21:25:37 on dist-test-slave-jdfx"
I20240502 21:25:37.815678 18661 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:25:37.835780 18661 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:25:37.836565 18661 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:25:37.836817 18661 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:25:37.837682 18661 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:25:37.839027 18661 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:25:37.839112 18661 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:37.839210 18661 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:25:37.839273 18661 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:37.867704 18661 rpc_server.cc:307] RPC server started. Bound to: 127.15.255.131:42501
I20240502 21:25:37.867765 18790 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.255.131:42501 every 8 connection(s)
I20240502 21:25:37.869009 18661 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240502 21:25:37.870657 16382 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvNciQZ/build/asan/bin/kudu as pid 18661
I20240502 21:25:37.870829 16382 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240502 21:25:37.873869 16382 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvNciQZ/build/asan/bin/kudu
/tmp/dist-test-taskvNciQZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.255.132:0
--local_ip_for_outbound_sockets=127.15.255.132
--webserver_interface=127.15.255.132
--webserver_port=0
--tserver_master_addrs=127.15.255.190:36947
--builtin_ntp_servers=127.15.255.148:40185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 21:25:37.881479 18791 heartbeater.cc:347] Connected to a master server at 127.15.255.190:36947
I20240502 21:25:37.881739 18791 heartbeater.cc:464] Registering TS with master...
I20240502 21:25:37.882462 18791 heartbeater.cc:511] Master 127.15.255.190:36947 requested a full tablet report, sending...
I20240502 21:25:37.883800 18329 ts_manager.cc:194] Registered new tserver with Master: 501e3c0d44fe4d70af9ff926d9622417 (127.15.255.131:42501)
I20240502 21:25:37.884830 18329 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.255.131:34635
I20240502 21:25:38.049129 18524 heartbeater.cc:503] Master 127.15.255.190:36947 was elected leader, sending a full tablet report...
W20240502 21:25:38.177649 18794 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:25:38.177906 18794 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:25:38.178001 18794 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:25:38.185148 18794 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 21:25:38.185256 18794 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:25:38.185369 18794 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.255.132
I20240502 21:25:38.193893 18794 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.255.148:40185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.255.132:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.15.255.132
--webserver_port=0
--tserver_master_addrs=127.15.255.190:36947
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.255.132
--log_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:25:38.195048 18794 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:25:38.196756 18794 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:25:38.208035 18799 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:38.208573 18800 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:38.208992 18802 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:25:38.209818 18794 server_base.cc:1034] running on GCE node
I20240502 21:25:38.210316 18794 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:25:38.211164 18794 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:25:38.212347 18794 hybrid_clock.cc:648] HybridClock initialized: now 1714685138212306 us; error 36 us; skew 500 ppm
I20240502 21:25:38.212667 18794 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:25:38.223865 18794 webserver.cc:460] Webserver started at http://127.15.255.132:34689/ using document root <none> and password file <none>
I20240502 21:25:38.224596 18794 fs_manager.cc:359] Metadata directory not provided
I20240502 21:25:38.224701 18794 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:25:38.225000 18794 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:25:38.227434 18794 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "c75831edb3b44dd59c00ef151adca49d"
format_stamp: "Formatted at 2024-05-02 21:25:38 on dist-test-slave-jdfx"
I20240502 21:25:38.228124 18794 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "c75831edb3b44dd59c00ef151adca49d"
format_stamp: "Formatted at 2024-05-02 21:25:38 on dist-test-slave-jdfx"
I20240502 21:25:38.232767 18794 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.008s sys 0.000s
I20240502 21:25:38.235898 18808 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:38.237103 18794 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240502 21:25:38.237248 18794 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "c75831edb3b44dd59c00ef151adca49d"
format_stamp: "Formatted at 2024-05-02 21:25:38 on dist-test-slave-jdfx"
I20240502 21:25:38.237421 18794 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:25:38.277822 18794 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:25:38.278576 18794 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:25:38.278829 18794 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:25:38.279685 18794 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:25:38.281082 18794 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:25:38.281175 18794 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:38.281280 18794 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:25:38.281345 18794 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:38.309422 18794 rpc_server.cc:307] RPC server started. Bound to: 127.15.255.132:43607
I20240502 21:25:38.309479 18923 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.255.132:43607 every 8 connection(s)
I20240502 21:25:38.310613 18794 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240502 21:25:38.320533 16382 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvNciQZ/build/asan/bin/kudu as pid 18794
I20240502 21:25:38.320755 16382 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240502 21:25:38.322446 18924 heartbeater.cc:347] Connected to a master server at 127.15.255.190:36947
I20240502 21:25:38.322755 18924 heartbeater.cc:464] Registering TS with master...
I20240502 21:25:38.323664 18924 heartbeater.cc:511] Master 127.15.255.190:36947 requested a full tablet report, sending...
I20240502 21:25:38.323745 16382 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvNciQZ/build/asan/bin/kudu
/tmp/dist-test-taskvNciQZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.255.133:0
--local_ip_for_outbound_sockets=127.15.255.133
--webserver_interface=127.15.255.133
--webserver_port=0
--tserver_master_addrs=127.15.255.190:36947
--builtin_ntp_servers=127.15.255.148:40185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 21:25:38.324986 18329 ts_manager.cc:194] Registered new tserver with Master: c75831edb3b44dd59c00ef151adca49d (127.15.255.132:43607)
I20240502 21:25:38.325731 18329 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.255.132:34713
I20240502 21:25:38.468125 18658 heartbeater.cc:503] Master 127.15.255.190:36947 was elected leader, sending a full tablet report...
W20240502 21:25:38.627991 18927 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:25:38.628253 18927 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:25:38.628382 18927 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:25:38.635607 18927 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 21:25:38.635730 18927 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:25:38.635881 18927 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.255.133
I20240502 21:25:38.644609 18927 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.255.148:40185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.255.133:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.15.255.133
--webserver_port=0
--tserver_master_addrs=127.15.255.190:36947
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.255.133
--log_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:25:38.645779 18927 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:25:38.647392 18927 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:25:38.658823 18935 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:38.659232 18933 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:38.659976 18932 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:25:38.660594 18927 server_base.cc:1034] running on GCE node
I20240502 21:25:38.661113 18927 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:25:38.661917 18927 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:25:38.663103 18927 hybrid_clock.cc:648] HybridClock initialized: now 1714685138663066 us; error 32 us; skew 500 ppm
I20240502 21:25:38.663383 18927 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:25:38.665402 18927 webserver.cc:460] Webserver started at http://127.15.255.133:45399/ using document root <none> and password file <none>
I20240502 21:25:38.666066 18927 fs_manager.cc:359] Metadata directory not provided
I20240502 21:25:38.666172 18927 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:25:38.666478 18927 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:25:38.668964 18927 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "9be399440ded4e2ebe6b6e0bf486cbb1"
format_stamp: "Formatted at 2024-05-02 21:25:38 on dist-test-slave-jdfx"
I20240502 21:25:38.669647 18927 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "9be399440ded4e2ebe6b6e0bf486cbb1"
format_stamp: "Formatted at 2024-05-02 21:25:38 on dist-test-slave-jdfx"
I20240502 21:25:38.674309 18927 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20240502 21:25:38.677322 18941 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:38.678504 18927 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240502 21:25:38.678647 18927 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "9be399440ded4e2ebe6b6e0bf486cbb1"
format_stamp: "Formatted at 2024-05-02 21:25:38 on dist-test-slave-jdfx"
I20240502 21:25:38.678820 18927 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:25:38.701238 18927 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:25:38.702029 18927 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:25:38.702296 18927 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:25:38.703186 18927 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:25:38.704627 18927 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:25:38.704720 18927 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:38.704828 18927 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:25:38.704896 18927 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:38.733798 18927 rpc_server.cc:307] RPC server started. Bound to: 127.15.255.133:34217
I20240502 21:25:38.733845 19057 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.255.133:34217 every 8 connection(s)
I20240502 21:25:38.735025 18927 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240502 21:25:38.738799 16382 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvNciQZ/build/asan/bin/kudu as pid 18927
I20240502 21:25:38.738976 16382 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240502 21:25:38.742163 16382 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvNciQZ/build/asan/bin/kudu
/tmp/dist-test-taskvNciQZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.255.134:0
--local_ip_for_outbound_sockets=127.15.255.134
--webserver_interface=127.15.255.134
--webserver_port=0
--tserver_master_addrs=127.15.255.190:36947
--builtin_ntp_servers=127.15.255.148:40185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 21:25:38.747617 19058 heartbeater.cc:347] Connected to a master server at 127.15.255.190:36947
I20240502 21:25:38.747879 19058 heartbeater.cc:464] Registering TS with master...
I20240502 21:25:38.748658 19058 heartbeater.cc:511] Master 127.15.255.190:36947 requested a full tablet report, sending...
I20240502 21:25:38.750103 18329 ts_manager.cc:194] Registered new tserver with Master: 9be399440ded4e2ebe6b6e0bf486cbb1 (127.15.255.133:34217)
I20240502 21:25:38.751152 18329 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.255.133:54549
I20240502 21:25:38.886998 18791 heartbeater.cc:503] Master 127.15.255.190:36947 was elected leader, sending a full tablet report...
W20240502 21:25:39.065256 19061 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:25:39.065536 19061 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:25:39.065678 19061 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:25:39.073024 19061 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 21:25:39.073137 19061 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:25:39.073297 19061 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.255.134
I20240502 21:25:39.081902 19061 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.255.148:40185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.255.134:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.15.255.134
--webserver_port=0
--tserver_master_addrs=127.15.255.190:36947
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.255.134
--log_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:25:39.083099 19061 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:25:39.084841 19061 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:25:39.096500 19067 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:39.097167 19069 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:39.097558 19066 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:25:39.098137 19061 server_base.cc:1034] running on GCE node
I20240502 21:25:39.098735 19061 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:25:39.099726 19061 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:25:39.100909 19061 hybrid_clock.cc:648] HybridClock initialized: now 1714685139100867 us; error 35 us; skew 500 ppm
I20240502 21:25:39.101194 19061 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:25:39.103155 19061 webserver.cc:460] Webserver started at http://127.15.255.134:40495/ using document root <none> and password file <none>
I20240502 21:25:39.103792 19061 fs_manager.cc:359] Metadata directory not provided
I20240502 21:25:39.103897 19061 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:25:39.104177 19061 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:25:39.106516 19061 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "2877108756304e05bd60ba1186b870da"
format_stamp: "Formatted at 2024-05-02 21:25:39 on dist-test-slave-jdfx"
I20240502 21:25:39.107198 19061 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "2877108756304e05bd60ba1186b870da"
format_stamp: "Formatted at 2024-05-02 21:25:39 on dist-test-slave-jdfx"
I20240502 21:25:39.111727 19061 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20240502 21:25:39.114773 19075 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:39.115926 19061 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20240502 21:25:39.116065 19061 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "2877108756304e05bd60ba1186b870da"
format_stamp: "Formatted at 2024-05-02 21:25:39 on dist-test-slave-jdfx"
I20240502 21:25:39.116232 19061 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:25:39.132452 19061 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:25:39.133137 19061 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:25:39.133373 19061 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:25:39.134164 19061 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:25:39.135465 19061 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:25:39.135555 19061 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:39.135658 19061 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:25:39.135723 19061 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:39.163935 19061 rpc_server.cc:307] RPC server started. Bound to: 127.15.255.134:43635
I20240502 21:25:39.163982 19191 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.255.134:43635 every 8 connection(s)
I20240502 21:25:39.165153 19061 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240502 21:25:39.168303 16382 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvNciQZ/build/asan/bin/kudu as pid 19061
I20240502 21:25:39.168557 16382 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240502 21:25:39.171636 16382 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvNciQZ/build/asan/bin/kudu
/tmp/dist-test-taskvNciQZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.255.135:0
--local_ip_for_outbound_sockets=127.15.255.135
--webserver_interface=127.15.255.135
--webserver_port=0
--tserver_master_addrs=127.15.255.190:36947
--builtin_ntp_servers=127.15.255.148:40185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 21:25:39.177311 19192 heartbeater.cc:347] Connected to a master server at 127.15.255.190:36947
I20240502 21:25:39.177601 19192 heartbeater.cc:464] Registering TS with master...
I20240502 21:25:39.178495 19192 heartbeater.cc:511] Master 127.15.255.190:36947 requested a full tablet report, sending...
I20240502 21:25:39.179870 18329 ts_manager.cc:194] Registered new tserver with Master: 2877108756304e05bd60ba1186b870da (127.15.255.134:43635)
I20240502 21:25:39.180840 18329 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.255.134:49343
I20240502 21:25:39.327926 18924 heartbeater.cc:503] Master 127.15.255.190:36947 was elected leader, sending a full tablet report...
W20240502 21:25:39.472213 19195 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:25:39.472467 19195 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:25:39.472570 19195 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:25:39.479517 19195 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 21:25:39.479614 19195 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:25:39.479728 19195 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.255.135
I20240502 21:25:39.488219 19195 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.255.148:40185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.255.135:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.15.255.135
--webserver_port=0
--tserver_master_addrs=127.15.255.190:36947
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.255.135
--log_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:25:39.489295 19195 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:25:39.490818 19195 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:25:39.501868 19200 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:39.502545 19203 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:39.503073 19201 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:25:39.503697 19195 server_base.cc:1034] running on GCE node
I20240502 21:25:39.504225 19195 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:25:39.505092 19195 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:25:39.506294 19195 hybrid_clock.cc:648] HybridClock initialized: now 1714685139506253 us; error 30 us; skew 500 ppm
I20240502 21:25:39.506587 19195 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:25:39.508579 19195 webserver.cc:460] Webserver started at http://127.15.255.135:46157/ using document root <none> and password file <none>
I20240502 21:25:39.509230 19195 fs_manager.cc:359] Metadata directory not provided
I20240502 21:25:39.509336 19195 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:25:39.509616 19195 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:25:39.512004 19195 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-6/data/instance:
uuid: "90e329b457c24caf93a9f880345eb288"
format_stamp: "Formatted at 2024-05-02 21:25:39 on dist-test-slave-jdfx"
I20240502 21:25:39.512786 19195 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-6/wal/instance:
uuid: "90e329b457c24caf93a9f880345eb288"
format_stamp: "Formatted at 2024-05-02 21:25:39 on dist-test-slave-jdfx"
I20240502 21:25:39.517410 19195 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20240502 21:25:39.520539 19209 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:39.521840 19195 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240502 21:25:39.521986 19195 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "90e329b457c24caf93a9f880345eb288"
format_stamp: "Formatted at 2024-05-02 21:25:39 on dist-test-slave-jdfx"
I20240502 21:25:39.522166 19195 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:25:39.532963 19195 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:25:39.533654 19195 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:25:39.533895 19195 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:25:39.534706 19195 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:25:39.536077 19195 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:25:39.536168 19195 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:39.536268 19195 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:25:39.536334 19195 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:39.572108 19195 rpc_server.cc:307] RPC server started. Bound to: 127.15.255.135:43395
I20240502 21:25:39.572165 19325 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.255.135:43395 every 8 connection(s)
I20240502 21:25:39.573331 19195 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240502 21:25:39.579906 16382 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvNciQZ/build/asan/bin/kudu as pid 19195
I20240502 21:25:39.580066 16382 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-6/wal/instance
I20240502 21:25:39.584861 19326 heartbeater.cc:347] Connected to a master server at 127.15.255.190:36947
I20240502 21:25:39.585117 19326 heartbeater.cc:464] Registering TS with master...
I20240502 21:25:39.585848 19326 heartbeater.cc:511] Master 127.15.255.190:36947 requested a full tablet report, sending...
I20240502 21:25:39.587090 18329 ts_manager.cc:194] Registered new tserver with Master: 90e329b457c24caf93a9f880345eb288 (127.15.255.135:43395)
I20240502 21:25:39.587817 18329 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.255.135:37679
I20240502 21:25:39.595352 16382 external_mini_cluster.cc:930] 7 TS(s) registered with all masters
I20240502 21:25:39.611193 18329 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60616:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 5
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20240502 21:25:39.647898 18590 tablet_service.cc:1467] Processing CreateTablet for tablet 40d4f61d0f6d4d31a5626c7743c88133 (DEFAULT_TABLE table=TestTable [id=f30df752b55b4a89883ebf66115566aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:39.649336 18590 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 40d4f61d0f6d4d31a5626c7743c88133. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:39.661557 18724 tablet_service.cc:1467] Processing CreateTablet for tablet 40d4f61d0f6d4d31a5626c7743c88133 (DEFAULT_TABLE table=TestTable [id=f30df752b55b4a89883ebf66115566aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:39.663228 18724 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 40d4f61d0f6d4d31a5626c7743c88133. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:39.664114 18453 tablet_service.cc:1467] Processing CreateTablet for tablet 40d4f61d0f6d4d31a5626c7743c88133 (DEFAULT_TABLE table=TestTable [id=f30df752b55b4a89883ebf66115566aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:39.665632 18453 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 40d4f61d0f6d4d31a5626c7743c88133. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:39.671967 19258 tablet_service.cc:1467] Processing CreateTablet for tablet 40d4f61d0f6d4d31a5626c7743c88133 (DEFAULT_TABLE table=TestTable [id=f30df752b55b4a89883ebf66115566aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:39.673278 19258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 40d4f61d0f6d4d31a5626c7743c88133. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:39.679901 19347 tablet_bootstrap.cc:492] T 40d4f61d0f6d4d31a5626c7743c88133 P 82f50f4b66f3450aa9f58a0b317f5cea: Bootstrap starting.
I20240502 21:25:39.681078 18990 tablet_service.cc:1467] Processing CreateTablet for tablet 40d4f61d0f6d4d31a5626c7743c88133 (DEFAULT_TABLE table=TestTable [id=f30df752b55b4a89883ebf66115566aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:39.682049 19348 tablet_bootstrap.cc:492] T 40d4f61d0f6d4d31a5626c7743c88133 P 501e3c0d44fe4d70af9ff926d9622417: Bootstrap starting.
I20240502 21:25:39.682287 18990 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 40d4f61d0f6d4d31a5626c7743c88133. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:39.684386 19347 tablet_bootstrap.cc:654] T 40d4f61d0f6d4d31a5626c7743c88133 P 82f50f4b66f3450aa9f58a0b317f5cea: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:39.685628 19347 log.cc:824] T 40d4f61d0f6d4d31a5626c7743c88133 P 82f50f4b66f3450aa9f58a0b317f5cea: Log is configured to *not* fsync() on all Append() calls
I20240502 21:25:39.686002 19348 tablet_bootstrap.cc:654] T 40d4f61d0f6d4d31a5626c7743c88133 P 501e3c0d44fe4d70af9ff926d9622417: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:39.687081 19350 tablet_bootstrap.cc:492] T 40d4f61d0f6d4d31a5626c7743c88133 P 90e329b457c24caf93a9f880345eb288: Bootstrap starting.
I20240502 21:25:39.687256 19348 log.cc:824] T 40d4f61d0f6d4d31a5626c7743c88133 P 501e3c0d44fe4d70af9ff926d9622417: Log is configured to *not* fsync() on all Append() calls
I20240502 21:25:39.690232 19350 tablet_bootstrap.cc:654] T 40d4f61d0f6d4d31a5626c7743c88133 P 90e329b457c24caf93a9f880345eb288: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:39.692893 19352 tablet_bootstrap.cc:492] T 40d4f61d0f6d4d31a5626c7743c88133 P faadabfb099b4abf90a466395be585b5: Bootstrap starting.
I20240502 21:25:39.693567 19350 log.cc:824] T 40d4f61d0f6d4d31a5626c7743c88133 P 90e329b457c24caf93a9f880345eb288: Log is configured to *not* fsync() on all Append() calls
I20240502 21:25:39.697845 19352 tablet_bootstrap.cc:654] T 40d4f61d0f6d4d31a5626c7743c88133 P faadabfb099b4abf90a466395be585b5: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:39.699090 19352 log.cc:824] T 40d4f61d0f6d4d31a5626c7743c88133 P faadabfb099b4abf90a466395be585b5: Log is configured to *not* fsync() on all Append() calls
I20240502 21:25:39.701107 19350 tablet_bootstrap.cc:492] T 40d4f61d0f6d4d31a5626c7743c88133 P 90e329b457c24caf93a9f880345eb288: No bootstrap required, opened a new log
I20240502 21:25:39.701155 19348 tablet_bootstrap.cc:492] T 40d4f61d0f6d4d31a5626c7743c88133 P 501e3c0d44fe4d70af9ff926d9622417: No bootstrap required, opened a new log
I20240502 21:25:39.701383 19350 ts_tablet_manager.cc:1367] T 40d4f61d0f6d4d31a5626c7743c88133 P 90e329b457c24caf93a9f880345eb288: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.001s
I20240502 21:25:39.701488 19348 ts_tablet_manager.cc:1367] T 40d4f61d0f6d4d31a5626c7743c88133 P 501e3c0d44fe4d70af9ff926d9622417: Time spent bootstrapping tablet: real 0.020s user 0.005s sys 0.013s
I20240502 21:25:39.701746 19347 tablet_bootstrap.cc:492] T 40d4f61d0f6d4d31a5626c7743c88133 P 82f50f4b66f3450aa9f58a0b317f5cea: No bootstrap required, opened a new log
I20240502 21:25:39.702044 19347 ts_tablet_manager.cc:1367] T 40d4f61d0f6d4d31a5626c7743c88133 P 82f50f4b66f3450aa9f58a0b317f5cea: Time spent bootstrapping tablet: real 0.022s user 0.008s sys 0.010s
I20240502 21:25:39.707902 19352 tablet_bootstrap.cc:492] T 40d4f61d0f6d4d31a5626c7743c88133 P faadabfb099b4abf90a466395be585b5: No bootstrap required, opened a new log
I20240502 21:25:39.708199 19352 ts_tablet_manager.cc:1367] T 40d4f61d0f6d4d31a5626c7743c88133 P faadabfb099b4abf90a466395be585b5: Time spent bootstrapping tablet: real 0.016s user 0.009s sys 0.003s
I20240502 21:25:39.709081 19354 tablet_bootstrap.cc:492] T 40d4f61d0f6d4d31a5626c7743c88133 P 9be399440ded4e2ebe6b6e0bf486cbb1: Bootstrap starting.
I20240502 21:25:39.709908 19348 raft_consensus.cc:348] T 40d4f61d0f6d4d31a5626c7743c88133 P 501e3c0d44fe4d70af9ff926d9622417 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } }
I20240502 21:25:39.710364 19348 raft_consensus.cc:374] T 40d4f61d0f6d4d31a5626c7743c88133 P 501e3c0d44fe4d70af9ff926d9622417 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:39.710500 19348 raft_consensus.cc:729] T 40d4f61d0f6d4d31a5626c7743c88133 P 501e3c0d44fe4d70af9ff926d9622417 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 501e3c0d44fe4d70af9ff926d9622417, State: Initialized, Role: FOLLOWER
I20240502 21:25:39.710505 19347 raft_consensus.cc:348] T 40d4f61d0f6d4d31a5626c7743c88133 P 82f50f4b66f3450aa9f58a0b317f5cea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } }
I20240502 21:25:39.710963 19347 raft_consensus.cc:374] T 40d4f61d0f6d4d31a5626c7743c88133 P 82f50f4b66f3450aa9f58a0b317f5cea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:39.711092 19347 raft_consensus.cc:729] T 40d4f61d0f6d4d31a5626c7743c88133 P 82f50f4b66f3450aa9f58a0b317f5cea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82f50f4b66f3450aa9f58a0b317f5cea, State: Initialized, Role: FOLLOWER
I20240502 21:25:39.711360 19348 consensus_queue.cc:260] T 40d4f61d0f6d4d31a5626c7743c88133 P 501e3c0d44fe4d70af9ff926d9622417 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } }
I20240502 21:25:39.711833 19347 consensus_queue.cc:260] T 40d4f61d0f6d4d31a5626c7743c88133 P 82f50f4b66f3450aa9f58a0b317f5cea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } }
I20240502 21:25:39.712206 19354 tablet_bootstrap.cc:654] T 40d4f61d0f6d4d31a5626c7743c88133 P 9be399440ded4e2ebe6b6e0bf486cbb1: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:39.713426 19354 log.cc:824] T 40d4f61d0f6d4d31a5626c7743c88133 P 9be399440ded4e2ebe6b6e0bf486cbb1: Log is configured to *not* fsync() on all Append() calls
I20240502 21:25:39.716059 19350 raft_consensus.cc:348] T 40d4f61d0f6d4d31a5626c7743c88133 P 90e329b457c24caf93a9f880345eb288 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } }
I20240502 21:25:39.716526 19350 raft_consensus.cc:374] T 40d4f61d0f6d4d31a5626c7743c88133 P 90e329b457c24caf93a9f880345eb288 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:39.716632 19350 raft_consensus.cc:729] T 40d4f61d0f6d4d31a5626c7743c88133 P 90e329b457c24caf93a9f880345eb288 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90e329b457c24caf93a9f880345eb288, State: Initialized, Role: FOLLOWER
I20240502 21:25:39.717304 19350 consensus_queue.cc:260] T 40d4f61d0f6d4d31a5626c7743c88133 P 90e329b457c24caf93a9f880345eb288 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } }
I20240502 21:25:39.718605 19352 raft_consensus.cc:348] T 40d4f61d0f6d4d31a5626c7743c88133 P faadabfb099b4abf90a466395be585b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } }
I20240502 21:25:39.719111 19352 raft_consensus.cc:374] T 40d4f61d0f6d4d31a5626c7743c88133 P faadabfb099b4abf90a466395be585b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:39.719252 19352 raft_consensus.cc:729] T 40d4f61d0f6d4d31a5626c7743c88133 P faadabfb099b4abf90a466395be585b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: faadabfb099b4abf90a466395be585b5, State: Initialized, Role: FOLLOWER
I20240502 21:25:39.720055 19352 consensus_queue.cc:260] T 40d4f61d0f6d4d31a5626c7743c88133 P faadabfb099b4abf90a466395be585b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } }
I20240502 21:25:39.722934 19354 tablet_bootstrap.cc:492] T 40d4f61d0f6d4d31a5626c7743c88133 P 9be399440ded4e2ebe6b6e0bf486cbb1: No bootstrap required, opened a new log
I20240502 21:25:39.723263 19354 ts_tablet_manager.cc:1367] T 40d4f61d0f6d4d31a5626c7743c88133 P 9be399440ded4e2ebe6b6e0bf486cbb1: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.005s
I20240502 21:25:39.738701 19348 ts_tablet_manager.cc:1398] T 40d4f61d0f6d4d31a5626c7743c88133 P 501e3c0d44fe4d70af9ff926d9622417: Time spent starting tablet: real 0.037s user 0.010s sys 0.020s
I20240502 21:25:39.745692 19347 ts_tablet_manager.cc:1398] T 40d4f61d0f6d4d31a5626c7743c88133 P 82f50f4b66f3450aa9f58a0b317f5cea: Time spent starting tablet: real 0.043s user 0.028s sys 0.012s
I20240502 21:25:39.746209 19352 ts_tablet_manager.cc:1398] T 40d4f61d0f6d4d31a5626c7743c88133 P faadabfb099b4abf90a466395be585b5: Time spent starting tablet: real 0.038s user 0.007s sys 0.003s
I20240502 21:25:39.747081 19326 heartbeater.cc:503] Master 127.15.255.190:36947 was elected leader, sending a full tablet report...
I20240502 21:25:39.732256 19354 raft_consensus.cc:348] T 40d4f61d0f6d4d31a5626c7743c88133 P 9be399440ded4e2ebe6b6e0bf486cbb1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } }
I20240502 21:25:39.747254 19354 raft_consensus.cc:374] T 40d4f61d0f6d4d31a5626c7743c88133 P 9be399440ded4e2ebe6b6e0bf486cbb1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:39.747357 19354 raft_consensus.cc:729] T 40d4f61d0f6d4d31a5626c7743c88133 P 9be399440ded4e2ebe6b6e0bf486cbb1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9be399440ded4e2ebe6b6e0bf486cbb1, State: Initialized, Role: FOLLOWER
I20240502 21:25:39.748073 19354 consensus_queue.cc:260] T 40d4f61d0f6d4d31a5626c7743c88133 P 9be399440ded4e2ebe6b6e0bf486cbb1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } }
I20240502 21:25:39.749549 19058 heartbeater.cc:503] Master 127.15.255.190:36947 was elected leader, sending a full tablet report...
I20240502 21:25:39.749744 19350 ts_tablet_manager.cc:1398] T 40d4f61d0f6d4d31a5626c7743c88133 P 90e329b457c24caf93a9f880345eb288: Time spent starting tablet: real 0.048s user 0.014s sys 0.023s
I20240502 21:25:39.750344 19354 ts_tablet_manager.cc:1398] T 40d4f61d0f6d4d31a5626c7743c88133 P 9be399440ded4e2ebe6b6e0bf486cbb1: Time spent starting tablet: real 0.027s user 0.011s sys 0.000s
W20240502 21:25:39.782603 18525 tablet.cc:2369] T 40d4f61d0f6d4d31a5626c7743c88133 P faadabfb099b4abf90a466395be585b5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240502 21:25:39.824772 19327 tablet.cc:2369] T 40d4f61d0f6d4d31a5626c7743c88133 P 90e329b457c24caf93a9f880345eb288: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 21:25:39.828745 19360 raft_consensus.cc:482] T 40d4f61d0f6d4d31a5626c7743c88133 P faadabfb099b4abf90a466395be585b5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:25:39.828980 19360 raft_consensus.cc:504] T 40d4f61d0f6d4d31a5626c7743c88133 P faadabfb099b4abf90a466395be585b5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } }
I20240502 21:25:39.830900 19360 leader_election.cc:288] T 40d4f61d0f6d4d31a5626c7743c88133 P faadabfb099b4abf90a466395be585b5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 82f50f4b66f3450aa9f58a0b317f5cea (127.15.255.130:45139), 90e329b457c24caf93a9f880345eb288 (127.15.255.135:43395), 501e3c0d44fe4d70af9ff926d9622417 (127.15.255.131:42501), 9be399440ded4e2ebe6b6e0bf486cbb1 (127.15.255.133:34217)
I20240502 21:25:39.846680 18611 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40d4f61d0f6d4d31a5626c7743c88133" candidate_uuid: "faadabfb099b4abf90a466395be585b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" is_pre_election: true
I20240502 21:25:39.847280 18611 raft_consensus.cc:2454] T 40d4f61d0f6d4d31a5626c7743c88133 P 82f50f4b66f3450aa9f58a0b317f5cea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate faadabfb099b4abf90a466395be585b5 in term 0.
I20240502 21:25:39.848827 19279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40d4f61d0f6d4d31a5626c7743c88133" candidate_uuid: "faadabfb099b4abf90a466395be585b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90e329b457c24caf93a9f880345eb288" is_pre_election: true
I20240502 21:25:39.849319 19279 raft_consensus.cc:2454] T 40d4f61d0f6d4d31a5626c7743c88133 P 90e329b457c24caf93a9f880345eb288 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate faadabfb099b4abf90a466395be585b5 in term 0.
I20240502 21:25:39.849898 18406 leader_election.cc:302] T 40d4f61d0f6d4d31a5626c7743c88133 P faadabfb099b4abf90a466395be585b5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 82f50f4b66f3450aa9f58a0b317f5cea, 90e329b457c24caf93a9f880345eb288, faadabfb099b4abf90a466395be585b5; no voters:
I20240502 21:25:39.850358 19360 raft_consensus.cc:2781] T 40d4f61d0f6d4d31a5626c7743c88133 P faadabfb099b4abf90a466395be585b5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:25:39.850512 19360 raft_consensus.cc:482] T 40d4f61d0f6d4d31a5626c7743c88133 P faadabfb099b4abf90a466395be585b5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:25:39.850606 19360 raft_consensus.cc:3037] T 40d4f61d0f6d4d31a5626c7743c88133 P faadabfb099b4abf90a466395be585b5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:39.851753 19010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40d4f61d0f6d4d31a5626c7743c88133" candidate_uuid: "faadabfb099b4abf90a466395be585b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" is_pre_election: true
I20240502 21:25:39.852151 19010 raft_consensus.cc:2454] T 40d4f61d0f6d4d31a5626c7743c88133 P 9be399440ded4e2ebe6b6e0bf486cbb1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate faadabfb099b4abf90a466395be585b5 in term 0.
I20240502 21:25:39.852382 18744 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40d4f61d0f6d4d31a5626c7743c88133" candidate_uuid: "faadabfb099b4abf90a466395be585b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "501e3c0d44fe4d70af9ff926d9622417" is_pre_election: true
I20240502 21:25:39.852877 18744 raft_consensus.cc:2454] T 40d4f61d0f6d4d31a5626c7743c88133 P 501e3c0d44fe4d70af9ff926d9622417 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate faadabfb099b4abf90a466395be585b5 in term 0.
I20240502 21:25:39.853307 19360 raft_consensus.cc:504] T 40d4f61d0f6d4d31a5626c7743c88133 P faadabfb099b4abf90a466395be585b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } }
I20240502 21:25:39.854126 19360 leader_election.cc:288] T 40d4f61d0f6d4d31a5626c7743c88133 P faadabfb099b4abf90a466395be585b5 [CANDIDATE]: Term 1 election: Requested vote from peers 82f50f4b66f3450aa9f58a0b317f5cea (127.15.255.130:45139), 90e329b457c24caf93a9f880345eb288 (127.15.255.135:43395), 501e3c0d44fe4d70af9ff926d9622417 (127.15.255.131:42501), 9be399440ded4e2ebe6b6e0bf486cbb1 (127.15.255.133:34217)
I20240502 21:25:39.854270 18611 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40d4f61d0f6d4d31a5626c7743c88133" candidate_uuid: "faadabfb099b4abf90a466395be585b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82f50f4b66f3450aa9f58a0b317f5cea"
I20240502 21:25:39.854456 18611 raft_consensus.cc:3037] T 40d4f61d0f6d4d31a5626c7743c88133 P 82f50f4b66f3450aa9f58a0b317f5cea [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:39.854521 18744 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40d4f61d0f6d4d31a5626c7743c88133" candidate_uuid: "faadabfb099b4abf90a466395be585b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "501e3c0d44fe4d70af9ff926d9622417"
I20240502 21:25:39.854712 18744 raft_consensus.cc:3037] T 40d4f61d0f6d4d31a5626c7743c88133 P 501e3c0d44fe4d70af9ff926d9622417 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:39.854738 19010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40d4f61d0f6d4d31a5626c7743c88133" candidate_uuid: "faadabfb099b4abf90a466395be585b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1"
I20240502 21:25:39.854842 19279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40d4f61d0f6d4d31a5626c7743c88133" candidate_uuid: "faadabfb099b4abf90a466395be585b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90e329b457c24caf93a9f880345eb288"
I20240502 21:25:39.854907 19010 raft_consensus.cc:3037] T 40d4f61d0f6d4d31a5626c7743c88133 P 9be399440ded4e2ebe6b6e0bf486cbb1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:39.855024 19279 raft_consensus.cc:3037] T 40d4f61d0f6d4d31a5626c7743c88133 P 90e329b457c24caf93a9f880345eb288 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:39.857465 18611 raft_consensus.cc:2454] T 40d4f61d0f6d4d31a5626c7743c88133 P 82f50f4b66f3450aa9f58a0b317f5cea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate faadabfb099b4abf90a466395be585b5 in term 1.
I20240502 21:25:39.857610 18744 raft_consensus.cc:2454] T 40d4f61d0f6d4d31a5626c7743c88133 P 501e3c0d44fe4d70af9ff926d9622417 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate faadabfb099b4abf90a466395be585b5 in term 1.
I20240502 21:25:39.857846 19279 raft_consensus.cc:2454] T 40d4f61d0f6d4d31a5626c7743c88133 P 90e329b457c24caf93a9f880345eb288 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate faadabfb099b4abf90a466395be585b5 in term 1.
I20240502 21:25:39.858002 18404 leader_election.cc:302] T 40d4f61d0f6d4d31a5626c7743c88133 P faadabfb099b4abf90a466395be585b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 501e3c0d44fe4d70af9ff926d9622417, 82f50f4b66f3450aa9f58a0b317f5cea, faadabfb099b4abf90a466395be585b5; no voters:
I20240502 21:25:39.858103 19010 raft_consensus.cc:2454] T 40d4f61d0f6d4d31a5626c7743c88133 P 9be399440ded4e2ebe6b6e0bf486cbb1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate faadabfb099b4abf90a466395be585b5 in term 1.
I20240502 21:25:39.858327 19360 raft_consensus.cc:2781] T 40d4f61d0f6d4d31a5626c7743c88133 P faadabfb099b4abf90a466395be585b5 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:25:39.858750 19360 raft_consensus.cc:686] T 40d4f61d0f6d4d31a5626c7743c88133 P faadabfb099b4abf90a466395be585b5 [term 1 LEADER]: Becoming Leader. State: Replica: faadabfb099b4abf90a466395be585b5, State: Running, Role: LEADER
I20240502 21:25:39.859189 19360 consensus_queue.cc:237] T 40d4f61d0f6d4d31a5626c7743c88133 P faadabfb099b4abf90a466395be585b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } }
I20240502 21:25:39.863942 18329 catalog_manager.cc:5484] T 40d4f61d0f6d4d31a5626c7743c88133 P faadabfb099b4abf90a466395be585b5 reported cstate change: term changed from 0 to 1, leader changed from <none> to faadabfb099b4abf90a466395be585b5 (127.15.255.129). New cstate: current_term: 1 leader_uuid: "faadabfb099b4abf90a466395be585b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } health_report { overall_health: UNKNOWN } } }
W20240502 21:25:39.871583 18792 tablet.cc:2369] T 40d4f61d0f6d4d31a5626c7743c88133 P 501e3c0d44fe4d70af9ff926d9622417: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 21:25:39.890933 16382 external_mini_cluster.cc:930] 7 TS(s) registered with all masters
I20240502 21:25:39.892944 16382 ts_itest-base.cc:233] Waiting for 1 tablets on tserver faadabfb099b4abf90a466395be585b5 to finish bootstrapping
I20240502 21:25:39.901031 16382 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 82f50f4b66f3450aa9f58a0b317f5cea to finish bootstrapping
I20240502 21:25:39.907356 16382 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 501e3c0d44fe4d70af9ff926d9622417 to finish bootstrapping
I20240502 21:25:39.913772 16382 ts_itest-base.cc:233] Waiting for 0 tablets on tserver c75831edb3b44dd59c00ef151adca49d to finish bootstrapping
I20240502 21:25:39.921756 16382 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 9be399440ded4e2ebe6b6e0bf486cbb1 to finish bootstrapping
I20240502 21:25:39.928025 16382 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 2877108756304e05bd60ba1186b870da to finish bootstrapping
I20240502 21:25:39.935468 16382 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 90e329b457c24caf93a9f880345eb288 to finish bootstrapping
I20240502 21:25:39.941994 16382 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvNciQZ/build/asan/bin/kudu pid 19195
I20240502 21:25:39.942778 19321 generic_service.cc:196] Checking for leaks (request via RPC)
W20240502 21:25:39.953053 18659 tablet.cc:2369] T 40d4f61d0f6d4d31a5626c7743c88133 P 82f50f4b66f3450aa9f58a0b317f5cea: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240502 21:25:39.987250 19059 tablet.cc:2369] T 40d4f61d0f6d4d31a5626c7743c88133 P 9be399440ded4e2ebe6b6e0bf486cbb1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 21:25:40.025687 16382 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvNciQZ/build/asan/bin/kudu with pid 19195
I20240502 21:25:40.183351 19192 heartbeater.cc:503] Master 127.15.255.190:36947 was elected leader, sending a full tablet report...
W20240502 21:25:40.244820 18406 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.255.135:43395: connect: Connection refused (error 111)
W20240502 21:25:40.246244 18406 consensus_peers.cc:485] T 40d4f61d0f6d4d31a5626c7743c88133 P faadabfb099b4abf90a466395be585b5 -> Peer 90e329b457c24caf93a9f880345eb288 (127.15.255.135:43395): Couldn't send request to peer 90e329b457c24caf93a9f880345eb288. Status: Network error: Client connection negotiation failed: client connection to 127.15.255.135:43395: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 21:25:40.334112 19360 consensus_queue.cc:1035] T 40d4f61d0f6d4d31a5626c7743c88133 P faadabfb099b4abf90a466395be585b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:25:40.369609 19360 consensus_queue.cc:1035] T 40d4f61d0f6d4d31a5626c7743c88133 P faadabfb099b4abf90a466395be585b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:25:40.376289 19360 consensus_queue.cc:1035] T 40d4f61d0f6d4d31a5626c7743c88133 P faadabfb099b4abf90a466395be585b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 21:25:42.897326 18406 consensus_peers.cc:485] T 40d4f61d0f6d4d31a5626c7743c88133 P faadabfb099b4abf90a466395be585b5 -> Peer 90e329b457c24caf93a9f880345eb288 (127.15.255.135:43395): Couldn't send request to peer 90e329b457c24caf93a9f880345eb288. Status: Network error: Client connection negotiation failed: client connection to 127.15.255.135:43395: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 21:25:43.791813 18326 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60616:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 5
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 21:25:43.803583 18724 tablet_service.cc:1467] Processing CreateTablet for tablet 32c9fc5a18c54207881124fda86f42ff (DEFAULT_TABLE table=rebalance_test_table_0 [id=086ede98fc604333833bce759f342073]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:43.803711 18453 tablet_service.cc:1467] Processing CreateTablet for tablet 32c9fc5a18c54207881124fda86f42ff (DEFAULT_TABLE table=rebalance_test_table_0 [id=086ede98fc604333833bce759f342073]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:43.804096 18724 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32c9fc5a18c54207881124fda86f42ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:43.804203 18453 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32c9fc5a18c54207881124fda86f42ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:43.809234 18590 tablet_service.cc:1467] Processing CreateTablet for tablet 32c9fc5a18c54207881124fda86f42ff (DEFAULT_TABLE table=rebalance_test_table_0 [id=086ede98fc604333833bce759f342073]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:43.809724 18590 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32c9fc5a18c54207881124fda86f42ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:43.814356 18452 tablet_service.cc:1467] Processing CreateTablet for tablet 84b235fd89634168938c0dddb534462a (DEFAULT_TABLE table=rebalance_test_table_0 [id=086ede98fc604333833bce759f342073]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:43.814430 18989 tablet_service.cc:1467] Processing CreateTablet for tablet ed78d40318fe4ab5bee4a01b182a34b6 (DEFAULT_TABLE table=rebalance_test_table_0 [id=086ede98fc604333833bce759f342073]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:43.814699 18451 tablet_service.cc:1467] Processing CreateTablet for tablet ed78d40318fe4ab5bee4a01b182a34b6 (DEFAULT_TABLE table=rebalance_test_table_0 [id=086ede98fc604333833bce759f342073]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:43.814841 18452 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84b235fd89634168938c0dddb534462a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:43.814868 18989 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed78d40318fe4ab5bee4a01b182a34b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:43.815187 18451 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed78d40318fe4ab5bee4a01b182a34b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:43.816293 18723 tablet_service.cc:1467] Processing CreateTablet for tablet ed78d40318fe4ab5bee4a01b182a34b6 (DEFAULT_TABLE table=rebalance_test_table_0 [id=086ede98fc604333833bce759f342073]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:43.816681 18723 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed78d40318fe4ab5bee4a01b182a34b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:43.819536 18990 tablet_service.cc:1467] Processing CreateTablet for tablet 84b235fd89634168938c0dddb534462a (DEFAULT_TABLE table=rebalance_test_table_0 [id=086ede98fc604333833bce759f342073]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:43.819917 18990 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84b235fd89634168938c0dddb534462a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:43.820950 18589 tablet_service.cc:1467] Processing CreateTablet for tablet 84b235fd89634168938c0dddb534462a (DEFAULT_TABLE table=rebalance_test_table_0 [id=086ede98fc604333833bce759f342073]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:43.821354 18589 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84b235fd89634168938c0dddb534462a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:43.828845 18855 tablet_service.cc:1467] Processing CreateTablet for tablet ed78d40318fe4ab5bee4a01b182a34b6 (DEFAULT_TABLE table=rebalance_test_table_0 [id=086ede98fc604333833bce759f342073]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:43.829536 18856 tablet_service.cc:1467] Processing CreateTablet for tablet 84b235fd89634168938c0dddb534462a (DEFAULT_TABLE table=rebalance_test_table_0 [id=086ede98fc604333833bce759f342073]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:43.829667 19405 tablet_bootstrap.cc:492] T ed78d40318fe4ab5bee4a01b182a34b6 P 9be399440ded4e2ebe6b6e0bf486cbb1: Bootstrap starting.
I20240502 21:25:43.828251 18857 tablet_service.cc:1467] Processing CreateTablet for tablet 32c9fc5a18c54207881124fda86f42ff (DEFAULT_TABLE table=rebalance_test_table_0 [id=086ede98fc604333833bce759f342073]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:43.830332 18855 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed78d40318fe4ab5bee4a01b182a34b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:43.830766 18856 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84b235fd89634168938c0dddb534462a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:43.832630 19405 tablet_bootstrap.cc:654] T ed78d40318fe4ab5bee4a01b182a34b6 P 9be399440ded4e2ebe6b6e0bf486cbb1: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:43.834107 18857 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32c9fc5a18c54207881124fda86f42ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:43.839345 19406 tablet_bootstrap.cc:492] T 84b235fd89634168938c0dddb534462a P faadabfb099b4abf90a466395be585b5: Bootstrap starting.
I20240502 21:25:43.840518 19405 tablet_bootstrap.cc:492] T ed78d40318fe4ab5bee4a01b182a34b6 P 9be399440ded4e2ebe6b6e0bf486cbb1: No bootstrap required, opened a new log
I20240502 21:25:43.840979 19406 tablet_bootstrap.cc:654] T 84b235fd89634168938c0dddb534462a P faadabfb099b4abf90a466395be585b5: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:43.841399 19405 ts_tablet_manager.cc:1367] T ed78d40318fe4ab5bee4a01b182a34b6 P 9be399440ded4e2ebe6b6e0bf486cbb1: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.002s
I20240502 21:25:43.842484 19404 tablet_bootstrap.cc:492] T 32c9fc5a18c54207881124fda86f42ff P 501e3c0d44fe4d70af9ff926d9622417: Bootstrap starting.
I20240502 21:25:43.842888 19405 raft_consensus.cc:348] T ed78d40318fe4ab5bee4a01b182a34b6 P 9be399440ded4e2ebe6b6e0bf486cbb1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } }
I20240502 21:25:43.843199 19405 raft_consensus.cc:374] T ed78d40318fe4ab5bee4a01b182a34b6 P 9be399440ded4e2ebe6b6e0bf486cbb1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:43.843334 19405 raft_consensus.cc:729] T ed78d40318fe4ab5bee4a01b182a34b6 P 9be399440ded4e2ebe6b6e0bf486cbb1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9be399440ded4e2ebe6b6e0bf486cbb1, State: Initialized, Role: FOLLOWER
I20240502 21:25:43.843677 19405 consensus_queue.cc:260] T ed78d40318fe4ab5bee4a01b182a34b6 P 9be399440ded4e2ebe6b6e0bf486cbb1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } }
I20240502 21:25:43.844076 19404 tablet_bootstrap.cc:654] T 32c9fc5a18c54207881124fda86f42ff P 501e3c0d44fe4d70af9ff926d9622417: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:43.851776 19407 tablet_bootstrap.cc:492] T 32c9fc5a18c54207881124fda86f42ff P 82f50f4b66f3450aa9f58a0b317f5cea: Bootstrap starting.
I20240502 21:25:43.852300 19125 tablet_service.cc:1467] Processing CreateTablet for tablet 32c9fc5a18c54207881124fda86f42ff (DEFAULT_TABLE table=rebalance_test_table_0 [id=086ede98fc604333833bce759f342073]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:43.853372 19407 tablet_bootstrap.cc:654] T 32c9fc5a18c54207881124fda86f42ff P 82f50f4b66f3450aa9f58a0b317f5cea: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:43.853794 19125 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32c9fc5a18c54207881124fda86f42ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:43.855763 19413 tablet_bootstrap.cc:492] T 84b235fd89634168938c0dddb534462a P c75831edb3b44dd59c00ef151adca49d: Bootstrap starting.
I20240502 21:25:43.856623 19124 tablet_service.cc:1467] Processing CreateTablet for tablet 84b235fd89634168938c0dddb534462a (DEFAULT_TABLE table=rebalance_test_table_0 [id=086ede98fc604333833bce759f342073]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:43.857111 19124 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84b235fd89634168938c0dddb534462a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:43.857739 19123 tablet_service.cc:1467] Processing CreateTablet for tablet ed78d40318fe4ab5bee4a01b182a34b6 (DEFAULT_TABLE table=rebalance_test_table_0 [id=086ede98fc604333833bce759f342073]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:43.858201 19123 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed78d40318fe4ab5bee4a01b182a34b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:43.859009 19413 tablet_bootstrap.cc:654] T 84b235fd89634168938c0dddb534462a P c75831edb3b44dd59c00ef151adca49d: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:43.860314 19413 log.cc:824] T 84b235fd89634168938c0dddb534462a P c75831edb3b44dd59c00ef151adca49d: Log is configured to *not* fsync() on all Append() calls
I20240502 21:25:43.861694 19404 tablet_bootstrap.cc:492] T 32c9fc5a18c54207881124fda86f42ff P 501e3c0d44fe4d70af9ff926d9622417: No bootstrap required, opened a new log
I20240502 21:25:43.861847 19404 ts_tablet_manager.cc:1367] T 32c9fc5a18c54207881124fda86f42ff P 501e3c0d44fe4d70af9ff926d9622417: Time spent bootstrapping tablet: real 0.019s user 0.007s sys 0.004s
I20240502 21:25:43.862126 19407 tablet_bootstrap.cc:492] T 32c9fc5a18c54207881124fda86f42ff P 82f50f4b66f3450aa9f58a0b317f5cea: No bootstrap required, opened a new log
I20240502 21:25:43.862310 19407 ts_tablet_manager.cc:1367] T 32c9fc5a18c54207881124fda86f42ff P 82f50f4b66f3450aa9f58a0b317f5cea: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.000s
I20240502 21:25:43.863525 19404 raft_consensus.cc:348] T 32c9fc5a18c54207881124fda86f42ff P 501e3c0d44fe4d70af9ff926d9622417 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } }
I20240502 21:25:43.863750 19404 raft_consensus.cc:374] T 32c9fc5a18c54207881124fda86f42ff P 501e3c0d44fe4d70af9ff926d9622417 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:43.863842 19404 raft_consensus.cc:729] T 32c9fc5a18c54207881124fda86f42ff P 501e3c0d44fe4d70af9ff926d9622417 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 501e3c0d44fe4d70af9ff926d9622417, State: Initialized, Role: FOLLOWER
I20240502 21:25:43.864931 19406 tablet_bootstrap.cc:492] T 84b235fd89634168938c0dddb534462a P faadabfb099b4abf90a466395be585b5: No bootstrap required, opened a new log
I20240502 21:25:43.865094 19406 ts_tablet_manager.cc:1367] T 84b235fd89634168938c0dddb534462a P faadabfb099b4abf90a466395be585b5: Time spent bootstrapping tablet: real 0.026s user 0.007s sys 0.004s
I20240502 21:25:43.865900 19407 raft_consensus.cc:348] T 32c9fc5a18c54207881124fda86f42ff P 82f50f4b66f3450aa9f58a0b317f5cea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } }
I20240502 21:25:43.865990 19406 raft_consensus.cc:348] T 84b235fd89634168938c0dddb534462a P faadabfb099b4abf90a466395be585b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } }
I20240502 21:25:43.866215 19406 raft_consensus.cc:374] T 84b235fd89634168938c0dddb534462a P faadabfb099b4abf90a466395be585b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:43.866307 19407 raft_consensus.cc:374] T 32c9fc5a18c54207881124fda86f42ff P 82f50f4b66f3450aa9f58a0b317f5cea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:43.866308 19406 raft_consensus.cc:729] T 84b235fd89634168938c0dddb534462a P faadabfb099b4abf90a466395be585b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: faadabfb099b4abf90a466395be585b5, State: Initialized, Role: FOLLOWER
I20240502 21:25:43.866446 19407 raft_consensus.cc:729] T 32c9fc5a18c54207881124fda86f42ff P 82f50f4b66f3450aa9f58a0b317f5cea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82f50f4b66f3450aa9f58a0b317f5cea, State: Initialized, Role: FOLLOWER
I20240502 21:25:43.866569 19406 consensus_queue.cc:260] T 84b235fd89634168938c0dddb534462a P faadabfb099b4abf90a466395be585b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } }
I20240502 21:25:43.867246 19406 ts_tablet_manager.cc:1398] T 84b235fd89634168938c0dddb534462a P faadabfb099b4abf90a466395be585b5: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240502 21:25:43.867527 19406 tablet_bootstrap.cc:492] T 32c9fc5a18c54207881124fda86f42ff P faadabfb099b4abf90a466395be585b5: Bootstrap starting.
I20240502 21:25:43.869081 19406 tablet_bootstrap.cc:654] T 32c9fc5a18c54207881124fda86f42ff P faadabfb099b4abf90a466395be585b5: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:43.870057 19404 consensus_queue.cc:260] T 32c9fc5a18c54207881124fda86f42ff P 501e3c0d44fe4d70af9ff926d9622417 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } }
I20240502 21:25:43.871060 19411 raft_consensus.cc:482] T ed78d40318fe4ab5bee4a01b182a34b6 P 9be399440ded4e2ebe6b6e0bf486cbb1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:25:43.871217 19411 raft_consensus.cc:504] T ed78d40318fe4ab5bee4a01b182a34b6 P 9be399440ded4e2ebe6b6e0bf486cbb1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } }
I20240502 21:25:43.866730 19407 consensus_queue.cc:260] T 32c9fc5a18c54207881124fda86f42ff P 82f50f4b66f3450aa9f58a0b317f5cea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } }
I20240502 21:25:43.872506 19405 ts_tablet_manager.cc:1398] T ed78d40318fe4ab5bee4a01b182a34b6 P 9be399440ded4e2ebe6b6e0bf486cbb1: Time spent starting tablet: real 0.031s user 0.012s sys 0.004s
I20240502 21:25:43.875859 19411 leader_election.cc:288] T ed78d40318fe4ab5bee4a01b182a34b6 P 9be399440ded4e2ebe6b6e0bf486cbb1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c75831edb3b44dd59c00ef151adca49d (127.15.255.132:43607), 501e3c0d44fe4d70af9ff926d9622417 (127.15.255.131:42501), 2877108756304e05bd60ba1186b870da (127.15.255.134:43635), faadabfb099b4abf90a466395be585b5 (127.15.255.129:32983)
I20240502 21:25:43.884804 19405 tablet_bootstrap.cc:492] T 84b235fd89634168938c0dddb534462a P 9be399440ded4e2ebe6b6e0bf486cbb1: Bootstrap starting.
I20240502 21:25:43.887027 19405 tablet_bootstrap.cc:654] T 84b235fd89634168938c0dddb534462a P 9be399440ded4e2ebe6b6e0bf486cbb1: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:43.900080 19407 ts_tablet_manager.cc:1398] T 32c9fc5a18c54207881124fda86f42ff P 82f50f4b66f3450aa9f58a0b317f5cea: Time spent starting tablet: real 0.038s user 0.005s sys 0.000s
I20240502 21:25:43.900445 19407 tablet_bootstrap.cc:492] T 84b235fd89634168938c0dddb534462a P 82f50f4b66f3450aa9f58a0b317f5cea: Bootstrap starting.
I20240502 21:25:43.902033 19407 tablet_bootstrap.cc:654] T 84b235fd89634168938c0dddb534462a P 82f50f4b66f3450aa9f58a0b317f5cea: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:43.906795 19413 tablet_bootstrap.cc:492] T 84b235fd89634168938c0dddb534462a P c75831edb3b44dd59c00ef151adca49d: No bootstrap required, opened a new log
I20240502 21:25:43.908020 19413 ts_tablet_manager.cc:1367] T 84b235fd89634168938c0dddb534462a P c75831edb3b44dd59c00ef151adca49d: Time spent bootstrapping tablet: real 0.052s user 0.026s sys 0.016s
I20240502 21:25:43.910084 19406 tablet_bootstrap.cc:492] T 32c9fc5a18c54207881124fda86f42ff P faadabfb099b4abf90a466395be585b5: No bootstrap required, opened a new log
I20240502 21:25:43.910321 19406 ts_tablet_manager.cc:1367] T 32c9fc5a18c54207881124fda86f42ff P faadabfb099b4abf90a466395be585b5: Time spent bootstrapping tablet: real 0.043s user 0.009s sys 0.004s
I20240502 21:25:43.911126 19406 raft_consensus.cc:348] T 32c9fc5a18c54207881124fda86f42ff P faadabfb099b4abf90a466395be585b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } }
I20240502 21:25:43.911478 19406 raft_consensus.cc:374] T 32c9fc5a18c54207881124fda86f42ff P faadabfb099b4abf90a466395be585b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:43.911609 19406 raft_consensus.cc:729] T 32c9fc5a18c54207881124fda86f42ff P faadabfb099b4abf90a466395be585b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: faadabfb099b4abf90a466395be585b5, State: Initialized, Role: FOLLOWER
I20240502 21:25:43.911939 19406 consensus_queue.cc:260] T 32c9fc5a18c54207881124fda86f42ff P faadabfb099b4abf90a466395be585b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } }
I20240502 21:25:43.912154 19427 tablet_bootstrap.cc:492] T ed78d40318fe4ab5bee4a01b182a34b6 P 2877108756304e05bd60ba1186b870da: Bootstrap starting.
I20240502 21:25:43.912737 19406 ts_tablet_manager.cc:1398] T 32c9fc5a18c54207881124fda86f42ff P faadabfb099b4abf90a466395be585b5: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 21:25:43.913054 19406 tablet_bootstrap.cc:492] T ed78d40318fe4ab5bee4a01b182a34b6 P faadabfb099b4abf90a466395be585b5: Bootstrap starting.
I20240502 21:25:43.914222 19404 ts_tablet_manager.cc:1398] T 32c9fc5a18c54207881124fda86f42ff P 501e3c0d44fe4d70af9ff926d9622417: Time spent starting tablet: real 0.052s user 0.002s sys 0.011s
I20240502 21:25:43.915283 19404 tablet_bootstrap.cc:492] T ed78d40318fe4ab5bee4a01b182a34b6 P 501e3c0d44fe4d70af9ff926d9622417: Bootstrap starting.
I20240502 21:25:43.917003 19404 tablet_bootstrap.cc:654] T ed78d40318fe4ab5bee4a01b182a34b6 P 501e3c0d44fe4d70af9ff926d9622417: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:43.918000 18744 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed78d40318fe4ab5bee4a01b182a34b6" candidate_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "501e3c0d44fe4d70af9ff926d9622417" is_pre_election: true
W20240502 21:25:43.918931 18943 leader_election.cc:341] T ed78d40318fe4ab5bee4a01b182a34b6 P 9be399440ded4e2ebe6b6e0bf486cbb1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 501e3c0d44fe4d70af9ff926d9622417 (127.15.255.131:42501): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:25:43.919466 19406 tablet_bootstrap.cc:654] T ed78d40318fe4ab5bee4a01b182a34b6 P faadabfb099b4abf90a466395be585b5: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:43.921669 19427 tablet_bootstrap.cc:654] T ed78d40318fe4ab5bee4a01b182a34b6 P 2877108756304e05bd60ba1186b870da: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:43.923051 19427 log.cc:824] T ed78d40318fe4ab5bee4a01b182a34b6 P 2877108756304e05bd60ba1186b870da: Log is configured to *not* fsync() on all Append() calls
I20240502 21:25:43.924294 18878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed78d40318fe4ab5bee4a01b182a34b6" candidate_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c75831edb3b44dd59c00ef151adca49d" is_pre_election: true
W20240502 21:25:43.925843 18945 leader_election.cc:341] T ed78d40318fe4ab5bee4a01b182a34b6 P 9be399440ded4e2ebe6b6e0bf486cbb1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c75831edb3b44dd59c00ef151adca49d (127.15.255.132:43607): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:25:43.927398 18474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed78d40318fe4ab5bee4a01b182a34b6" candidate_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "faadabfb099b4abf90a466395be585b5" is_pre_election: true
I20240502 21:25:43.927901 19404 tablet_bootstrap.cc:492] T ed78d40318fe4ab5bee4a01b182a34b6 P 501e3c0d44fe4d70af9ff926d9622417: No bootstrap required, opened a new log
I20240502 21:25:43.928112 19404 ts_tablet_manager.cc:1367] T ed78d40318fe4ab5bee4a01b182a34b6 P 501e3c0d44fe4d70af9ff926d9622417: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.000s
W20240502 21:25:43.928412 18944 leader_election.cc:341] T ed78d40318fe4ab5bee4a01b182a34b6 P 9be399440ded4e2ebe6b6e0bf486cbb1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer faadabfb099b4abf90a466395be585b5 (127.15.255.129:32983): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:25:43.928591 18944 leader_election.cc:302] T ed78d40318fe4ab5bee4a01b182a34b6 P 9be399440ded4e2ebe6b6e0bf486cbb1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 5 voters: 1 yes votes; 3 no votes. yes voters: 9be399440ded4e2ebe6b6e0bf486cbb1; no voters: 501e3c0d44fe4d70af9ff926d9622417, c75831edb3b44dd59c00ef151adca49d, faadabfb099b4abf90a466395be585b5
I20240502 21:25:43.929052 19411 raft_consensus.cc:2726] T ed78d40318fe4ab5bee4a01b182a34b6 P 9be399440ded4e2ebe6b6e0bf486cbb1 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 21:25:43.928972 19404 raft_consensus.cc:348] T ed78d40318fe4ab5bee4a01b182a34b6 P 501e3c0d44fe4d70af9ff926d9622417 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } }
I20240502 21:25:43.929237 19404 raft_consensus.cc:374] T ed78d40318fe4ab5bee4a01b182a34b6 P 501e3c0d44fe4d70af9ff926d9622417 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:43.929333 19404 raft_consensus.cc:729] T ed78d40318fe4ab5bee4a01b182a34b6 P 501e3c0d44fe4d70af9ff926d9622417 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 501e3c0d44fe4d70af9ff926d9622417, State: Initialized, Role: FOLLOWER
I20240502 21:25:43.929646 19405 tablet_bootstrap.cc:492] T 84b235fd89634168938c0dddb534462a P 9be399440ded4e2ebe6b6e0bf486cbb1: No bootstrap required, opened a new log
I20240502 21:25:43.929620 19404 consensus_queue.cc:260] T ed78d40318fe4ab5bee4a01b182a34b6 P 501e3c0d44fe4d70af9ff926d9622417 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } }
I20240502 21:25:43.929849 19405 ts_tablet_manager.cc:1367] T 84b235fd89634168938c0dddb534462a P 9be399440ded4e2ebe6b6e0bf486cbb1: Time spent bootstrapping tablet: real 0.045s user 0.003s sys 0.006s
I20240502 21:25:43.930435 19404 ts_tablet_manager.cc:1398] T ed78d40318fe4ab5bee4a01b182a34b6 P 501e3c0d44fe4d70af9ff926d9622417: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:25:43.930179 19413 raft_consensus.cc:348] T 84b235fd89634168938c0dddb534462a P c75831edb3b44dd59c00ef151adca49d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } }
I20240502 21:25:43.930593 19413 raft_consensus.cc:374] T 84b235fd89634168938c0dddb534462a P c75831edb3b44dd59c00ef151adca49d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:43.930711 19413 raft_consensus.cc:729] T 84b235fd89634168938c0dddb534462a P c75831edb3b44dd59c00ef151adca49d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c75831edb3b44dd59c00ef151adca49d, State: Initialized, Role: FOLLOWER
I20240502 21:25:43.930655 19405 raft_consensus.cc:348] T 84b235fd89634168938c0dddb534462a P 9be399440ded4e2ebe6b6e0bf486cbb1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } }
I20240502 21:25:43.930984 19405 raft_consensus.cc:374] T 84b235fd89634168938c0dddb534462a P 9be399440ded4e2ebe6b6e0bf486cbb1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:43.931078 19405 raft_consensus.cc:729] T 84b235fd89634168938c0dddb534462a P 9be399440ded4e2ebe6b6e0bf486cbb1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9be399440ded4e2ebe6b6e0bf486cbb1, State: Initialized, Role: FOLLOWER
I20240502 21:25:43.931360 19405 consensus_queue.cc:260] T 84b235fd89634168938c0dddb534462a P 9be399440ded4e2ebe6b6e0bf486cbb1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } }
I20240502 21:25:43.931865 19406 tablet_bootstrap.cc:492] T ed78d40318fe4ab5bee4a01b182a34b6 P faadabfb099b4abf90a466395be585b5: No bootstrap required, opened a new log
I20240502 21:25:43.931980 19405 ts_tablet_manager.cc:1398] T 84b235fd89634168938c0dddb534462a P 9be399440ded4e2ebe6b6e0bf486cbb1: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 21:25:43.932104 19406 ts_tablet_manager.cc:1367] T ed78d40318fe4ab5bee4a01b182a34b6 P faadabfb099b4abf90a466395be585b5: Time spent bootstrapping tablet: real 0.019s user 0.003s sys 0.004s
I20240502 21:25:43.932826 19406 raft_consensus.cc:348] T ed78d40318fe4ab5bee4a01b182a34b6 P faadabfb099b4abf90a466395be585b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } }
I20240502 21:25:43.933137 19406 raft_consensus.cc:374] T ed78d40318fe4ab5bee4a01b182a34b6 P faadabfb099b4abf90a466395be585b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:43.933349 19407 tablet_bootstrap.cc:492] T 84b235fd89634168938c0dddb534462a P 82f50f4b66f3450aa9f58a0b317f5cea: No bootstrap required, opened a new log
I20240502 21:25:43.933511 19407 ts_tablet_manager.cc:1367] T 84b235fd89634168938c0dddb534462a P 82f50f4b66f3450aa9f58a0b317f5cea: Time spent bootstrapping tablet: real 0.033s user 0.000s sys 0.010s
I20240502 21:25:43.933274 19406 raft_consensus.cc:729] T ed78d40318fe4ab5bee4a01b182a34b6 P faadabfb099b4abf90a466395be585b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: faadabfb099b4abf90a466395be585b5, State: Initialized, Role: FOLLOWER
I20240502 21:25:43.933945 19406 consensus_queue.cc:260] T ed78d40318fe4ab5bee4a01b182a34b6 P faadabfb099b4abf90a466395be585b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } }
I20240502 21:25:43.934248 19407 raft_consensus.cc:348] T 84b235fd89634168938c0dddb534462a P 82f50f4b66f3450aa9f58a0b317f5cea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } }
I20240502 21:25:43.934510 19407 raft_consensus.cc:374] T 84b235fd89634168938c0dddb534462a P 82f50f4b66f3450aa9f58a0b317f5cea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:43.934583 19407 raft_consensus.cc:729] T 84b235fd89634168938c0dddb534462a P 82f50f4b66f3450aa9f58a0b317f5cea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82f50f4b66f3450aa9f58a0b317f5cea, State: Initialized, Role: FOLLOWER
I20240502 21:25:43.934897 19407 consensus_queue.cc:260] T 84b235fd89634168938c0dddb534462a P 82f50f4b66f3450aa9f58a0b317f5cea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } }
I20240502 21:25:43.935551 19407 ts_tablet_manager.cc:1398] T 84b235fd89634168938c0dddb534462a P 82f50f4b66f3450aa9f58a0b317f5cea: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 21:25:43.936028 19406 ts_tablet_manager.cc:1398] T ed78d40318fe4ab5bee4a01b182a34b6 P faadabfb099b4abf90a466395be585b5: Time spent starting tablet: real 0.004s user 0.000s sys 0.001s
I20240502 21:25:43.936817 19413 consensus_queue.cc:260] T 84b235fd89634168938c0dddb534462a P c75831edb3b44dd59c00ef151adca49d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } }
I20240502 21:25:43.938999 19427 tablet_bootstrap.cc:492] T ed78d40318fe4ab5bee4a01b182a34b6 P 2877108756304e05bd60ba1186b870da: No bootstrap required, opened a new log
I20240502 21:25:43.935887 19146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed78d40318fe4ab5bee4a01b182a34b6" candidate_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2877108756304e05bd60ba1186b870da" is_pre_election: true
I20240502 21:25:43.939327 19427 ts_tablet_manager.cc:1367] T ed78d40318fe4ab5bee4a01b182a34b6 P 2877108756304e05bd60ba1186b870da: Time spent bootstrapping tablet: real 0.028s user 0.010s sys 0.000s
I20240502 21:25:43.941179 19413 ts_tablet_manager.cc:1398] T 84b235fd89634168938c0dddb534462a P c75831edb3b44dd59c00ef151adca49d: Time spent starting tablet: real 0.032s user 0.010s sys 0.004s
I20240502 21:25:43.941610 19413 tablet_bootstrap.cc:492] T ed78d40318fe4ab5bee4a01b182a34b6 P c75831edb3b44dd59c00ef151adca49d: Bootstrap starting.
W20240502 21:25:43.942917 18946 leader_election.cc:341] T ed78d40318fe4ab5bee4a01b182a34b6 P 9be399440ded4e2ebe6b6e0bf486cbb1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2877108756304e05bd60ba1186b870da (127.15.255.134:43635): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:25:43.943260 19413 tablet_bootstrap.cc:654] T ed78d40318fe4ab5bee4a01b182a34b6 P c75831edb3b44dd59c00ef151adca49d: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:43.944011 19388 raft_consensus.cc:482] T 32c9fc5a18c54207881124fda86f42ff P faadabfb099b4abf90a466395be585b5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:25:43.944162 19388 raft_consensus.cc:504] T 32c9fc5a18c54207881124fda86f42ff P faadabfb099b4abf90a466395be585b5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } }
I20240502 21:25:43.945277 19388 leader_election.cc:288] T 32c9fc5a18c54207881124fda86f42ff P faadabfb099b4abf90a466395be585b5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c75831edb3b44dd59c00ef151adca49d (127.15.255.132:43607), 82f50f4b66f3450aa9f58a0b317f5cea (127.15.255.130:45139), 2877108756304e05bd60ba1186b870da (127.15.255.134:43635), 501e3c0d44fe4d70af9ff926d9622417 (127.15.255.131:42501)
I20240502 21:25:43.945899 18744 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32c9fc5a18c54207881124fda86f42ff" candidate_uuid: "faadabfb099b4abf90a466395be585b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "501e3c0d44fe4d70af9ff926d9622417" is_pre_election: true
I20240502 21:25:43.945771 19427 raft_consensus.cc:348] T ed78d40318fe4ab5bee4a01b182a34b6 P 2877108756304e05bd60ba1186b870da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } }
I20240502 21:25:43.946094 18744 raft_consensus.cc:2454] T 32c9fc5a18c54207881124fda86f42ff P 501e3c0d44fe4d70af9ff926d9622417 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate faadabfb099b4abf90a466395be585b5 in term 0.
I20240502 21:25:43.946161 19427 raft_consensus.cc:374] T ed78d40318fe4ab5bee4a01b182a34b6 P 2877108756304e05bd60ba1186b870da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:43.946266 19427 raft_consensus.cc:729] T ed78d40318fe4ab5bee4a01b182a34b6 P 2877108756304e05bd60ba1186b870da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2877108756304e05bd60ba1186b870da, State: Initialized, Role: FOLLOWER
I20240502 21:25:43.946955 19427 consensus_queue.cc:260] T ed78d40318fe4ab5bee4a01b182a34b6 P 2877108756304e05bd60ba1186b870da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } }
I20240502 21:25:43.947194 18611 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32c9fc5a18c54207881124fda86f42ff" candidate_uuid: "faadabfb099b4abf90a466395be585b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" is_pre_election: true
I20240502 21:25:43.947429 18611 raft_consensus.cc:2454] T 32c9fc5a18c54207881124fda86f42ff P 82f50f4b66f3450aa9f58a0b317f5cea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate faadabfb099b4abf90a466395be585b5 in term 0.
I20240502 21:25:43.950834 19427 ts_tablet_manager.cc:1398] T ed78d40318fe4ab5bee4a01b182a34b6 P 2877108756304e05bd60ba1186b870da: Time spent starting tablet: real 0.011s user 0.007s sys 0.002s
I20240502 21:25:43.951258 19427 tablet_bootstrap.cc:492] T 84b235fd89634168938c0dddb534462a P 2877108756304e05bd60ba1186b870da: Bootstrap starting.
I20240502 21:25:43.951220 18407 leader_election.cc:302] T 32c9fc5a18c54207881124fda86f42ff P faadabfb099b4abf90a466395be585b5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 501e3c0d44fe4d70af9ff926d9622417, 82f50f4b66f3450aa9f58a0b317f5cea, faadabfb099b4abf90a466395be585b5; no voters:
I20240502 21:25:43.951530 19388 raft_consensus.cc:2781] T 32c9fc5a18c54207881124fda86f42ff P faadabfb099b4abf90a466395be585b5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:25:43.951648 19388 raft_consensus.cc:482] T 32c9fc5a18c54207881124fda86f42ff P faadabfb099b4abf90a466395be585b5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:25:43.951736 19388 raft_consensus.cc:3037] T 32c9fc5a18c54207881124fda86f42ff P faadabfb099b4abf90a466395be585b5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:43.953058 19427 tablet_bootstrap.cc:654] T 84b235fd89634168938c0dddb534462a P 2877108756304e05bd60ba1186b870da: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:43.954788 19388 raft_consensus.cc:504] T 32c9fc5a18c54207881124fda86f42ff P faadabfb099b4abf90a466395be585b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } }
I20240502 21:25:43.960201 18878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32c9fc5a18c54207881124fda86f42ff" candidate_uuid: "faadabfb099b4abf90a466395be585b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c75831edb3b44dd59c00ef151adca49d" is_pre_election: true
I20240502 21:25:43.960321 18877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32c9fc5a18c54207881124fda86f42ff" candidate_uuid: "faadabfb099b4abf90a466395be585b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c75831edb3b44dd59c00ef151adca49d"
W20240502 21:25:43.961202 18406 leader_election.cc:341] T 32c9fc5a18c54207881124fda86f42ff P faadabfb099b4abf90a466395be585b5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c75831edb3b44dd59c00ef151adca49d (127.15.255.132:43607): Illegal state: must be running to vote when last-logged opid is not known
W20240502 21:25:43.961479 18406 leader_election.cc:341] T 32c9fc5a18c54207881124fda86f42ff P faadabfb099b4abf90a466395be585b5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c75831edb3b44dd59c00ef151adca49d (127.15.255.132:43607): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:25:43.963436 18611 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32c9fc5a18c54207881124fda86f42ff" candidate_uuid: "faadabfb099b4abf90a466395be585b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82f50f4b66f3450aa9f58a0b317f5cea"
I20240502 21:25:43.963521 18744 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32c9fc5a18c54207881124fda86f42ff" candidate_uuid: "faadabfb099b4abf90a466395be585b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "501e3c0d44fe4d70af9ff926d9622417"
I20240502 21:25:43.963663 18611 raft_consensus.cc:3037] T 32c9fc5a18c54207881124fda86f42ff P 82f50f4b66f3450aa9f58a0b317f5cea [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:43.963747 18744 raft_consensus.cc:3037] T 32c9fc5a18c54207881124fda86f42ff P 501e3c0d44fe4d70af9ff926d9622417 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:43.966516 18611 raft_consensus.cc:2454] T 32c9fc5a18c54207881124fda86f42ff P 82f50f4b66f3450aa9f58a0b317f5cea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate faadabfb099b4abf90a466395be585b5 in term 1.
I20240502 21:25:43.966800 18744 raft_consensus.cc:2454] T 32c9fc5a18c54207881124fda86f42ff P 501e3c0d44fe4d70af9ff926d9622417 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate faadabfb099b4abf90a466395be585b5 in term 1.
I20240502 21:25:43.967319 18404 leader_election.cc:302] T 32c9fc5a18c54207881124fda86f42ff P faadabfb099b4abf90a466395be585b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 5 voters: 3 yes votes; 1 no votes. yes voters: 501e3c0d44fe4d70af9ff926d9622417, 82f50f4b66f3450aa9f58a0b317f5cea, faadabfb099b4abf90a466395be585b5; no voters: c75831edb3b44dd59c00ef151adca49d
I20240502 21:25:43.968485 19388 leader_election.cc:288] T 32c9fc5a18c54207881124fda86f42ff P faadabfb099b4abf90a466395be585b5 [CANDIDATE]: Term 1 election: Requested vote from peers c75831edb3b44dd59c00ef151adca49d (127.15.255.132:43607), 82f50f4b66f3450aa9f58a0b317f5cea (127.15.255.130:45139), 2877108756304e05bd60ba1186b870da (127.15.255.134:43635), 501e3c0d44fe4d70af9ff926d9622417 (127.15.255.131:42501)
I20240502 21:25:43.968703 19388 raft_consensus.cc:2781] T 32c9fc5a18c54207881124fda86f42ff P faadabfb099b4abf90a466395be585b5 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:25:43.968827 19388 raft_consensus.cc:686] T 32c9fc5a18c54207881124fda86f42ff P faadabfb099b4abf90a466395be585b5 [term 1 LEADER]: Becoming Leader. State: Replica: faadabfb099b4abf90a466395be585b5, State: Running, Role: LEADER
I20240502 21:25:43.969156 19388 consensus_queue.cc:237] T 32c9fc5a18c54207881124fda86f42ff P faadabfb099b4abf90a466395be585b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } }
I20240502 21:25:43.969487 19146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32c9fc5a18c54207881124fda86f42ff" candidate_uuid: "faadabfb099b4abf90a466395be585b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2877108756304e05bd60ba1186b870da" is_pre_election: true
I20240502 21:25:43.969954 19145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32c9fc5a18c54207881124fda86f42ff" candidate_uuid: "faadabfb099b4abf90a466395be585b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2877108756304e05bd60ba1186b870da"
W20240502 21:25:43.970883 18407 leader_election.cc:341] T 32c9fc5a18c54207881124fda86f42ff P faadabfb099b4abf90a466395be585b5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2877108756304e05bd60ba1186b870da (127.15.255.134:43635): Illegal state: must be running to vote when last-logged opid is not known
W20240502 21:25:43.971369 18407 leader_election.cc:341] T 32c9fc5a18c54207881124fda86f42ff P faadabfb099b4abf90a466395be585b5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 2877108756304e05bd60ba1186b870da (127.15.255.134:43635): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:25:43.973860 18329 catalog_manager.cc:5484] T 32c9fc5a18c54207881124fda86f42ff P faadabfb099b4abf90a466395be585b5 reported cstate change: term changed from 0 to 1, leader changed from <none> to faadabfb099b4abf90a466395be585b5 (127.15.255.129). New cstate: current_term: 1 leader_uuid: "faadabfb099b4abf90a466395be585b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } health_report { overall_health: UNKNOWN } } }
I20240502 21:25:43.974243 19413 tablet_bootstrap.cc:492] T ed78d40318fe4ab5bee4a01b182a34b6 P c75831edb3b44dd59c00ef151adca49d: No bootstrap required, opened a new log
I20240502 21:25:43.974292 19427 tablet_bootstrap.cc:492] T 84b235fd89634168938c0dddb534462a P 2877108756304e05bd60ba1186b870da: No bootstrap required, opened a new log
I20240502 21:25:43.974424 19427 ts_tablet_manager.cc:1367] T 84b235fd89634168938c0dddb534462a P 2877108756304e05bd60ba1186b870da: Time spent bootstrapping tablet: real 0.023s user 0.008s sys 0.013s
I20240502 21:25:43.974417 19413 ts_tablet_manager.cc:1367] T ed78d40318fe4ab5bee4a01b182a34b6 P c75831edb3b44dd59c00ef151adca49d: Time spent bootstrapping tablet: real 0.033s user 0.016s sys 0.015s
I20240502 21:25:43.975462 19388 raft_consensus.cc:482] T ed78d40318fe4ab5bee4a01b182a34b6 P faadabfb099b4abf90a466395be585b5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:25:43.975360 19413 raft_consensus.cc:348] T ed78d40318fe4ab5bee4a01b182a34b6 P c75831edb3b44dd59c00ef151adca49d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } }
I20240502 21:25:43.975469 19427 raft_consensus.cc:348] T 84b235fd89634168938c0dddb534462a P 2877108756304e05bd60ba1186b870da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } }
I20240502 21:25:43.975648 19413 raft_consensus.cc:374] T ed78d40318fe4ab5bee4a01b182a34b6 P c75831edb3b44dd59c00ef151adca49d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:43.975701 19427 raft_consensus.cc:374] T 84b235fd89634168938c0dddb534462a P 2877108756304e05bd60ba1186b870da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:43.975739 19413 raft_consensus.cc:729] T ed78d40318fe4ab5bee4a01b182a34b6 P c75831edb3b44dd59c00ef151adca49d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c75831edb3b44dd59c00ef151adca49d, State: Initialized, Role: FOLLOWER
I20240502 21:25:43.975616 19388 raft_consensus.cc:504] T ed78d40318fe4ab5bee4a01b182a34b6 P faadabfb099b4abf90a466395be585b5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } }
I20240502 21:25:43.975788 19427 raft_consensus.cc:729] T 84b235fd89634168938c0dddb534462a P 2877108756304e05bd60ba1186b870da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2877108756304e05bd60ba1186b870da, State: Initialized, Role: FOLLOWER
I20240502 21:25:43.975981 19413 consensus_queue.cc:260] T ed78d40318fe4ab5bee4a01b182a34b6 P c75831edb3b44dd59c00ef151adca49d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } }
I20240502 21:25:43.976140 19427 consensus_queue.cc:260] T 84b235fd89634168938c0dddb534462a P 2877108756304e05bd60ba1186b870da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } }
I20240502 21:25:43.976554 19388 leader_election.cc:288] T ed78d40318fe4ab5bee4a01b182a34b6 P faadabfb099b4abf90a466395be585b5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c75831edb3b44dd59c00ef151adca49d (127.15.255.132:43607), 501e3c0d44fe4d70af9ff926d9622417 (127.15.255.131:42501), 2877108756304e05bd60ba1186b870da (127.15.255.134:43635), 9be399440ded4e2ebe6b6e0bf486cbb1 (127.15.255.133:34217)
I20240502 21:25:43.976796 19413 ts_tablet_manager.cc:1398] T ed78d40318fe4ab5bee4a01b182a34b6 P c75831edb3b44dd59c00ef151adca49d: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20240502 21:25:43.977090 19413 tablet_bootstrap.cc:492] T 32c9fc5a18c54207881124fda86f42ff P c75831edb3b44dd59c00ef151adca49d: Bootstrap starting.
I20240502 21:25:43.977113 19010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed78d40318fe4ab5bee4a01b182a34b6" candidate_uuid: "faadabfb099b4abf90a466395be585b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" is_pre_election: true
I20240502 21:25:43.977257 19427 ts_tablet_manager.cc:1398] T 84b235fd89634168938c0dddb534462a P 2877108756304e05bd60ba1186b870da: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240502 21:25:43.977344 19010 raft_consensus.cc:2454] T ed78d40318fe4ab5bee4a01b182a34b6 P 9be399440ded4e2ebe6b6e0bf486cbb1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate faadabfb099b4abf90a466395be585b5 in term 0.
I20240502 21:25:43.977531 19427 tablet_bootstrap.cc:492] T 32c9fc5a18c54207881124fda86f42ff P 2877108756304e05bd60ba1186b870da: Bootstrap starting.
I20240502 21:25:43.978376 19145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed78d40318fe4ab5bee4a01b182a34b6" candidate_uuid: "faadabfb099b4abf90a466395be585b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2877108756304e05bd60ba1186b870da" is_pre_election: true
I20240502 21:25:43.978457 18744 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed78d40318fe4ab5bee4a01b182a34b6" candidate_uuid: "faadabfb099b4abf90a466395be585b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "501e3c0d44fe4d70af9ff926d9622417" is_pre_election: true
I20240502 21:25:43.978655 18744 raft_consensus.cc:2454] T ed78d40318fe4ab5bee4a01b182a34b6 P 501e3c0d44fe4d70af9ff926d9622417 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate faadabfb099b4abf90a466395be585b5 in term 0.
I20240502 21:25:43.978703 19145 raft_consensus.cc:2454] T ed78d40318fe4ab5bee4a01b182a34b6 P 2877108756304e05bd60ba1186b870da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate faadabfb099b4abf90a466395be585b5 in term 0.
I20240502 21:25:43.978992 18877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed78d40318fe4ab5bee4a01b182a34b6" candidate_uuid: "faadabfb099b4abf90a466395be585b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c75831edb3b44dd59c00ef151adca49d" is_pre_election: true
I20240502 21:25:43.979080 18407 leader_election.cc:302] T ed78d40318fe4ab5bee4a01b182a34b6 P faadabfb099b4abf90a466395be585b5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 2877108756304e05bd60ba1186b870da, 9be399440ded4e2ebe6b6e0bf486cbb1, faadabfb099b4abf90a466395be585b5; no voters:
I20240502 21:25:43.979199 19427 tablet_bootstrap.cc:654] T 32c9fc5a18c54207881124fda86f42ff P 2877108756304e05bd60ba1186b870da: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:43.979203 19413 tablet_bootstrap.cc:654] T 32c9fc5a18c54207881124fda86f42ff P c75831edb3b44dd59c00ef151adca49d: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:43.979312 18877 raft_consensus.cc:2454] T ed78d40318fe4ab5bee4a01b182a34b6 P c75831edb3b44dd59c00ef151adca49d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate faadabfb099b4abf90a466395be585b5 in term 0.
I20240502 21:25:43.979353 19388 raft_consensus.cc:2781] T ed78d40318fe4ab5bee4a01b182a34b6 P faadabfb099b4abf90a466395be585b5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:25:43.979425 19388 raft_consensus.cc:482] T ed78d40318fe4ab5bee4a01b182a34b6 P faadabfb099b4abf90a466395be585b5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:25:43.979512 19388 raft_consensus.cc:3037] T ed78d40318fe4ab5bee4a01b182a34b6 P faadabfb099b4abf90a466395be585b5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:43.982144 19388 raft_consensus.cc:504] T ed78d40318fe4ab5bee4a01b182a34b6 P faadabfb099b4abf90a466395be585b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } }
I20240502 21:25:43.983219 18877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed78d40318fe4ab5bee4a01b182a34b6" candidate_uuid: "faadabfb099b4abf90a466395be585b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c75831edb3b44dd59c00ef151adca49d"
I20240502 21:25:43.983424 18877 raft_consensus.cc:3037] T ed78d40318fe4ab5bee4a01b182a34b6 P c75831edb3b44dd59c00ef151adca49d [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:43.986529 18877 raft_consensus.cc:2454] T ed78d40318fe4ab5bee4a01b182a34b6 P c75831edb3b44dd59c00ef151adca49d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate faadabfb099b4abf90a466395be585b5 in term 1.
I20240502 21:25:43.987598 18744 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed78d40318fe4ab5bee4a01b182a34b6" candidate_uuid: "faadabfb099b4abf90a466395be585b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "501e3c0d44fe4d70af9ff926d9622417"
I20240502 21:25:43.987818 18744 raft_consensus.cc:3037] T ed78d40318fe4ab5bee4a01b182a34b6 P 501e3c0d44fe4d70af9ff926d9622417 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:43.987872 19413 tablet_bootstrap.cc:492] T 32c9fc5a18c54207881124fda86f42ff P c75831edb3b44dd59c00ef151adca49d: No bootstrap required, opened a new log
I20240502 21:25:43.988036 19413 ts_tablet_manager.cc:1367] T 32c9fc5a18c54207881124fda86f42ff P c75831edb3b44dd59c00ef151adca49d: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.005s
I20240502 21:25:43.988047 19427 tablet_bootstrap.cc:492] T 32c9fc5a18c54207881124fda86f42ff P 2877108756304e05bd60ba1186b870da: No bootstrap required, opened a new log
I20240502 21:25:43.988194 19427 ts_tablet_manager.cc:1367] T 32c9fc5a18c54207881124fda86f42ff P 2877108756304e05bd60ba1186b870da: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.005s
I20240502 21:25:43.988811 19413 raft_consensus.cc:348] T 32c9fc5a18c54207881124fda86f42ff P c75831edb3b44dd59c00ef151adca49d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } }
I20240502 21:25:43.988871 19427 raft_consensus.cc:348] T 32c9fc5a18c54207881124fda86f42ff P 2877108756304e05bd60ba1186b870da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } }
I20240502 21:25:43.989053 19413 raft_consensus.cc:374] T 32c9fc5a18c54207881124fda86f42ff P c75831edb3b44dd59c00ef151adca49d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:43.989104 19427 raft_consensus.cc:374] T 32c9fc5a18c54207881124fda86f42ff P 2877108756304e05bd60ba1186b870da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:43.989136 19413 raft_consensus.cc:729] T 32c9fc5a18c54207881124fda86f42ff P c75831edb3b44dd59c00ef151adca49d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c75831edb3b44dd59c00ef151adca49d, State: Initialized, Role: FOLLOWER
I20240502 21:25:43.989166 19427 raft_consensus.cc:729] T 32c9fc5a18c54207881124fda86f42ff P 2877108756304e05bd60ba1186b870da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2877108756304e05bd60ba1186b870da, State: Initialized, Role: FOLLOWER
I20240502 21:25:43.989390 19413 consensus_queue.cc:260] T 32c9fc5a18c54207881124fda86f42ff P c75831edb3b44dd59c00ef151adca49d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } }
I20240502 21:25:43.989437 19427 consensus_queue.cc:260] T 32c9fc5a18c54207881124fda86f42ff P 2877108756304e05bd60ba1186b870da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } }
I20240502 21:25:43.990149 19413 ts_tablet_manager.cc:1398] T 32c9fc5a18c54207881124fda86f42ff P c75831edb3b44dd59c00ef151adca49d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:25:43.990345 18744 raft_consensus.cc:2454] T ed78d40318fe4ab5bee4a01b182a34b6 P 501e3c0d44fe4d70af9ff926d9622417 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate faadabfb099b4abf90a466395be585b5 in term 1.
I20240502 21:25:43.990738 19427 ts_tablet_manager.cc:1398] T 32c9fc5a18c54207881124fda86f42ff P 2877108756304e05bd60ba1186b870da: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:25:43.990949 19431 raft_consensus.cc:482] T 84b235fd89634168938c0dddb534462a P c75831edb3b44dd59c00ef151adca49d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:25:43.990967 18404 leader_election.cc:302] T ed78d40318fe4ab5bee4a01b182a34b6 P faadabfb099b4abf90a466395be585b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 501e3c0d44fe4d70af9ff926d9622417, c75831edb3b44dd59c00ef151adca49d, faadabfb099b4abf90a466395be585b5; no voters:
I20240502 21:25:43.991262 19436 raft_consensus.cc:2781] T ed78d40318fe4ab5bee4a01b182a34b6 P faadabfb099b4abf90a466395be585b5 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:25:43.991124 19431 raft_consensus.cc:504] T 84b235fd89634168938c0dddb534462a P c75831edb3b44dd59c00ef151adca49d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } }
I20240502 21:25:43.992895 19145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed78d40318fe4ab5bee4a01b182a34b6" candidate_uuid: "faadabfb099b4abf90a466395be585b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2877108756304e05bd60ba1186b870da"
I20240502 21:25:43.993119 19145 raft_consensus.cc:3037] T ed78d40318fe4ab5bee4a01b182a34b6 P 2877108756304e05bd60ba1186b870da [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:43.993600 19431 leader_election.cc:288] T 84b235fd89634168938c0dddb534462a P c75831edb3b44dd59c00ef151adca49d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2877108756304e05bd60ba1186b870da (127.15.255.134:43635), 9be399440ded4e2ebe6b6e0bf486cbb1 (127.15.255.133:34217), faadabfb099b4abf90a466395be585b5 (127.15.255.129:32983), 82f50f4b66f3450aa9f58a0b317f5cea (127.15.255.130:45139)
I20240502 21:25:43.995246 19010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed78d40318fe4ab5bee4a01b182a34b6" candidate_uuid: "faadabfb099b4abf90a466395be585b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1"
I20240502 21:25:43.995321 19388 leader_election.cc:288] T ed78d40318fe4ab5bee4a01b182a34b6 P faadabfb099b4abf90a466395be585b5 [CANDIDATE]: Term 1 election: Requested vote from peers c75831edb3b44dd59c00ef151adca49d (127.15.255.132:43607), 501e3c0d44fe4d70af9ff926d9622417 (127.15.255.131:42501), 2877108756304e05bd60ba1186b870da (127.15.255.134:43635), 9be399440ded4e2ebe6b6e0bf486cbb1 (127.15.255.133:34217)
I20240502 21:25:43.995469 19010 raft_consensus.cc:3037] T ed78d40318fe4ab5bee4a01b182a34b6 P 9be399440ded4e2ebe6b6e0bf486cbb1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:43.996114 19145 raft_consensus.cc:2454] T ed78d40318fe4ab5bee4a01b182a34b6 P 2877108756304e05bd60ba1186b870da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate faadabfb099b4abf90a466395be585b5 in term 1.
I20240502 21:25:43.996608 19436 raft_consensus.cc:686] T ed78d40318fe4ab5bee4a01b182a34b6 P faadabfb099b4abf90a466395be585b5 [term 1 LEADER]: Becoming Leader. State: Replica: faadabfb099b4abf90a466395be585b5, State: Running, Role: LEADER
I20240502 21:25:43.996990 19436 consensus_queue.cc:237] T ed78d40318fe4ab5bee4a01b182a34b6 P faadabfb099b4abf90a466395be585b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } }
I20240502 21:25:43.997591 19010 raft_consensus.cc:2454] T ed78d40318fe4ab5bee4a01b182a34b6 P 9be399440ded4e2ebe6b6e0bf486cbb1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate faadabfb099b4abf90a466395be585b5 in term 1.
I20240502 21:25:44.000876 18329 catalog_manager.cc:5484] T ed78d40318fe4ab5bee4a01b182a34b6 P faadabfb099b4abf90a466395be585b5 reported cstate change: term changed from 0 to 1, leader changed from <none> to faadabfb099b4abf90a466395be585b5 (127.15.255.129). New cstate: current_term: 1 leader_uuid: "faadabfb099b4abf90a466395be585b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } health_report { overall_health: HEALTHY } } }
I20240502 21:25:44.007068 19145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84b235fd89634168938c0dddb534462a" candidate_uuid: "c75831edb3b44dd59c00ef151adca49d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2877108756304e05bd60ba1186b870da" is_pre_election: true
I20240502 21:25:44.007257 19145 raft_consensus.cc:2454] T 84b235fd89634168938c0dddb534462a P 2877108756304e05bd60ba1186b870da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c75831edb3b44dd59c00ef151adca49d in term 0.
I20240502 21:25:44.014200 18611 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84b235fd89634168938c0dddb534462a" candidate_uuid: "c75831edb3b44dd59c00ef151adca49d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" is_pre_election: true
I20240502 21:25:44.014420 18611 raft_consensus.cc:2454] T 84b235fd89634168938c0dddb534462a P 82f50f4b66f3450aa9f58a0b317f5cea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c75831edb3b44dd59c00ef151adca49d in term 0.
I20240502 21:25:44.015259 19010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84b235fd89634168938c0dddb534462a" candidate_uuid: "c75831edb3b44dd59c00ef151adca49d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" is_pre_election: true
I20240502 21:25:44.015471 19010 raft_consensus.cc:2454] T 84b235fd89634168938c0dddb534462a P 9be399440ded4e2ebe6b6e0bf486cbb1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c75831edb3b44dd59c00ef151adca49d in term 0.
I20240502 21:25:44.015832 18812 leader_election.cc:302] T 84b235fd89634168938c0dddb534462a P c75831edb3b44dd59c00ef151adca49d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 2877108756304e05bd60ba1186b870da, 82f50f4b66f3450aa9f58a0b317f5cea, c75831edb3b44dd59c00ef151adca49d; no voters:
I20240502 21:25:44.015901 19436 raft_consensus.cc:482] T 84b235fd89634168938c0dddb534462a P faadabfb099b4abf90a466395be585b5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:25:44.016068 19436 raft_consensus.cc:504] T 84b235fd89634168938c0dddb534462a P faadabfb099b4abf90a466395be585b5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } }
I20240502 21:25:44.016196 19431 raft_consensus.cc:2781] T 84b235fd89634168938c0dddb534462a P c75831edb3b44dd59c00ef151adca49d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:25:44.016299 19431 raft_consensus.cc:482] T 84b235fd89634168938c0dddb534462a P c75831edb3b44dd59c00ef151adca49d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:25:44.016389 19431 raft_consensus.cc:3037] T 84b235fd89634168938c0dddb534462a P c75831edb3b44dd59c00ef151adca49d [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:44.016629 18474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84b235fd89634168938c0dddb534462a" candidate_uuid: "c75831edb3b44dd59c00ef151adca49d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "faadabfb099b4abf90a466395be585b5" is_pre_election: true
I20240502 21:25:44.016862 18474 raft_consensus.cc:2454] T 84b235fd89634168938c0dddb534462a P faadabfb099b4abf90a466395be585b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c75831edb3b44dd59c00ef151adca49d in term 0.
I20240502 21:25:44.017079 19436 leader_election.cc:288] T 84b235fd89634168938c0dddb534462a P faadabfb099b4abf90a466395be585b5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2877108756304e05bd60ba1186b870da (127.15.255.134:43635), c75831edb3b44dd59c00ef151adca49d (127.15.255.132:43607), 9be399440ded4e2ebe6b6e0bf486cbb1 (127.15.255.133:34217), 82f50f4b66f3450aa9f58a0b317f5cea (127.15.255.130:45139)
I20240502 21:25:44.017513 19145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84b235fd89634168938c0dddb534462a" candidate_uuid: "faadabfb099b4abf90a466395be585b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2877108756304e05bd60ba1186b870da" is_pre_election: true
I20240502 21:25:44.017695 19145 raft_consensus.cc:2454] T 84b235fd89634168938c0dddb534462a P 2877108756304e05bd60ba1186b870da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate faadabfb099b4abf90a466395be585b5 in term 0.
I20240502 21:25:44.017695 19010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84b235fd89634168938c0dddb534462a" candidate_uuid: "faadabfb099b4abf90a466395be585b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" is_pre_election: true
I20240502 21:25:44.017886 19010 raft_consensus.cc:2454] T 84b235fd89634168938c0dddb534462a P 9be399440ded4e2ebe6b6e0bf486cbb1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate faadabfb099b4abf90a466395be585b5 in term 0.
I20240502 21:25:44.018054 18611 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84b235fd89634168938c0dddb534462a" candidate_uuid: "faadabfb099b4abf90a466395be585b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" is_pre_election: true
I20240502 21:25:44.018196 18405 leader_election.cc:302] T 84b235fd89634168938c0dddb534462a P faadabfb099b4abf90a466395be585b5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 2877108756304e05bd60ba1186b870da, 9be399440ded4e2ebe6b6e0bf486cbb1, faadabfb099b4abf90a466395be585b5; no voters:
I20240502 21:25:44.018234 18611 raft_consensus.cc:2454] T 84b235fd89634168938c0dddb534462a P 82f50f4b66f3450aa9f58a0b317f5cea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate faadabfb099b4abf90a466395be585b5 in term 0.
I20240502 21:25:44.018620 19436 raft_consensus.cc:2781] T 84b235fd89634168938c0dddb534462a P faadabfb099b4abf90a466395be585b5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:25:44.018718 19436 raft_consensus.cc:482] T 84b235fd89634168938c0dddb534462a P faadabfb099b4abf90a466395be585b5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:25:44.018817 19436 raft_consensus.cc:3037] T 84b235fd89634168938c0dddb534462a P faadabfb099b4abf90a466395be585b5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:44.018764 18877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84b235fd89634168938c0dddb534462a" candidate_uuid: "faadabfb099b4abf90a466395be585b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c75831edb3b44dd59c00ef151adca49d" is_pre_election: true
I20240502 21:25:44.019021 19431 raft_consensus.cc:504] T 84b235fd89634168938c0dddb534462a P c75831edb3b44dd59c00ef151adca49d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } }
I20240502 21:25:44.019459 18877 raft_consensus.cc:2379] T 84b235fd89634168938c0dddb534462a P c75831edb3b44dd59c00ef151adca49d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate faadabfb099b4abf90a466395be585b5 in current term 1: Already voted for candidate c75831edb3b44dd59c00ef151adca49d in this term.
I20240502 21:25:44.020010 19431 leader_election.cc:288] T 84b235fd89634168938c0dddb534462a P c75831edb3b44dd59c00ef151adca49d [CANDIDATE]: Term 1 election: Requested vote from peers 2877108756304e05bd60ba1186b870da (127.15.255.134:43635), 9be399440ded4e2ebe6b6e0bf486cbb1 (127.15.255.133:34217), faadabfb099b4abf90a466395be585b5 (127.15.255.129:32983), 82f50f4b66f3450aa9f58a0b317f5cea (127.15.255.130:45139)
I20240502 21:25:44.020046 19145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84b235fd89634168938c0dddb534462a" candidate_uuid: "c75831edb3b44dd59c00ef151adca49d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2877108756304e05bd60ba1186b870da"
I20240502 21:25:44.020243 19145 raft_consensus.cc:3037] T 84b235fd89634168938c0dddb534462a P 2877108756304e05bd60ba1186b870da [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:44.020383 18474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84b235fd89634168938c0dddb534462a" candidate_uuid: "c75831edb3b44dd59c00ef151adca49d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "faadabfb099b4abf90a466395be585b5"
I20240502 21:25:44.021304 19436 raft_consensus.cc:504] T 84b235fd89634168938c0dddb534462a P faadabfb099b4abf90a466395be585b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } }
I20240502 21:25:44.021703 18474 raft_consensus.cc:2379] T 84b235fd89634168938c0dddb534462a P faadabfb099b4abf90a466395be585b5 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate c75831edb3b44dd59c00ef151adca49d in current term 1: Already voted for candidate faadabfb099b4abf90a466395be585b5 in this term.
I20240502 21:25:44.022053 19436 leader_election.cc:288] T 84b235fd89634168938c0dddb534462a P faadabfb099b4abf90a466395be585b5 [CANDIDATE]: Term 1 election: Requested vote from peers 2877108756304e05bd60ba1186b870da (127.15.255.134:43635), c75831edb3b44dd59c00ef151adca49d (127.15.255.132:43607), 9be399440ded4e2ebe6b6e0bf486cbb1 (127.15.255.133:34217), 82f50f4b66f3450aa9f58a0b317f5cea (127.15.255.130:45139)
I20240502 21:25:44.022598 19145 raft_consensus.cc:2454] T 84b235fd89634168938c0dddb534462a P 2877108756304e05bd60ba1186b870da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c75831edb3b44dd59c00ef151adca49d in term 1.
I20240502 21:25:44.022889 18877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84b235fd89634168938c0dddb534462a" candidate_uuid: "faadabfb099b4abf90a466395be585b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c75831edb3b44dd59c00ef151adca49d"
I20240502 21:25:44.023037 18877 raft_consensus.cc:2379] T 84b235fd89634168938c0dddb534462a P c75831edb3b44dd59c00ef151adca49d [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate faadabfb099b4abf90a466395be585b5 in current term 1: Already voted for candidate c75831edb3b44dd59c00ef151adca49d in this term.
I20240502 21:25:44.023334 19010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84b235fd89634168938c0dddb534462a" candidate_uuid: "faadabfb099b4abf90a466395be585b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1"
I20240502 21:25:44.023466 19010 raft_consensus.cc:3037] T 84b235fd89634168938c0dddb534462a P 9be399440ded4e2ebe6b6e0bf486cbb1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:44.023730 19145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84b235fd89634168938c0dddb534462a" candidate_uuid: "faadabfb099b4abf90a466395be585b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2877108756304e05bd60ba1186b870da"
I20240502 21:25:44.023779 18611 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84b235fd89634168938c0dddb534462a" candidate_uuid: "faadabfb099b4abf90a466395be585b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82f50f4b66f3450aa9f58a0b317f5cea"
I20240502 21:25:44.023927 18611 raft_consensus.cc:3037] T 84b235fd89634168938c0dddb534462a P 82f50f4b66f3450aa9f58a0b317f5cea [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:44.023994 19145 raft_consensus.cc:2379] T 84b235fd89634168938c0dddb534462a P 2877108756304e05bd60ba1186b870da [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate faadabfb099b4abf90a466395be585b5 in current term 1: Already voted for candidate c75831edb3b44dd59c00ef151adca49d in this term.
I20240502 21:25:44.025396 19010 raft_consensus.cc:2454] T 84b235fd89634168938c0dddb534462a P 9be399440ded4e2ebe6b6e0bf486cbb1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate faadabfb099b4abf90a466395be585b5 in term 1.
I20240502 21:25:44.026015 18611 raft_consensus.cc:2454] T 84b235fd89634168938c0dddb534462a P 82f50f4b66f3450aa9f58a0b317f5cea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate faadabfb099b4abf90a466395be585b5 in term 1.
I20240502 21:25:44.026567 19010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84b235fd89634168938c0dddb534462a" candidate_uuid: "c75831edb3b44dd59c00ef151adca49d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1"
I20240502 21:25:44.026681 18407 leader_election.cc:302] T 84b235fd89634168938c0dddb534462a P faadabfb099b4abf90a466395be585b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 5 responses out of 5 voters: 3 yes votes; 2 no votes. yes voters: 82f50f4b66f3450aa9f58a0b317f5cea, 9be399440ded4e2ebe6b6e0bf486cbb1, faadabfb099b4abf90a466395be585b5; no voters: 2877108756304e05bd60ba1186b870da, c75831edb3b44dd59c00ef151adca49d
I20240502 21:25:44.026829 19010 raft_consensus.cc:2379] T 84b235fd89634168938c0dddb534462a P 9be399440ded4e2ebe6b6e0bf486cbb1 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate c75831edb3b44dd59c00ef151adca49d in current term 1: Already voted for candidate faadabfb099b4abf90a466395be585b5 in this term.
I20240502 21:25:44.026749 18611 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84b235fd89634168938c0dddb534462a" candidate_uuid: "c75831edb3b44dd59c00ef151adca49d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82f50f4b66f3450aa9f58a0b317f5cea"
I20240502 21:25:44.026954 19436 raft_consensus.cc:2781] T 84b235fd89634168938c0dddb534462a P faadabfb099b4abf90a466395be585b5 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:25:44.027048 18611 raft_consensus.cc:2379] T 84b235fd89634168938c0dddb534462a P 82f50f4b66f3450aa9f58a0b317f5cea [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate c75831edb3b44dd59c00ef151adca49d in current term 1: Already voted for candidate faadabfb099b4abf90a466395be585b5 in this term.
I20240502 21:25:44.027218 19436 raft_consensus.cc:686] T 84b235fd89634168938c0dddb534462a P faadabfb099b4abf90a466395be585b5 [term 1 LEADER]: Becoming Leader. State: Replica: faadabfb099b4abf90a466395be585b5, State: Running, Role: LEADER
I20240502 21:25:44.027505 19436 consensus_queue.cc:237] T 84b235fd89634168938c0dddb534462a P faadabfb099b4abf90a466395be585b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } }
I20240502 21:25:44.027563 18812 leader_election.cc:302] T 84b235fd89634168938c0dddb534462a P c75831edb3b44dd59c00ef151adca49d [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 5 responses out of 5 voters: 2 yes votes; 3 no votes. yes voters: 2877108756304e05bd60ba1186b870da, c75831edb3b44dd59c00ef151adca49d; no voters: 82f50f4b66f3450aa9f58a0b317f5cea, 9be399440ded4e2ebe6b6e0bf486cbb1, faadabfb099b4abf90a466395be585b5
I20240502 21:25:44.027825 19431 raft_consensus.cc:2726] T 84b235fd89634168938c0dddb534462a P c75831edb3b44dd59c00ef151adca49d [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240502 21:25:44.030107 18329 catalog_manager.cc:5484] T 84b235fd89634168938c0dddb534462a P faadabfb099b4abf90a466395be585b5 reported cstate change: term changed from 0 to 1, leader changed from <none> to faadabfb099b4abf90a466395be585b5 (127.15.255.129). New cstate: current_term: 1 leader_uuid: "faadabfb099b4abf90a466395be585b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } health_report { overall_health: UNKNOWN } } }
W20240502 21:25:44.065637 18925 tablet.cc:2369] T 32c9fc5a18c54207881124fda86f42ff P c75831edb3b44dd59c00ef151adca49d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240502 21:25:44.170081 19193 tablet.cc:2369] T 32c9fc5a18c54207881124fda86f42ff P 2877108756304e05bd60ba1186b870da: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 21:25:44.279668 19431 raft_consensus.cc:482] T 32c9fc5a18c54207881124fda86f42ff P c75831edb3b44dd59c00ef151adca49d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:25:44.279871 19431 raft_consensus.cc:504] T 32c9fc5a18c54207881124fda86f42ff P c75831edb3b44dd59c00ef151adca49d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } }
I20240502 21:25:44.280820 19431 leader_election.cc:288] T 32c9fc5a18c54207881124fda86f42ff P c75831edb3b44dd59c00ef151adca49d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 82f50f4b66f3450aa9f58a0b317f5cea (127.15.255.130:45139), 2877108756304e05bd60ba1186b870da (127.15.255.134:43635), faadabfb099b4abf90a466395be585b5 (127.15.255.129:32983), 501e3c0d44fe4d70af9ff926d9622417 (127.15.255.131:42501)
I20240502 21:25:44.281209 18611 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32c9fc5a18c54207881124fda86f42ff" candidate_uuid: "c75831edb3b44dd59c00ef151adca49d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" is_pre_election: true
I20240502 21:25:44.281489 18611 raft_consensus.cc:2379] T 32c9fc5a18c54207881124fda86f42ff P 82f50f4b66f3450aa9f58a0b317f5cea [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c75831edb3b44dd59c00ef151adca49d in current term 1: Already voted for candidate faadabfb099b4abf90a466395be585b5 in this term.
I20240502 21:25:44.281867 18474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32c9fc5a18c54207881124fda86f42ff" candidate_uuid: "c75831edb3b44dd59c00ef151adca49d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "faadabfb099b4abf90a466395be585b5" is_pre_election: true
I20240502 21:25:44.282639 19145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32c9fc5a18c54207881124fda86f42ff" candidate_uuid: "c75831edb3b44dd59c00ef151adca49d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2877108756304e05bd60ba1186b870da" is_pre_election: true
I20240502 21:25:44.282886 19145 raft_consensus.cc:2454] T 32c9fc5a18c54207881124fda86f42ff P 2877108756304e05bd60ba1186b870da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c75831edb3b44dd59c00ef151adca49d in term 0.
I20240502 21:25:44.287169 18744 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32c9fc5a18c54207881124fda86f42ff" candidate_uuid: "c75831edb3b44dd59c00ef151adca49d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "501e3c0d44fe4d70af9ff926d9622417" is_pre_election: true
I20240502 21:25:44.287520 18744 raft_consensus.cc:2379] T 32c9fc5a18c54207881124fda86f42ff P 501e3c0d44fe4d70af9ff926d9622417 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c75831edb3b44dd59c00ef151adca49d in current term 1: Already voted for candidate faadabfb099b4abf90a466395be585b5 in this term.
I20240502 21:25:44.288031 18809 leader_election.cc:302] T 32c9fc5a18c54207881124fda86f42ff P c75831edb3b44dd59c00ef151adca49d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 5 responses out of 5 voters: 2 yes votes; 3 no votes. yes voters: 2877108756304e05bd60ba1186b870da, c75831edb3b44dd59c00ef151adca49d; no voters: 501e3c0d44fe4d70af9ff926d9622417, 82f50f4b66f3450aa9f58a0b317f5cea, faadabfb099b4abf90a466395be585b5
I20240502 21:25:44.288556 19431 raft_consensus.cc:3037] T 32c9fc5a18c54207881124fda86f42ff P c75831edb3b44dd59c00ef151adca49d [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:44.290426 19431 raft_consensus.cc:2726] T 32c9fc5a18c54207881124fda86f42ff P c75831edb3b44dd59c00ef151adca49d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 21:25:44.347402 19145 raft_consensus.cc:3037] T 32c9fc5a18c54207881124fda86f42ff P 2877108756304e05bd60ba1186b870da [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:44.350313 19436 consensus_queue.cc:1035] T 32c9fc5a18c54207881124fda86f42ff P faadabfb099b4abf90a466395be585b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 21:25:44.378039 19450 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:25:44.378304 19450 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:25:44.385870 19450 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 21:25:44.395705 19436 consensus_queue.cc:1035] T 32c9fc5a18c54207881124fda86f42ff P faadabfb099b4abf90a466395be585b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:25:44.403481 19388 consensus_queue.cc:1035] T 32c9fc5a18c54207881124fda86f42ff P faadabfb099b4abf90a466395be585b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:25:44.405869 19388 consensus_queue.cc:1035] T 32c9fc5a18c54207881124fda86f42ff P faadabfb099b4abf90a466395be585b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:25:44.442627 19462 consensus_queue.cc:1035] T 84b235fd89634168938c0dddb534462a P faadabfb099b4abf90a466395be585b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:25:44.446815 19388 consensus_queue.cc:1035] T 84b235fd89634168938c0dddb534462a P faadabfb099b4abf90a466395be585b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:25:44.474684 19436 consensus_queue.cc:1035] T 84b235fd89634168938c0dddb534462a P faadabfb099b4abf90a466395be585b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:25:44.477766 19388 consensus_queue.cc:1035] T 84b235fd89634168938c0dddb534462a P faadabfb099b4abf90a466395be585b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_0'
I20240502 21:25:44.498500 19388 consensus_queue.cc:1035] T ed78d40318fe4ab5bee4a01b182a34b6 P faadabfb099b4abf90a466395be585b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:25:44.564846 19388 consensus_queue.cc:1035] T ed78d40318fe4ab5bee4a01b182a34b6 P faadabfb099b4abf90a466395be585b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:25:44.568673 19388 consensus_queue.cc:1035] T ed78d40318fe4ab5bee4a01b182a34b6 P faadabfb099b4abf90a466395be585b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:25:44.576797 19436 consensus_queue.cc:1035] T ed78d40318fe4ab5bee4a01b182a34b6 P faadabfb099b4abf90a466395be585b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 21:25:44.761425 18404 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.131:42501, user_credentials={real_user=slave}} blocked reactor thread for 68424.6us
INSERT report
rows total: 2000
time total: 426.017 ms
time per row: 0.213009 ms
W20240502 21:25:45.589574 18406 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.255.135:43395: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 21:25:45.590945 18406 consensus_peers.cc:485] T 40d4f61d0f6d4d31a5626c7743c88133 P faadabfb099b4abf90a466395be585b5 -> Peer 90e329b457c24caf93a9f880345eb288 (127.15.255.135:43395): Couldn't send request to peer 90e329b457c24caf93a9f880345eb288. Status: Network error: Client connection negotiation failed: client connection to 127.15.255.135:43395: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
-----------------------------------------------------
Suppressions used:
count bytes template
4 1192 libcrypto.so.1.1
-----------------------------------------------------
I20240502 21:25:45.916466 18324 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60616:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 5
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 21:25:45.925530 18589 tablet_service.cc:1467] Processing CreateTablet for tablet 37e9dcc6191c42d3a5c1503dedc5511e (DEFAULT_TABLE table=rebalance_test_table_1 [id=76deee1e4597471ebedfc30f6d635eaa]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:45.926018 18589 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37e9dcc6191c42d3a5c1503dedc5511e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:45.926179 18857 tablet_service.cc:1467] Processing CreateTablet for tablet 37e9dcc6191c42d3a5c1503dedc5511e (DEFAULT_TABLE table=rebalance_test_table_1 [id=76deee1e4597471ebedfc30f6d635eaa]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:45.926602 18857 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37e9dcc6191c42d3a5c1503dedc5511e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:45.926851 18590 tablet_service.cc:1467] Processing CreateTablet for tablet e0e69b88ad014363b97d1f650df693a8 (DEFAULT_TABLE table=rebalance_test_table_1 [id=76deee1e4597471ebedfc30f6d635eaa]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:45.927230 18590 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0e69b88ad014363b97d1f650df693a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:45.928769 19123 tablet_service.cc:1467] Processing CreateTablet for tablet 37e9dcc6191c42d3a5c1503dedc5511e (DEFAULT_TABLE table=rebalance_test_table_1 [id=76deee1e4597471ebedfc30f6d635eaa]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:45.929186 19123 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37e9dcc6191c42d3a5c1503dedc5511e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:45.929380 18856 tablet_service.cc:1467] Processing CreateTablet for tablet e0e69b88ad014363b97d1f650df693a8 (DEFAULT_TABLE table=rebalance_test_table_1 [id=76deee1e4597471ebedfc30f6d635eaa]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:45.929735 18856 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0e69b88ad014363b97d1f650df693a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:45.932298 18724 tablet_service.cc:1467] Processing CreateTablet for tablet 37e9dcc6191c42d3a5c1503dedc5511e (DEFAULT_TABLE table=rebalance_test_table_1 [id=76deee1e4597471ebedfc30f6d635eaa]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:45.932744 18724 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37e9dcc6191c42d3a5c1503dedc5511e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:45.933440 19125 tablet_service.cc:1467] Processing CreateTablet for tablet e0e69b88ad014363b97d1f650df693a8 (DEFAULT_TABLE table=rebalance_test_table_1 [id=76deee1e4597471ebedfc30f6d635eaa]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:45.933810 19125 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0e69b88ad014363b97d1f650df693a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:45.933881 18989 tablet_service.cc:1467] Processing CreateTablet for tablet 37e9dcc6191c42d3a5c1503dedc5511e (DEFAULT_TABLE table=rebalance_test_table_1 [id=76deee1e4597471ebedfc30f6d635eaa]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:45.934262 18989 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37e9dcc6191c42d3a5c1503dedc5511e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:45.936779 18990 tablet_service.cc:1467] Processing CreateTablet for tablet e0e69b88ad014363b97d1f650df693a8 (DEFAULT_TABLE table=rebalance_test_table_1 [id=76deee1e4597471ebedfc30f6d635eaa]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:45.937111 18990 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0e69b88ad014363b97d1f650df693a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:45.937331 18588 tablet_service.cc:1467] Processing CreateTablet for tablet a7a9256f2b784abcb780b190e2ff3983 (DEFAULT_TABLE table=rebalance_test_table_1 [id=76deee1e4597471ebedfc30f6d635eaa]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:45.937680 18588 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7a9256f2b784abcb780b190e2ff3983. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:45.938838 18855 tablet_service.cc:1467] Processing CreateTablet for tablet a7a9256f2b784abcb780b190e2ff3983 (DEFAULT_TABLE table=rebalance_test_table_1 [id=76deee1e4597471ebedfc30f6d635eaa]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:45.939231 18855 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7a9256f2b784abcb780b190e2ff3983. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:45.940778 18988 tablet_service.cc:1467] Processing CreateTablet for tablet a7a9256f2b784abcb780b190e2ff3983 (DEFAULT_TABLE table=rebalance_test_table_1 [id=76deee1e4597471ebedfc30f6d635eaa]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:45.941138 18988 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7a9256f2b784abcb780b190e2ff3983. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:45.944821 18452 tablet_service.cc:1467] Processing CreateTablet for tablet a7a9256f2b784abcb780b190e2ff3983 (DEFAULT_TABLE table=rebalance_test_table_1 [id=76deee1e4597471ebedfc30f6d635eaa]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:45.945240 18452 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7a9256f2b784abcb780b190e2ff3983. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:45.946868 19548 tablet_bootstrap.cc:492] T 37e9dcc6191c42d3a5c1503dedc5511e P 82f50f4b66f3450aa9f58a0b317f5cea: Bootstrap starting.
I20240502 21:25:45.950582 19549 tablet_bootstrap.cc:492] T 37e9dcc6191c42d3a5c1503dedc5511e P c75831edb3b44dd59c00ef151adca49d: Bootstrap starting.
I20240502 21:25:45.950677 19548 tablet_bootstrap.cc:654] T 37e9dcc6191c42d3a5c1503dedc5511e P 82f50f4b66f3450aa9f58a0b317f5cea: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:45.952685 19549 tablet_bootstrap.cc:654] T 37e9dcc6191c42d3a5c1503dedc5511e P c75831edb3b44dd59c00ef151adca49d: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:45.953801 19548 tablet_bootstrap.cc:492] T 37e9dcc6191c42d3a5c1503dedc5511e P 82f50f4b66f3450aa9f58a0b317f5cea: No bootstrap required, opened a new log
I20240502 21:25:45.954034 19548 ts_tablet_manager.cc:1367] T 37e9dcc6191c42d3a5c1503dedc5511e P 82f50f4b66f3450aa9f58a0b317f5cea: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.003s
I20240502 21:25:45.954175 19553 tablet_bootstrap.cc:492] T a7a9256f2b784abcb780b190e2ff3983 P faadabfb099b4abf90a466395be585b5: Bootstrap starting.
I20240502 21:25:45.954811 19548 raft_consensus.cc:348] T 37e9dcc6191c42d3a5c1503dedc5511e P 82f50f4b66f3450aa9f58a0b317f5cea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } }
I20240502 21:25:45.955534 19548 raft_consensus.cc:374] T 37e9dcc6191c42d3a5c1503dedc5511e P 82f50f4b66f3450aa9f58a0b317f5cea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:45.955699 19548 raft_consensus.cc:729] T 37e9dcc6191c42d3a5c1503dedc5511e P 82f50f4b66f3450aa9f58a0b317f5cea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82f50f4b66f3450aa9f58a0b317f5cea, State: Initialized, Role: FOLLOWER
I20240502 21:25:45.955994 19548 consensus_queue.cc:260] T 37e9dcc6191c42d3a5c1503dedc5511e P 82f50f4b66f3450aa9f58a0b317f5cea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } }
I20240502 21:25:45.956858 19553 tablet_bootstrap.cc:654] T a7a9256f2b784abcb780b190e2ff3983 P faadabfb099b4abf90a466395be585b5: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:45.960932 19557 tablet_bootstrap.cc:492] T e0e69b88ad014363b97d1f650df693a8 P 2877108756304e05bd60ba1186b870da: Bootstrap starting.
I20240502 21:25:45.961642 19549 tablet_bootstrap.cc:492] T 37e9dcc6191c42d3a5c1503dedc5511e P c75831edb3b44dd59c00ef151adca49d: No bootstrap required, opened a new log
I20240502 21:25:45.962253 19549 ts_tablet_manager.cc:1367] T 37e9dcc6191c42d3a5c1503dedc5511e P c75831edb3b44dd59c00ef151adca49d: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.000s
I20240502 21:25:45.963199 19557 tablet_bootstrap.cc:654] T e0e69b88ad014363b97d1f650df693a8 P 2877108756304e05bd60ba1186b870da: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:45.963186 19549 raft_consensus.cc:348] T 37e9dcc6191c42d3a5c1503dedc5511e P c75831edb3b44dd59c00ef151adca49d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } }
I20240502 21:25:45.963434 19549 raft_consensus.cc:374] T 37e9dcc6191c42d3a5c1503dedc5511e P c75831edb3b44dd59c00ef151adca49d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:45.963527 19549 raft_consensus.cc:729] T 37e9dcc6191c42d3a5c1503dedc5511e P c75831edb3b44dd59c00ef151adca49d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c75831edb3b44dd59c00ef151adca49d, State: Initialized, Role: FOLLOWER
I20240502 21:25:45.963763 19549 consensus_queue.cc:260] T 37e9dcc6191c42d3a5c1503dedc5511e P c75831edb3b44dd59c00ef151adca49d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } }
I20240502 21:25:45.966209 19548 ts_tablet_manager.cc:1398] T 37e9dcc6191c42d3a5c1503dedc5511e P 82f50f4b66f3450aa9f58a0b317f5cea: Time spent starting tablet: real 0.012s user 0.004s sys 0.004s
I20240502 21:25:45.966634 19548 tablet_bootstrap.cc:492] T e0e69b88ad014363b97d1f650df693a8 P 82f50f4b66f3450aa9f58a0b317f5cea: Bootstrap starting.
I20240502 21:25:45.966869 19553 tablet_bootstrap.cc:492] T a7a9256f2b784abcb780b190e2ff3983 P faadabfb099b4abf90a466395be585b5: No bootstrap required, opened a new log
I20240502 21:25:45.967073 19553 ts_tablet_manager.cc:1367] T a7a9256f2b784abcb780b190e2ff3983 P faadabfb099b4abf90a466395be585b5: Time spent bootstrapping tablet: real 0.013s user 0.009s sys 0.000s
I20240502 21:25:45.967895 19553 raft_consensus.cc:348] T a7a9256f2b784abcb780b190e2ff3983 P faadabfb099b4abf90a466395be585b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } }
I20240502 21:25:45.968148 19548 tablet_bootstrap.cc:654] T e0e69b88ad014363b97d1f650df693a8 P 82f50f4b66f3450aa9f58a0b317f5cea: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:45.968186 19553 raft_consensus.cc:374] T a7a9256f2b784abcb780b190e2ff3983 P faadabfb099b4abf90a466395be585b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:45.968333 19553 raft_consensus.cc:729] T a7a9256f2b784abcb780b190e2ff3983 P faadabfb099b4abf90a466395be585b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: faadabfb099b4abf90a466395be585b5, State: Initialized, Role: FOLLOWER
I20240502 21:25:45.968708 19553 consensus_queue.cc:260] T a7a9256f2b784abcb780b190e2ff3983 P faadabfb099b4abf90a466395be585b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } }
I20240502 21:25:45.972944 19552 tablet_bootstrap.cc:492] T 37e9dcc6191c42d3a5c1503dedc5511e P 9be399440ded4e2ebe6b6e0bf486cbb1: Bootstrap starting.
I20240502 21:25:45.973196 19553 ts_tablet_manager.cc:1398] T a7a9256f2b784abcb780b190e2ff3983 P faadabfb099b4abf90a466395be585b5: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240502 21:25:45.973412 19549 ts_tablet_manager.cc:1398] T 37e9dcc6191c42d3a5c1503dedc5511e P c75831edb3b44dd59c00ef151adca49d: Time spent starting tablet: real 0.011s user 0.001s sys 0.004s
I20240502 21:25:45.974090 19549 tablet_bootstrap.cc:492] T e0e69b88ad014363b97d1f650df693a8 P c75831edb3b44dd59c00ef151adca49d: Bootstrap starting.
I20240502 21:25:45.975627 18723 tablet_service.cc:1467] Processing CreateTablet for tablet e0e69b88ad014363b97d1f650df693a8 (DEFAULT_TABLE table=rebalance_test_table_1 [id=76deee1e4597471ebedfc30f6d635eaa]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:45.976080 18723 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0e69b88ad014363b97d1f650df693a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:45.975733 19550 tablet_bootstrap.cc:492] T 37e9dcc6191c42d3a5c1503dedc5511e P 501e3c0d44fe4d70af9ff926d9622417: Bootstrap starting.
I20240502 21:25:45.976198 19549 tablet_bootstrap.cc:654] T e0e69b88ad014363b97d1f650df693a8 P c75831edb3b44dd59c00ef151adca49d: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:45.977849 19550 tablet_bootstrap.cc:654] T 37e9dcc6191c42d3a5c1503dedc5511e P 501e3c0d44fe4d70af9ff926d9622417: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:45.976011 19552 tablet_bootstrap.cc:654] T 37e9dcc6191c42d3a5c1503dedc5511e P 9be399440ded4e2ebe6b6e0bf486cbb1: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:45.980870 18722 tablet_service.cc:1467] Processing CreateTablet for tablet a7a9256f2b784abcb780b190e2ff3983 (DEFAULT_TABLE table=rebalance_test_table_1 [id=76deee1e4597471ebedfc30f6d635eaa]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:45.981289 18722 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7a9256f2b784abcb780b190e2ff3983. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:45.985450 19549 tablet_bootstrap.cc:492] T e0e69b88ad014363b97d1f650df693a8 P c75831edb3b44dd59c00ef151adca49d: No bootstrap required, opened a new log
I20240502 21:25:45.985672 19549 ts_tablet_manager.cc:1367] T e0e69b88ad014363b97d1f650df693a8 P c75831edb3b44dd59c00ef151adca49d: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.004s
I20240502 21:25:45.986789 19549 raft_consensus.cc:348] T e0e69b88ad014363b97d1f650df693a8 P c75831edb3b44dd59c00ef151adca49d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } }
I20240502 21:25:45.987102 19549 raft_consensus.cc:374] T e0e69b88ad014363b97d1f650df693a8 P c75831edb3b44dd59c00ef151adca49d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:45.987208 19549 raft_consensus.cc:729] T e0e69b88ad014363b97d1f650df693a8 P c75831edb3b44dd59c00ef151adca49d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c75831edb3b44dd59c00ef151adca49d, State: Initialized, Role: FOLLOWER
I20240502 21:25:45.988219 19548 tablet_bootstrap.cc:492] T e0e69b88ad014363b97d1f650df693a8 P 82f50f4b66f3450aa9f58a0b317f5cea: No bootstrap required, opened a new log
I20240502 21:25:45.988269 19550 tablet_bootstrap.cc:492] T 37e9dcc6191c42d3a5c1503dedc5511e P 501e3c0d44fe4d70af9ff926d9622417: No bootstrap required, opened a new log
I20240502 21:25:45.988354 19557 tablet_bootstrap.cc:492] T e0e69b88ad014363b97d1f650df693a8 P 2877108756304e05bd60ba1186b870da: No bootstrap required, opened a new log
I20240502 21:25:45.988449 19548 ts_tablet_manager.cc:1367] T e0e69b88ad014363b97d1f650df693a8 P 82f50f4b66f3450aa9f58a0b317f5cea: Time spent bootstrapping tablet: real 0.022s user 0.005s sys 0.001s
I20240502 21:25:45.988478 19550 ts_tablet_manager.cc:1367] T 37e9dcc6191c42d3a5c1503dedc5511e P 501e3c0d44fe4d70af9ff926d9622417: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.000s
I20240502 21:25:45.988572 19557 ts_tablet_manager.cc:1367] T e0e69b88ad014363b97d1f650df693a8 P 2877108756304e05bd60ba1186b870da: Time spent bootstrapping tablet: real 0.028s user 0.001s sys 0.005s
I20240502 21:25:45.987726 19549 consensus_queue.cc:260] T e0e69b88ad014363b97d1f650df693a8 P c75831edb3b44dd59c00ef151adca49d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } }
I20240502 21:25:45.989286 19549 ts_tablet_manager.cc:1398] T e0e69b88ad014363b97d1f650df693a8 P c75831edb3b44dd59c00ef151adca49d: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240502 21:25:45.989174 19548 raft_consensus.cc:348] T e0e69b88ad014363b97d1f650df693a8 P 82f50f4b66f3450aa9f58a0b317f5cea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } }
I20240502 21:25:45.989220 19550 raft_consensus.cc:348] T 37e9dcc6191c42d3a5c1503dedc5511e P 501e3c0d44fe4d70af9ff926d9622417 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } }
I20240502 21:25:45.989424 19550 raft_consensus.cc:374] T 37e9dcc6191c42d3a5c1503dedc5511e P 501e3c0d44fe4d70af9ff926d9622417 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:45.989598 19548 raft_consensus.cc:374] T e0e69b88ad014363b97d1f650df693a8 P 82f50f4b66f3450aa9f58a0b317f5cea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:45.989634 19549 tablet_bootstrap.cc:492] T a7a9256f2b784abcb780b190e2ff3983 P c75831edb3b44dd59c00ef151adca49d: Bootstrap starting.
I20240502 21:25:45.989746 19548 raft_consensus.cc:729] T e0e69b88ad014363b97d1f650df693a8 P 82f50f4b66f3450aa9f58a0b317f5cea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82f50f4b66f3450aa9f58a0b317f5cea, State: Initialized, Role: FOLLOWER
I20240502 21:25:45.989781 19550 raft_consensus.cc:729] T 37e9dcc6191c42d3a5c1503dedc5511e P 501e3c0d44fe4d70af9ff926d9622417 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 501e3c0d44fe4d70af9ff926d9622417, State: Initialized, Role: FOLLOWER
I20240502 21:25:45.989375 19557 raft_consensus.cc:348] T e0e69b88ad014363b97d1f650df693a8 P 2877108756304e05bd60ba1186b870da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } }
I20240502 21:25:45.990047 19548 consensus_queue.cc:260] T e0e69b88ad014363b97d1f650df693a8 P 82f50f4b66f3450aa9f58a0b317f5cea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } }
I20240502 21:25:45.990170 19557 raft_consensus.cc:374] T e0e69b88ad014363b97d1f650df693a8 P 2877108756304e05bd60ba1186b870da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:45.990265 19557 raft_consensus.cc:729] T e0e69b88ad014363b97d1f650df693a8 P 2877108756304e05bd60ba1186b870da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2877108756304e05bd60ba1186b870da, State: Initialized, Role: FOLLOWER
I20240502 21:25:45.990545 19557 consensus_queue.cc:260] T e0e69b88ad014363b97d1f650df693a8 P 2877108756304e05bd60ba1186b870da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } }
I20240502 21:25:45.990720 19548 ts_tablet_manager.cc:1398] T e0e69b88ad014363b97d1f650df693a8 P 82f50f4b66f3450aa9f58a0b317f5cea: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 21:25:45.990762 19552 tablet_bootstrap.cc:492] T 37e9dcc6191c42d3a5c1503dedc5511e P 9be399440ded4e2ebe6b6e0bf486cbb1: No bootstrap required, opened a new log
I20240502 21:25:45.990971 19548 tablet_bootstrap.cc:492] T a7a9256f2b784abcb780b190e2ff3983 P 82f50f4b66f3450aa9f58a0b317f5cea: Bootstrap starting.
I20240502 21:25:45.990981 19552 ts_tablet_manager.cc:1367] T 37e9dcc6191c42d3a5c1503dedc5511e P 9be399440ded4e2ebe6b6e0bf486cbb1: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.005s
I20240502 21:25:45.992599 19548 tablet_bootstrap.cc:654] T a7a9256f2b784abcb780b190e2ff3983 P 82f50f4b66f3450aa9f58a0b317f5cea: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:45.993108 19552 raft_consensus.cc:348] T 37e9dcc6191c42d3a5c1503dedc5511e P 9be399440ded4e2ebe6b6e0bf486cbb1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } }
I20240502 21:25:45.993358 19552 raft_consensus.cc:374] T 37e9dcc6191c42d3a5c1503dedc5511e P 9be399440ded4e2ebe6b6e0bf486cbb1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:45.993465 19552 raft_consensus.cc:729] T 37e9dcc6191c42d3a5c1503dedc5511e P 9be399440ded4e2ebe6b6e0bf486cbb1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9be399440ded4e2ebe6b6e0bf486cbb1, State: Initialized, Role: FOLLOWER
I20240502 21:25:45.993499 19558 raft_consensus.cc:482] T 37e9dcc6191c42d3a5c1503dedc5511e P c75831edb3b44dd59c00ef151adca49d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:25:45.993714 19552 consensus_queue.cc:260] T 37e9dcc6191c42d3a5c1503dedc5511e P 9be399440ded4e2ebe6b6e0bf486cbb1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } }
I20240502 21:25:45.993762 19558 raft_consensus.cc:504] T 37e9dcc6191c42d3a5c1503dedc5511e P c75831edb3b44dd59c00ef151adca49d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } }
I20240502 21:25:45.994334 19549 tablet_bootstrap.cc:654] T a7a9256f2b784abcb780b190e2ff3983 P c75831edb3b44dd59c00ef151adca49d: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:45.994225 19550 consensus_queue.cc:260] T 37e9dcc6191c42d3a5c1503dedc5511e P 501e3c0d44fe4d70af9ff926d9622417 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } }
I20240502 21:25:45.995677 18611 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37e9dcc6191c42d3a5c1503dedc5511e" candidate_uuid: "c75831edb3b44dd59c00ef151adca49d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" is_pre_election: true
I20240502 21:25:45.995908 18611 raft_consensus.cc:2454] T 37e9dcc6191c42d3a5c1503dedc5511e P 82f50f4b66f3450aa9f58a0b317f5cea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c75831edb3b44dd59c00ef151adca49d in term 0.
I20240502 21:25:45.997124 18744 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37e9dcc6191c42d3a5c1503dedc5511e" candidate_uuid: "c75831edb3b44dd59c00ef151adca49d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "501e3c0d44fe4d70af9ff926d9622417" is_pre_election: true
I20240502 21:25:45.997355 18744 raft_consensus.cc:2454] T 37e9dcc6191c42d3a5c1503dedc5511e P 501e3c0d44fe4d70af9ff926d9622417 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c75831edb3b44dd59c00ef151adca49d in term 0.
I20240502 21:25:45.997766 18809 leader_election.cc:302] T 37e9dcc6191c42d3a5c1503dedc5511e P c75831edb3b44dd59c00ef151adca49d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 501e3c0d44fe4d70af9ff926d9622417, 82f50f4b66f3450aa9f58a0b317f5cea, c75831edb3b44dd59c00ef151adca49d; no voters:
I20240502 21:25:45.998106 19557 ts_tablet_manager.cc:1398] T e0e69b88ad014363b97d1f650df693a8 P 2877108756304e05bd60ba1186b870da: Time spent starting tablet: real 0.009s user 0.002s sys 0.005s
I20240502 21:25:45.998853 19557 tablet_bootstrap.cc:492] T 37e9dcc6191c42d3a5c1503dedc5511e P 2877108756304e05bd60ba1186b870da: Bootstrap starting.
I20240502 21:25:46.000484 19009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37e9dcc6191c42d3a5c1503dedc5511e" candidate_uuid: "c75831edb3b44dd59c00ef151adca49d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" is_pre_election: true
I20240502 21:25:46.000692 19009 raft_consensus.cc:2454] T 37e9dcc6191c42d3a5c1503dedc5511e P 9be399440ded4e2ebe6b6e0bf486cbb1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c75831edb3b44dd59c00ef151adca49d in term 0.
I20240502 21:25:46.001904 19146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37e9dcc6191c42d3a5c1503dedc5511e" candidate_uuid: "c75831edb3b44dd59c00ef151adca49d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2877108756304e05bd60ba1186b870da" is_pre_election: true
I20240502 21:25:46.002048 19557 tablet_bootstrap.cc:654] T 37e9dcc6191c42d3a5c1503dedc5511e P 2877108756304e05bd60ba1186b870da: Neither blocks nor log segments found. Creating new log.
W20240502 21:25:46.002633 18812 leader_election.cc:341] T 37e9dcc6191c42d3a5c1503dedc5511e P c75831edb3b44dd59c00ef151adca49d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2877108756304e05bd60ba1186b870da (127.15.255.134:43635): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:25:46.003324 19550 ts_tablet_manager.cc:1398] T 37e9dcc6191c42d3a5c1503dedc5511e P 501e3c0d44fe4d70af9ff926d9622417: Time spent starting tablet: real 0.015s user 0.005s sys 0.000s
I20240502 21:25:46.003679 19550 tablet_bootstrap.cc:492] T e0e69b88ad014363b97d1f650df693a8 P 501e3c0d44fe4d70af9ff926d9622417: Bootstrap starting.
I20240502 21:25:46.005293 19550 tablet_bootstrap.cc:654] T e0e69b88ad014363b97d1f650df693a8 P 501e3c0d44fe4d70af9ff926d9622417: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:46.007069 19557 tablet_bootstrap.cc:492] T 37e9dcc6191c42d3a5c1503dedc5511e P 2877108756304e05bd60ba1186b870da: No bootstrap required, opened a new log
I20240502 21:25:46.007279 19557 ts_tablet_manager.cc:1367] T 37e9dcc6191c42d3a5c1503dedc5511e P 2877108756304e05bd60ba1186b870da: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.006s
I20240502 21:25:46.003078 19558 leader_election.cc:288] T 37e9dcc6191c42d3a5c1503dedc5511e P c75831edb3b44dd59c00ef151adca49d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 82f50f4b66f3450aa9f58a0b317f5cea (127.15.255.130:45139), 501e3c0d44fe4d70af9ff926d9622417 (127.15.255.131:42501), 9be399440ded4e2ebe6b6e0bf486cbb1 (127.15.255.133:34217), 2877108756304e05bd60ba1186b870da (127.15.255.134:43635)
I20240502 21:25:46.007499 19558 raft_consensus.cc:2781] T 37e9dcc6191c42d3a5c1503dedc5511e P c75831edb3b44dd59c00ef151adca49d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:25:46.007620 19558 raft_consensus.cc:482] T 37e9dcc6191c42d3a5c1503dedc5511e P c75831edb3b44dd59c00ef151adca49d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:25:46.007714 19558 raft_consensus.cc:3037] T 37e9dcc6191c42d3a5c1503dedc5511e P c75831edb3b44dd59c00ef151adca49d [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:46.008201 19557 raft_consensus.cc:348] T 37e9dcc6191c42d3a5c1503dedc5511e P 2877108756304e05bd60ba1186b870da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } }
I20240502 21:25:46.008530 19557 raft_consensus.cc:374] T 37e9dcc6191c42d3a5c1503dedc5511e P 2877108756304e05bd60ba1186b870da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:46.008648 19557 raft_consensus.cc:729] T 37e9dcc6191c42d3a5c1503dedc5511e P 2877108756304e05bd60ba1186b870da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2877108756304e05bd60ba1186b870da, State: Initialized, Role: FOLLOWER
I20240502 21:25:46.008976 19557 consensus_queue.cc:260] T 37e9dcc6191c42d3a5c1503dedc5511e P 2877108756304e05bd60ba1186b870da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } }
I20240502 21:25:46.011525 19557 ts_tablet_manager.cc:1398] T 37e9dcc6191c42d3a5c1503dedc5511e P 2877108756304e05bd60ba1186b870da: Time spent starting tablet: real 0.004s user 0.000s sys 0.001s
I20240502 21:25:46.011539 19558 raft_consensus.cc:504] T 37e9dcc6191c42d3a5c1503dedc5511e P c75831edb3b44dd59c00ef151adca49d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } }
I20240502 21:25:46.013379 19550 tablet_bootstrap.cc:492] T e0e69b88ad014363b97d1f650df693a8 P 501e3c0d44fe4d70af9ff926d9622417: No bootstrap required, opened a new log
I20240502 21:25:46.013978 19550 ts_tablet_manager.cc:1367] T e0e69b88ad014363b97d1f650df693a8 P 501e3c0d44fe4d70af9ff926d9622417: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.000s
I20240502 21:25:46.014657 19558 leader_election.cc:288] T 37e9dcc6191c42d3a5c1503dedc5511e P c75831edb3b44dd59c00ef151adca49d [CANDIDATE]: Term 1 election: Requested vote from peers 82f50f4b66f3450aa9f58a0b317f5cea (127.15.255.130:45139), 501e3c0d44fe4d70af9ff926d9622417 (127.15.255.131:42501), 9be399440ded4e2ebe6b6e0bf486cbb1 (127.15.255.133:34217), 2877108756304e05bd60ba1186b870da (127.15.255.134:43635)
I20240502 21:25:46.015116 18611 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37e9dcc6191c42d3a5c1503dedc5511e" candidate_uuid: "c75831edb3b44dd59c00ef151adca49d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82f50f4b66f3450aa9f58a0b317f5cea"
I20240502 21:25:46.015329 18611 raft_consensus.cc:3037] T 37e9dcc6191c42d3a5c1503dedc5511e P 82f50f4b66f3450aa9f58a0b317f5cea [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:46.015424 19550 raft_consensus.cc:348] T e0e69b88ad014363b97d1f650df693a8 P 501e3c0d44fe4d70af9ff926d9622417 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } }
I20240502 21:25:46.015678 19550 raft_consensus.cc:374] T e0e69b88ad014363b97d1f650df693a8 P 501e3c0d44fe4d70af9ff926d9622417 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:46.015740 19550 raft_consensus.cc:729] T e0e69b88ad014363b97d1f650df693a8 P 501e3c0d44fe4d70af9ff926d9622417 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 501e3c0d44fe4d70af9ff926d9622417, State: Initialized, Role: FOLLOWER
I20240502 21:25:46.017310 19009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37e9dcc6191c42d3a5c1503dedc5511e" candidate_uuid: "c75831edb3b44dd59c00ef151adca49d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1"
I20240502 21:25:46.017482 19009 raft_consensus.cc:3037] T 37e9dcc6191c42d3a5c1503dedc5511e P 9be399440ded4e2ebe6b6e0bf486cbb1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:46.017633 19549 tablet_bootstrap.cc:492] T a7a9256f2b784abcb780b190e2ff3983 P c75831edb3b44dd59c00ef151adca49d: No bootstrap required, opened a new log
I20240502 21:25:46.017764 19549 ts_tablet_manager.cc:1367] T a7a9256f2b784abcb780b190e2ff3983 P c75831edb3b44dd59c00ef151adca49d: Time spent bootstrapping tablet: real 0.028s user 0.007s sys 0.015s
I20240502 21:25:46.018000 18611 raft_consensus.cc:2454] T 37e9dcc6191c42d3a5c1503dedc5511e P 82f50f4b66f3450aa9f58a0b317f5cea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c75831edb3b44dd59c00ef151adca49d in term 1.
I20240502 21:25:46.018460 19549 raft_consensus.cc:348] T a7a9256f2b784abcb780b190e2ff3983 P c75831edb3b44dd59c00ef151adca49d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } }
I20240502 21:25:46.018677 19552 ts_tablet_manager.cc:1398] T 37e9dcc6191c42d3a5c1503dedc5511e P 9be399440ded4e2ebe6b6e0bf486cbb1: Time spent starting tablet: real 0.028s user 0.002s sys 0.008s
I20240502 21:25:46.018678 19549 raft_consensus.cc:374] T a7a9256f2b784abcb780b190e2ff3983 P c75831edb3b44dd59c00ef151adca49d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:46.018761 19549 raft_consensus.cc:729] T a7a9256f2b784abcb780b190e2ff3983 P c75831edb3b44dd59c00ef151adca49d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c75831edb3b44dd59c00ef151adca49d, State: Initialized, Role: FOLLOWER
I20240502 21:25:46.019088 19552 tablet_bootstrap.cc:492] T a7a9256f2b784abcb780b190e2ff3983 P 9be399440ded4e2ebe6b6e0bf486cbb1: Bootstrap starting.
I20240502 21:25:46.018961 19549 consensus_queue.cc:260] T a7a9256f2b784abcb780b190e2ff3983 P c75831edb3b44dd59c00ef151adca49d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } }
I20240502 21:25:46.019485 19549 ts_tablet_manager.cc:1398] T a7a9256f2b784abcb780b190e2ff3983 P c75831edb3b44dd59c00ef151adca49d: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 21:25:46.019860 19009 raft_consensus.cc:2454] T 37e9dcc6191c42d3a5c1503dedc5511e P 9be399440ded4e2ebe6b6e0bf486cbb1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c75831edb3b44dd59c00ef151adca49d in term 1.
I20240502 21:25:46.020396 19552 tablet_bootstrap.cc:654] T a7a9256f2b784abcb780b190e2ff3983 P 9be399440ded4e2ebe6b6e0bf486cbb1: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:46.021312 19550 consensus_queue.cc:260] T e0e69b88ad014363b97d1f650df693a8 P 501e3c0d44fe4d70af9ff926d9622417 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } }
I20240502 21:25:46.021968 18810 leader_election.cc:302] T 37e9dcc6191c42d3a5c1503dedc5511e P c75831edb3b44dd59c00ef151adca49d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 82f50f4b66f3450aa9f58a0b317f5cea, 9be399440ded4e2ebe6b6e0bf486cbb1, c75831edb3b44dd59c00ef151adca49d; no voters:
I20240502 21:25:46.022219 19568 raft_consensus.cc:2781] T 37e9dcc6191c42d3a5c1503dedc5511e P c75831edb3b44dd59c00ef151adca49d [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:25:46.022270 19548 tablet_bootstrap.cc:492] T a7a9256f2b784abcb780b190e2ff3983 P 82f50f4b66f3450aa9f58a0b317f5cea: No bootstrap required, opened a new log
I20240502 21:25:46.022364 19568 raft_consensus.cc:686] T 37e9dcc6191c42d3a5c1503dedc5511e P c75831edb3b44dd59c00ef151adca49d [term 1 LEADER]: Becoming Leader. State: Replica: c75831edb3b44dd59c00ef151adca49d, State: Running, Role: LEADER
I20240502 21:25:46.022394 19548 ts_tablet_manager.cc:1367] T a7a9256f2b784abcb780b190e2ff3983 P 82f50f4b66f3450aa9f58a0b317f5cea: Time spent bootstrapping tablet: real 0.032s user 0.000s sys 0.004s
I20240502 21:25:46.022431 19146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37e9dcc6191c42d3a5c1503dedc5511e" candidate_uuid: "c75831edb3b44dd59c00ef151adca49d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2877108756304e05bd60ba1186b870da"
I20240502 21:25:46.022605 19146 raft_consensus.cc:3037] T 37e9dcc6191c42d3a5c1503dedc5511e P 2877108756304e05bd60ba1186b870da [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:46.022917 19548 raft_consensus.cc:348] T a7a9256f2b784abcb780b190e2ff3983 P 82f50f4b66f3450aa9f58a0b317f5cea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } }
I20240502 21:25:46.023058 19548 raft_consensus.cc:374] T a7a9256f2b784abcb780b190e2ff3983 P 82f50f4b66f3450aa9f58a0b317f5cea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:46.022898 19568 consensus_queue.cc:237] T 37e9dcc6191c42d3a5c1503dedc5511e P c75831edb3b44dd59c00ef151adca49d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } }
I20240502 21:25:46.023111 19548 raft_consensus.cc:729] T a7a9256f2b784abcb780b190e2ff3983 P 82f50f4b66f3450aa9f58a0b317f5cea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82f50f4b66f3450aa9f58a0b317f5cea, State: Initialized, Role: FOLLOWER
I20240502 21:25:46.023303 19548 consensus_queue.cc:260] T a7a9256f2b784abcb780b190e2ff3983 P 82f50f4b66f3450aa9f58a0b317f5cea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } }
I20240502 21:25:46.023734 19548 ts_tablet_manager.cc:1398] T a7a9256f2b784abcb780b190e2ff3983 P 82f50f4b66f3450aa9f58a0b317f5cea: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20240502 21:25:46.024752 19550 ts_tablet_manager.cc:1398] T e0e69b88ad014363b97d1f650df693a8 P 501e3c0d44fe4d70af9ff926d9622417: Time spent starting tablet: real 0.010s user 0.001s sys 0.000s
I20240502 21:25:46.025079 19550 tablet_bootstrap.cc:492] T a7a9256f2b784abcb780b190e2ff3983 P 501e3c0d44fe4d70af9ff926d9622417: Bootstrap starting.
I20240502 21:25:46.025127 19146 raft_consensus.cc:2454] T 37e9dcc6191c42d3a5c1503dedc5511e P 2877108756304e05bd60ba1186b870da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c75831edb3b44dd59c00ef151adca49d in term 1.
I20240502 21:25:46.025595 18744 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "37e9dcc6191c42d3a5c1503dedc5511e" candidate_uuid: "c75831edb3b44dd59c00ef151adca49d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "501e3c0d44fe4d70af9ff926d9622417"
I20240502 21:25:46.025796 18744 raft_consensus.cc:3037] T 37e9dcc6191c42d3a5c1503dedc5511e P 501e3c0d44fe4d70af9ff926d9622417 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:46.026480 19550 tablet_bootstrap.cc:654] T a7a9256f2b784abcb780b190e2ff3983 P 501e3c0d44fe4d70af9ff926d9622417: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:46.028048 19552 tablet_bootstrap.cc:492] T a7a9256f2b784abcb780b190e2ff3983 P 9be399440ded4e2ebe6b6e0bf486cbb1: No bootstrap required, opened a new log
I20240502 21:25:46.028188 19552 ts_tablet_manager.cc:1367] T a7a9256f2b784abcb780b190e2ff3983 P 9be399440ded4e2ebe6b6e0bf486cbb1: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240502 21:25:46.028558 18744 raft_consensus.cc:2454] T 37e9dcc6191c42d3a5c1503dedc5511e P 501e3c0d44fe4d70af9ff926d9622417 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c75831edb3b44dd59c00ef151adca49d in term 1.
I20240502 21:25:46.028961 19552 raft_consensus.cc:348] T a7a9256f2b784abcb780b190e2ff3983 P 9be399440ded4e2ebe6b6e0bf486cbb1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } }
I20240502 21:25:46.029275 19552 raft_consensus.cc:374] T a7a9256f2b784abcb780b190e2ff3983 P 9be399440ded4e2ebe6b6e0bf486cbb1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:46.029381 19552 raft_consensus.cc:729] T a7a9256f2b784abcb780b190e2ff3983 P 9be399440ded4e2ebe6b6e0bf486cbb1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9be399440ded4e2ebe6b6e0bf486cbb1, State: Initialized, Role: FOLLOWER
I20240502 21:25:46.029685 19552 consensus_queue.cc:260] T a7a9256f2b784abcb780b190e2ff3983 P 9be399440ded4e2ebe6b6e0bf486cbb1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } }
I20240502 21:25:46.030257 19552 ts_tablet_manager.cc:1398] T a7a9256f2b784abcb780b190e2ff3983 P 9be399440ded4e2ebe6b6e0bf486cbb1: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:25:46.037273 19552 tablet_bootstrap.cc:492] T e0e69b88ad014363b97d1f650df693a8 P 9be399440ded4e2ebe6b6e0bf486cbb1: Bootstrap starting.
I20240502 21:25:46.038789 18320 catalog_manager.cc:5484] T 37e9dcc6191c42d3a5c1503dedc5511e P c75831edb3b44dd59c00ef151adca49d reported cstate change: term changed from 0 to 1, leader changed from <none> to c75831edb3b44dd59c00ef151adca49d (127.15.255.132). New cstate: current_term: 1 leader_uuid: "c75831edb3b44dd59c00ef151adca49d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } health_report { overall_health: UNKNOWN } } }
I20240502 21:25:46.039278 19550 tablet_bootstrap.cc:492] T a7a9256f2b784abcb780b190e2ff3983 P 501e3c0d44fe4d70af9ff926d9622417: No bootstrap required, opened a new log
I20240502 21:25:46.039367 19552 tablet_bootstrap.cc:654] T e0e69b88ad014363b97d1f650df693a8 P 9be399440ded4e2ebe6b6e0bf486cbb1: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:46.039425 19550 ts_tablet_manager.cc:1367] T a7a9256f2b784abcb780b190e2ff3983 P 501e3c0d44fe4d70af9ff926d9622417: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.009s
I20240502 21:25:46.040148 19550 raft_consensus.cc:348] T a7a9256f2b784abcb780b190e2ff3983 P 501e3c0d44fe4d70af9ff926d9622417 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } }
I20240502 21:25:46.040380 19550 raft_consensus.cc:374] T a7a9256f2b784abcb780b190e2ff3983 P 501e3c0d44fe4d70af9ff926d9622417 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:46.040516 19550 raft_consensus.cc:729] T a7a9256f2b784abcb780b190e2ff3983 P 501e3c0d44fe4d70af9ff926d9622417 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 501e3c0d44fe4d70af9ff926d9622417, State: Initialized, Role: FOLLOWER
I20240502 21:25:46.040760 19550 consensus_queue.cc:260] T a7a9256f2b784abcb780b190e2ff3983 P 501e3c0d44fe4d70af9ff926d9622417 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } }
I20240502 21:25:46.041842 19550 ts_tablet_manager.cc:1398] T a7a9256f2b784abcb780b190e2ff3983 P 501e3c0d44fe4d70af9ff926d9622417: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:25:46.043695 19552 tablet_bootstrap.cc:492] T e0e69b88ad014363b97d1f650df693a8 P 9be399440ded4e2ebe6b6e0bf486cbb1: No bootstrap required, opened a new log
I20240502 21:25:46.043833 19552 ts_tablet_manager.cc:1367] T e0e69b88ad014363b97d1f650df693a8 P 9be399440ded4e2ebe6b6e0bf486cbb1: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20240502 21:25:46.044412 19552 raft_consensus.cc:348] T e0e69b88ad014363b97d1f650df693a8 P 9be399440ded4e2ebe6b6e0bf486cbb1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } }
I20240502 21:25:46.044612 19552 raft_consensus.cc:374] T e0e69b88ad014363b97d1f650df693a8 P 9be399440ded4e2ebe6b6e0bf486cbb1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:46.044697 19552 raft_consensus.cc:729] T e0e69b88ad014363b97d1f650df693a8 P 9be399440ded4e2ebe6b6e0bf486cbb1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9be399440ded4e2ebe6b6e0bf486cbb1, State: Initialized, Role: FOLLOWER
I20240502 21:25:46.044898 19552 consensus_queue.cc:260] T e0e69b88ad014363b97d1f650df693a8 P 9be399440ded4e2ebe6b6e0bf486cbb1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } }
I20240502 21:25:46.045364 19552 ts_tablet_manager.cc:1398] T e0e69b88ad014363b97d1f650df693a8 P 9be399440ded4e2ebe6b6e0bf486cbb1: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 21:25:46.113327 19555 raft_consensus.cc:482] T a7a9256f2b784abcb780b190e2ff3983 P 82f50f4b66f3450aa9f58a0b317f5cea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:25:46.113523 19555 raft_consensus.cc:504] T a7a9256f2b784abcb780b190e2ff3983 P 82f50f4b66f3450aa9f58a0b317f5cea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } }
I20240502 21:25:46.115312 19555 leader_election.cc:288] T a7a9256f2b784abcb780b190e2ff3983 P 82f50f4b66f3450aa9f58a0b317f5cea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 501e3c0d44fe4d70af9ff926d9622417 (127.15.255.131:42501), c75831edb3b44dd59c00ef151adca49d (127.15.255.132:43607), 9be399440ded4e2ebe6b6e0bf486cbb1 (127.15.255.133:34217), faadabfb099b4abf90a466395be585b5 (127.15.255.129:32983)
I20240502 21:25:46.134239 19565 raft_consensus.cc:482] T e0e69b88ad014363b97d1f650df693a8 P 9be399440ded4e2ebe6b6e0bf486cbb1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:25:46.134416 19565 raft_consensus.cc:504] T e0e69b88ad014363b97d1f650df693a8 P 9be399440ded4e2ebe6b6e0bf486cbb1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } }
I20240502 21:25:46.135524 19565 leader_election.cc:288] T e0e69b88ad014363b97d1f650df693a8 P 9be399440ded4e2ebe6b6e0bf486cbb1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 82f50f4b66f3450aa9f58a0b317f5cea (127.15.255.130:45139), 501e3c0d44fe4d70af9ff926d9622417 (127.15.255.131:42501), c75831edb3b44dd59c00ef151adca49d (127.15.255.132:43607), 2877108756304e05bd60ba1186b870da (127.15.255.134:43635)
I20240502 21:25:46.135716 18877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7a9256f2b784abcb780b190e2ff3983" candidate_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c75831edb3b44dd59c00ef151adca49d" is_pre_election: true
I20240502 21:25:46.135958 18877 raft_consensus.cc:2454] T a7a9256f2b784abcb780b190e2ff3983 P c75831edb3b44dd59c00ef151adca49d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82f50f4b66f3450aa9f58a0b317f5cea in term 0.
I20240502 21:25:46.136770 18744 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0e69b88ad014363b97d1f650df693a8" candidate_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "501e3c0d44fe4d70af9ff926d9622417" is_pre_election: true
I20240502 21:25:46.137029 18744 raft_consensus.cc:2454] T e0e69b88ad014363b97d1f650df693a8 P 501e3c0d44fe4d70af9ff926d9622417 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9be399440ded4e2ebe6b6e0bf486cbb1 in term 0.
I20240502 21:25:46.137177 18743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7a9256f2b784abcb780b190e2ff3983" candidate_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "501e3c0d44fe4d70af9ff926d9622417" is_pre_election: true
I20240502 21:25:46.137380 18877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0e69b88ad014363b97d1f650df693a8" candidate_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c75831edb3b44dd59c00ef151adca49d" is_pre_election: true
I20240502 21:25:46.137428 18743 raft_consensus.cc:2454] T a7a9256f2b784abcb780b190e2ff3983 P 501e3c0d44fe4d70af9ff926d9622417 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82f50f4b66f3450aa9f58a0b317f5cea in term 0.
I20240502 21:25:46.137566 18877 raft_consensus.cc:2454] T e0e69b88ad014363b97d1f650df693a8 P c75831edb3b44dd59c00ef151adca49d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9be399440ded4e2ebe6b6e0bf486cbb1 in term 0.
I20240502 21:25:46.137986 18945 leader_election.cc:302] T e0e69b88ad014363b97d1f650df693a8 P 9be399440ded4e2ebe6b6e0bf486cbb1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 501e3c0d44fe4d70af9ff926d9622417, 9be399440ded4e2ebe6b6e0bf486cbb1, c75831edb3b44dd59c00ef151adca49d; no voters:
I20240502 21:25:46.138327 19565 raft_consensus.cc:2781] T e0e69b88ad014363b97d1f650df693a8 P 9be399440ded4e2ebe6b6e0bf486cbb1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:25:46.138443 19565 raft_consensus.cc:482] T e0e69b88ad014363b97d1f650df693a8 P 9be399440ded4e2ebe6b6e0bf486cbb1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:25:46.138527 19565 raft_consensus.cc:3037] T e0e69b88ad014363b97d1f650df693a8 P 9be399440ded4e2ebe6b6e0bf486cbb1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:46.141922 19565 raft_consensus.cc:504] T e0e69b88ad014363b97d1f650df693a8 P 9be399440ded4e2ebe6b6e0bf486cbb1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } }
I20240502 21:25:46.143298 18877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0e69b88ad014363b97d1f650df693a8" candidate_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c75831edb3b44dd59c00ef151adca49d"
I20240502 21:25:46.143324 18743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0e69b88ad014363b97d1f650df693a8" candidate_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "501e3c0d44fe4d70af9ff926d9622417"
I20240502 21:25:46.143476 18877 raft_consensus.cc:3037] T e0e69b88ad014363b97d1f650df693a8 P c75831edb3b44dd59c00ef151adca49d [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:46.143496 18743 raft_consensus.cc:3037] T e0e69b88ad014363b97d1f650df693a8 P 501e3c0d44fe4d70af9ff926d9622417 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:46.146386 18877 raft_consensus.cc:2454] T e0e69b88ad014363b97d1f650df693a8 P c75831edb3b44dd59c00ef151adca49d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9be399440ded4e2ebe6b6e0bf486cbb1 in term 1.
I20240502 21:25:46.146508 18743 raft_consensus.cc:2454] T e0e69b88ad014363b97d1f650df693a8 P 501e3c0d44fe4d70af9ff926d9622417 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9be399440ded4e2ebe6b6e0bf486cbb1 in term 1.
I20240502 21:25:46.146957 18943 leader_election.cc:302] T e0e69b88ad014363b97d1f650df693a8 P 9be399440ded4e2ebe6b6e0bf486cbb1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 501e3c0d44fe4d70af9ff926d9622417, 9be399440ded4e2ebe6b6e0bf486cbb1, c75831edb3b44dd59c00ef151adca49d; no voters:
I20240502 21:25:46.147735 19565 leader_election.cc:288] T e0e69b88ad014363b97d1f650df693a8 P 9be399440ded4e2ebe6b6e0bf486cbb1 [CANDIDATE]: Term 1 election: Requested vote from peers 82f50f4b66f3450aa9f58a0b317f5cea (127.15.255.130:45139), 501e3c0d44fe4d70af9ff926d9622417 (127.15.255.131:42501), c75831edb3b44dd59c00ef151adca49d (127.15.255.132:43607), 2877108756304e05bd60ba1186b870da (127.15.255.134:43635)
I20240502 21:25:46.147728 19585 raft_consensus.cc:2781] T e0e69b88ad014363b97d1f650df693a8 P 9be399440ded4e2ebe6b6e0bf486cbb1 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:25:46.148329 19585 raft_consensus.cc:686] T e0e69b88ad014363b97d1f650df693a8 P 9be399440ded4e2ebe6b6e0bf486cbb1 [term 1 LEADER]: Becoming Leader. State: Replica: 9be399440ded4e2ebe6b6e0bf486cbb1, State: Running, Role: LEADER
I20240502 21:25:46.148548 19146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0e69b88ad014363b97d1f650df693a8" candidate_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2877108756304e05bd60ba1186b870da" is_pre_election: true
I20240502 21:25:46.148813 19146 raft_consensus.cc:2454] T e0e69b88ad014363b97d1f650df693a8 P 2877108756304e05bd60ba1186b870da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9be399440ded4e2ebe6b6e0bf486cbb1 in term 0.
I20240502 21:25:46.148993 19585 consensus_queue.cc:237] T e0e69b88ad014363b97d1f650df693a8 P 9be399440ded4e2ebe6b6e0bf486cbb1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } }
I20240502 21:25:46.149832 18543 leader_election.cc:302] T a7a9256f2b784abcb780b190e2ff3983 P 82f50f4b66f3450aa9f58a0b317f5cea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 501e3c0d44fe4d70af9ff926d9622417, 82f50f4b66f3450aa9f58a0b317f5cea, c75831edb3b44dd59c00ef151adca49d; no voters:
I20240502 21:25:46.149933 19146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0e69b88ad014363b97d1f650df693a8" candidate_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2877108756304e05bd60ba1186b870da"
I20240502 21:25:46.150094 19146 raft_consensus.cc:3037] T e0e69b88ad014363b97d1f650df693a8 P 2877108756304e05bd60ba1186b870da [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:46.150278 19555 raft_consensus.cc:2781] T a7a9256f2b784abcb780b190e2ff3983 P 82f50f4b66f3450aa9f58a0b317f5cea [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:25:46.150384 19555 raft_consensus.cc:482] T a7a9256f2b784abcb780b190e2ff3983 P 82f50f4b66f3450aa9f58a0b317f5cea [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:25:46.150470 19555 raft_consensus.cc:3037] T a7a9256f2b784abcb780b190e2ff3983 P 82f50f4b66f3450aa9f58a0b317f5cea [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:46.150810 19009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7a9256f2b784abcb780b190e2ff3983" candidate_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" is_pre_election: true
I20240502 21:25:46.151032 19009 raft_consensus.cc:2454] T a7a9256f2b784abcb780b190e2ff3983 P 9be399440ded4e2ebe6b6e0bf486cbb1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82f50f4b66f3450aa9f58a0b317f5cea in term 0.
I20240502 21:25:46.153038 19146 raft_consensus.cc:2454] T e0e69b88ad014363b97d1f650df693a8 P 2877108756304e05bd60ba1186b870da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9be399440ded4e2ebe6b6e0bf486cbb1 in term 1.
I20240502 21:25:46.153324 19555 raft_consensus.cc:504] T a7a9256f2b784abcb780b190e2ff3983 P 82f50f4b66f3450aa9f58a0b317f5cea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } }
I20240502 21:25:46.154489 18877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7a9256f2b784abcb780b190e2ff3983" candidate_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c75831edb3b44dd59c00ef151adca49d"
I20240502 21:25:46.154675 18877 raft_consensus.cc:3037] T a7a9256f2b784abcb780b190e2ff3983 P c75831edb3b44dd59c00ef151adca49d [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:46.157085 18877 raft_consensus.cc:2454] T a7a9256f2b784abcb780b190e2ff3983 P c75831edb3b44dd59c00ef151adca49d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82f50f4b66f3450aa9f58a0b317f5cea in term 1.
I20240502 21:25:46.157367 18324 catalog_manager.cc:5484] T e0e69b88ad014363b97d1f650df693a8 P 9be399440ded4e2ebe6b6e0bf486cbb1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9be399440ded4e2ebe6b6e0bf486cbb1 (127.15.255.133). New cstate: current_term: 1 leader_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } health_report { overall_health: UNKNOWN } } }
I20240502 21:25:46.157929 19009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7a9256f2b784abcb780b190e2ff3983" candidate_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1"
I20240502 21:25:46.158114 19009 raft_consensus.cc:3037] T a7a9256f2b784abcb780b190e2ff3983 P 9be399440ded4e2ebe6b6e0bf486cbb1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:46.158154 19555 leader_election.cc:288] T a7a9256f2b784abcb780b190e2ff3983 P 82f50f4b66f3450aa9f58a0b317f5cea [CANDIDATE]: Term 1 election: Requested vote from peers 501e3c0d44fe4d70af9ff926d9622417 (127.15.255.131:42501), c75831edb3b44dd59c00ef151adca49d (127.15.255.132:43607), 9be399440ded4e2ebe6b6e0bf486cbb1 (127.15.255.133:34217), faadabfb099b4abf90a466395be585b5 (127.15.255.129:32983)
I20240502 21:25:46.158215 18611 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0e69b88ad014363b97d1f650df693a8" candidate_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" is_pre_election: true
I20240502 21:25:46.158412 18611 raft_consensus.cc:2454] T e0e69b88ad014363b97d1f650df693a8 P 82f50f4b66f3450aa9f58a0b317f5cea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9be399440ded4e2ebe6b6e0bf486cbb1 in term 0.
I20240502 21:25:46.158458 18743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7a9256f2b784abcb780b190e2ff3983" candidate_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "501e3c0d44fe4d70af9ff926d9622417"
I20240502 21:25:46.158643 18743 raft_consensus.cc:3037] T a7a9256f2b784abcb780b190e2ff3983 P 501e3c0d44fe4d70af9ff926d9622417 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:46.158931 18611 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0e69b88ad014363b97d1f650df693a8" candidate_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82f50f4b66f3450aa9f58a0b317f5cea"
I20240502 21:25:46.159101 18611 raft_consensus.cc:3037] T e0e69b88ad014363b97d1f650df693a8 P 82f50f4b66f3450aa9f58a0b317f5cea [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:46.160691 19009 raft_consensus.cc:2454] T a7a9256f2b784abcb780b190e2ff3983 P 9be399440ded4e2ebe6b6e0bf486cbb1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82f50f4b66f3450aa9f58a0b317f5cea in term 1.
I20240502 21:25:46.161203 18743 raft_consensus.cc:2454] T a7a9256f2b784abcb780b190e2ff3983 P 501e3c0d44fe4d70af9ff926d9622417 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82f50f4b66f3450aa9f58a0b317f5cea in term 1.
I20240502 21:25:46.161444 18474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7a9256f2b784abcb780b190e2ff3983" candidate_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "faadabfb099b4abf90a466395be585b5" is_pre_election: true
I20240502 21:25:46.161563 18611 raft_consensus.cc:2454] T e0e69b88ad014363b97d1f650df693a8 P 82f50f4b66f3450aa9f58a0b317f5cea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9be399440ded4e2ebe6b6e0bf486cbb1 in term 1.
I20240502 21:25:46.161535 18473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7a9256f2b784abcb780b190e2ff3983" candidate_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "faadabfb099b4abf90a466395be585b5"
I20240502 21:25:46.161710 18474 raft_consensus.cc:2454] T a7a9256f2b784abcb780b190e2ff3983 P faadabfb099b4abf90a466395be585b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82f50f4b66f3450aa9f58a0b317f5cea in term 0.
I20240502 21:25:46.162014 18544 leader_election.cc:302] T a7a9256f2b784abcb780b190e2ff3983 P 82f50f4b66f3450aa9f58a0b317f5cea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 82f50f4b66f3450aa9f58a0b317f5cea, 9be399440ded4e2ebe6b6e0bf486cbb1, c75831edb3b44dd59c00ef151adca49d; no voters:
I20240502 21:25:46.162331 19555 raft_consensus.cc:2781] T a7a9256f2b784abcb780b190e2ff3983 P 82f50f4b66f3450aa9f58a0b317f5cea [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:25:46.162743 19555 raft_consensus.cc:686] T a7a9256f2b784abcb780b190e2ff3983 P 82f50f4b66f3450aa9f58a0b317f5cea [term 1 LEADER]: Becoming Leader. State: Replica: 82f50f4b66f3450aa9f58a0b317f5cea, State: Running, Role: LEADER
I20240502 21:25:46.163178 19555 consensus_queue.cc:237] T a7a9256f2b784abcb780b190e2ff3983 P 82f50f4b66f3450aa9f58a0b317f5cea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } }
I20240502 21:25:46.166605 18324 catalog_manager.cc:5484] T a7a9256f2b784abcb780b190e2ff3983 P 82f50f4b66f3450aa9f58a0b317f5cea reported cstate change: term changed from 0 to 1, leader changed from <none> to 82f50f4b66f3450aa9f58a0b317f5cea (127.15.255.130). New cstate: current_term: 1 leader_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } health_report { overall_health: HEALTHY } } }
W20240502 21:25:46.491276 19589 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:25:46.491564 19589 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:25:46.499100 19589 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 21:25:46.527953 19568 consensus_queue.cc:1035] T 37e9dcc6191c42d3a5c1503dedc5511e P c75831edb3b44dd59c00ef151adca49d [LEADER]: Connected to new peer: Peer: permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_1'
I20240502 21:25:46.532734 19585 consensus_queue.cc:1035] T e0e69b88ad014363b97d1f650df693a8 P 9be399440ded4e2ebe6b6e0bf486cbb1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:25:46.579783 19568 consensus_queue.cc:1035] T 37e9dcc6191c42d3a5c1503dedc5511e P c75831edb3b44dd59c00ef151adca49d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:25:46.590371 19568 consensus_queue.cc:1035] T 37e9dcc6191c42d3a5c1503dedc5511e P c75831edb3b44dd59c00ef151adca49d [LEADER]: Connected to new peer: Peer: permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:25:46.595686 19568 consensus_queue.cc:1035] T 37e9dcc6191c42d3a5c1503dedc5511e P c75831edb3b44dd59c00ef151adca49d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:25:46.604198 18474 raft_consensus.cc:3037] T a7a9256f2b784abcb780b190e2ff3983 P faadabfb099b4abf90a466395be585b5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:46.608278 19555 consensus_queue.cc:1035] T a7a9256f2b784abcb780b190e2ff3983 P 82f50f4b66f3450aa9f58a0b317f5cea [LEADER]: Connected to new peer: Peer: permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:25:46.616309 19585 consensus_queue.cc:1035] T e0e69b88ad014363b97d1f650df693a8 P 9be399440ded4e2ebe6b6e0bf486cbb1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:25:46.641999 19555 consensus_queue.cc:1035] T a7a9256f2b784abcb780b190e2ff3983 P 82f50f4b66f3450aa9f58a0b317f5cea [LEADER]: Connected to new peer: Peer: permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:25:46.660459 19555 consensus_queue.cc:1035] T a7a9256f2b784abcb780b190e2ff3983 P 82f50f4b66f3450aa9f58a0b317f5cea [LEADER]: Connected to new peer: Peer: permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 21:25:46.664739 19614 outbound_call.cc:322] RPC callback for RPC call kudu.master.MasterService.GetTableLocations -> {remote=127.15.255.190:36947, user_credentials={real_user=slave}} blocked reactor thread for 55920.2us
W20240502 21:25:46.689824 18810 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.133:34217, user_credentials={real_user=slave}} blocked reactor thread for 69791.3us
W20240502 21:25:46.737155 18544 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.133:34217, user_credentials={real_user=slave}} blocked reactor thread for 54717.2us
I20240502 21:25:46.737829 19641 consensus_queue.cc:1035] T a7a9256f2b784abcb780b190e2ff3983 P 82f50f4b66f3450aa9f58a0b317f5cea [LEADER]: Connected to new peer: Peer: permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:25:46.740738 19146 raft_consensus.cc:1261] T e0e69b88ad014363b97d1f650df693a8 P 2877108756304e05bd60ba1186b870da [term 1 FOLLOWER]: Refusing update from remote peer 9be399440ded4e2ebe6b6e0bf486cbb1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:25:46.760628 19585 consensus_queue.cc:1035] T e0e69b88ad014363b97d1f650df693a8 P 9be399440ded4e2ebe6b6e0bf486cbb1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:25:46.764447 18878 raft_consensus.cc:1261] T e0e69b88ad014363b97d1f650df693a8 P c75831edb3b44dd59c00ef151adca49d [term 1 FOLLOWER]: Refusing update from remote peer 9be399440ded4e2ebe6b6e0bf486cbb1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:25:46.773128 19585 consensus_queue.cc:1035] T e0e69b88ad014363b97d1f650df693a8 P 9be399440ded4e2ebe6b6e0bf486cbb1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:25:46.804265 19663 mvcc.cc:205] Tried to move back new op lower bound from 7023350360938516480 to 7023350358640537600. Current Snapshot: MvccSnapshot[applied={T|T < 7023350360938516480}]
W20240502 21:25:46.924679 18406 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.135:43395, user_credentials={real_user=slave}} blocked reactor thread for 102598us
INSERT report
rows total: 2000
time total: 429.429 ms
time per row: 0.214715 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240502 21:25:47.940517 18329 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60616:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 5
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 21:25:47.949556 18589 tablet_service.cc:1467] Processing CreateTablet for tablet 2d111096a9af46068a2476a6231d6520 (DEFAULT_TABLE table=rebalance_test_table_2 [id=59b7e3f7d6ce4a008110576d81d3cf39]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:47.950040 18589 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d111096a9af46068a2476a6231d6520. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:47.950101 19123 tablet_service.cc:1467] Processing CreateTablet for tablet 2d111096a9af46068a2476a6231d6520 (DEFAULT_TABLE table=rebalance_test_table_2 [id=59b7e3f7d6ce4a008110576d81d3cf39]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:47.950500 19123 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d111096a9af46068a2476a6231d6520. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:47.950452 18857 tablet_service.cc:1467] Processing CreateTablet for tablet 2d111096a9af46068a2476a6231d6520 (DEFAULT_TABLE table=rebalance_test_table_2 [id=59b7e3f7d6ce4a008110576d81d3cf39]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:47.950896 18857 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d111096a9af46068a2476a6231d6520. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:47.952692 18855 tablet_service.cc:1467] Processing CreateTablet for tablet 51edafdf2a9348548a494119668bfd31 (DEFAULT_TABLE table=rebalance_test_table_2 [id=59b7e3f7d6ce4a008110576d81d3cf39]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:47.953156 18855 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51edafdf2a9348548a494119668bfd31. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:47.956692 18722 tablet_service.cc:1467] Processing CreateTablet for tablet 2d111096a9af46068a2476a6231d6520 (DEFAULT_TABLE table=rebalance_test_table_2 [id=59b7e3f7d6ce4a008110576d81d3cf39]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:47.957137 18722 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d111096a9af46068a2476a6231d6520. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:47.957159 18588 tablet_service.cc:1467] Processing CreateTablet for tablet 693a8666bbe54ae1b774f7f4da5ba67e (DEFAULT_TABLE table=rebalance_test_table_2 [id=59b7e3f7d6ce4a008110576d81d3cf39]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:47.957345 18723 tablet_service.cc:1467] Processing CreateTablet for tablet 693a8666bbe54ae1b774f7f4da5ba67e (DEFAULT_TABLE table=rebalance_test_table_2 [id=59b7e3f7d6ce4a008110576d81d3cf39]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:47.957551 18588 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 693a8666bbe54ae1b774f7f4da5ba67e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:47.957665 18723 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 693a8666bbe54ae1b774f7f4da5ba67e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:47.960214 19689 tablet_bootstrap.cc:492] T 2d111096a9af46068a2476a6231d6520 P 2877108756304e05bd60ba1186b870da: Bootstrap starting.
I20240502 21:25:47.960788 18590 tablet_service.cc:1467] Processing CreateTablet for tablet 51edafdf2a9348548a494119668bfd31 (DEFAULT_TABLE table=rebalance_test_table_2 [id=59b7e3f7d6ce4a008110576d81d3cf39]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:47.961134 18590 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51edafdf2a9348548a494119668bfd31. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:47.961791 19689 tablet_bootstrap.cc:654] T 2d111096a9af46068a2476a6231d6520 P 2877108756304e05bd60ba1186b870da: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:47.965287 18452 tablet_service.cc:1467] Processing CreateTablet for tablet 2d111096a9af46068a2476a6231d6520 (DEFAULT_TABLE table=rebalance_test_table_2 [id=59b7e3f7d6ce4a008110576d81d3cf39]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:47.965732 18452 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d111096a9af46068a2476a6231d6520. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:47.968783 18453 tablet_service.cc:1467] Processing CreateTablet for tablet 693a8666bbe54ae1b774f7f4da5ba67e (DEFAULT_TABLE table=rebalance_test_table_2 [id=59b7e3f7d6ce4a008110576d81d3cf39]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:47.969149 18453 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 693a8666bbe54ae1b774f7f4da5ba67e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:47.970379 19688 tablet_bootstrap.cc:492] T 2d111096a9af46068a2476a6231d6520 P 82f50f4b66f3450aa9f58a0b317f5cea: Bootstrap starting.
I20240502 21:25:47.971873 19688 tablet_bootstrap.cc:654] T 2d111096a9af46068a2476a6231d6520 P 82f50f4b66f3450aa9f58a0b317f5cea: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:47.972764 18451 tablet_service.cc:1467] Processing CreateTablet for tablet 51edafdf2a9348548a494119668bfd31 (DEFAULT_TABLE table=rebalance_test_table_2 [id=59b7e3f7d6ce4a008110576d81d3cf39]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:47.973104 18451 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51edafdf2a9348548a494119668bfd31. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:47.974470 19692 tablet_bootstrap.cc:492] T 2d111096a9af46068a2476a6231d6520 P c75831edb3b44dd59c00ef151adca49d: Bootstrap starting.
I20240502 21:25:47.976325 19689 tablet_bootstrap.cc:492] T 2d111096a9af46068a2476a6231d6520 P 2877108756304e05bd60ba1186b870da: No bootstrap required, opened a new log
I20240502 21:25:47.976491 19689 ts_tablet_manager.cc:1367] T 2d111096a9af46068a2476a6231d6520 P 2877108756304e05bd60ba1186b870da: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.003s
I20240502 21:25:47.976809 18989 tablet_service.cc:1467] Processing CreateTablet for tablet 693a8666bbe54ae1b774f7f4da5ba67e (DEFAULT_TABLE table=rebalance_test_table_2 [id=59b7e3f7d6ce4a008110576d81d3cf39]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:47.977191 18989 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 693a8666bbe54ae1b774f7f4da5ba67e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:47.977202 19689 raft_consensus.cc:348] T 2d111096a9af46068a2476a6231d6520 P 2877108756304e05bd60ba1186b870da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } }
I20240502 21:25:47.977423 19689 raft_consensus.cc:374] T 2d111096a9af46068a2476a6231d6520 P 2877108756304e05bd60ba1186b870da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:47.977507 19689 raft_consensus.cc:729] T 2d111096a9af46068a2476a6231d6520 P 2877108756304e05bd60ba1186b870da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2877108756304e05bd60ba1186b870da, State: Initialized, Role: FOLLOWER
I20240502 21:25:47.977747 19689 consensus_queue.cc:260] T 2d111096a9af46068a2476a6231d6520 P 2877108756304e05bd60ba1186b870da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } }
I20240502 21:25:47.978305 19692 tablet_bootstrap.cc:654] T 2d111096a9af46068a2476a6231d6520 P c75831edb3b44dd59c00ef151adca49d: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:47.978828 19688 tablet_bootstrap.cc:492] T 2d111096a9af46068a2476a6231d6520 P 82f50f4b66f3450aa9f58a0b317f5cea: No bootstrap required, opened a new log
I20240502 21:25:47.978968 19688 ts_tablet_manager.cc:1367] T 2d111096a9af46068a2476a6231d6520 P 82f50f4b66f3450aa9f58a0b317f5cea: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s
I20240502 21:25:47.979631 19688 raft_consensus.cc:348] T 2d111096a9af46068a2476a6231d6520 P 82f50f4b66f3450aa9f58a0b317f5cea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } }
I20240502 21:25:47.979844 19688 raft_consensus.cc:374] T 2d111096a9af46068a2476a6231d6520 P 82f50f4b66f3450aa9f58a0b317f5cea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:47.979921 19688 raft_consensus.cc:729] T 2d111096a9af46068a2476a6231d6520 P 82f50f4b66f3450aa9f58a0b317f5cea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82f50f4b66f3450aa9f58a0b317f5cea, State: Initialized, Role: FOLLOWER
I20240502 21:25:47.980154 19688 consensus_queue.cc:260] T 2d111096a9af46068a2476a6231d6520 P 82f50f4b66f3450aa9f58a0b317f5cea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } }
I20240502 21:25:47.980763 18990 tablet_service.cc:1467] Processing CreateTablet for tablet 51edafdf2a9348548a494119668bfd31 (DEFAULT_TABLE table=rebalance_test_table_2 [id=59b7e3f7d6ce4a008110576d81d3cf39]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:47.980857 19688 ts_tablet_manager.cc:1398] T 2d111096a9af46068a2476a6231d6520 P 82f50f4b66f3450aa9f58a0b317f5cea: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:25:47.981105 18990 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51edafdf2a9348548a494119668bfd31. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:47.986562 19125 tablet_service.cc:1467] Processing CreateTablet for tablet 693a8666bbe54ae1b774f7f4da5ba67e (DEFAULT_TABLE table=rebalance_test_table_2 [id=59b7e3f7d6ce4a008110576d81d3cf39]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:47.986979 19125 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 693a8666bbe54ae1b774f7f4da5ba67e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:47.990376 19124 tablet_service.cc:1467] Processing CreateTablet for tablet 51edafdf2a9348548a494119668bfd31 (DEFAULT_TABLE table=rebalance_test_table_2 [id=59b7e3f7d6ce4a008110576d81d3cf39]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:47.990721 19124 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51edafdf2a9348548a494119668bfd31. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:47.991469 19692 tablet_bootstrap.cc:492] T 2d111096a9af46068a2476a6231d6520 P c75831edb3b44dd59c00ef151adca49d: No bootstrap required, opened a new log
I20240502 21:25:47.991621 19692 ts_tablet_manager.cc:1367] T 2d111096a9af46068a2476a6231d6520 P c75831edb3b44dd59c00ef151adca49d: Time spent bootstrapping tablet: real 0.017s user 0.000s sys 0.009s
I20240502 21:25:47.992370 19692 raft_consensus.cc:348] T 2d111096a9af46068a2476a6231d6520 P c75831edb3b44dd59c00ef151adca49d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } }
I20240502 21:25:47.992703 19692 raft_consensus.cc:374] T 2d111096a9af46068a2476a6231d6520 P c75831edb3b44dd59c00ef151adca49d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:47.992816 19692 raft_consensus.cc:729] T 2d111096a9af46068a2476a6231d6520 P c75831edb3b44dd59c00ef151adca49d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c75831edb3b44dd59c00ef151adca49d, State: Initialized, Role: FOLLOWER
I20240502 21:25:47.993106 19692 consensus_queue.cc:260] T 2d111096a9af46068a2476a6231d6520 P c75831edb3b44dd59c00ef151adca49d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } }
I20240502 21:25:47.997778 19692 ts_tablet_manager.cc:1398] T 2d111096a9af46068a2476a6231d6520 P c75831edb3b44dd59c00ef151adca49d: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240502 21:25:48.001567 19692 tablet_bootstrap.cc:492] T 51edafdf2a9348548a494119668bfd31 P c75831edb3b44dd59c00ef151adca49d: Bootstrap starting.
I20240502 21:25:48.003063 19688 tablet_bootstrap.cc:492] T 693a8666bbe54ae1b774f7f4da5ba67e P 82f50f4b66f3450aa9f58a0b317f5cea: Bootstrap starting.
I20240502 21:25:48.003453 19692 tablet_bootstrap.cc:654] T 51edafdf2a9348548a494119668bfd31 P c75831edb3b44dd59c00ef151adca49d: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:48.004660 19688 tablet_bootstrap.cc:654] T 693a8666bbe54ae1b774f7f4da5ba67e P 82f50f4b66f3450aa9f58a0b317f5cea: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:48.011819 19692 tablet_bootstrap.cc:492] T 51edafdf2a9348548a494119668bfd31 P c75831edb3b44dd59c00ef151adca49d: No bootstrap required, opened a new log
I20240502 21:25:48.012001 19692 ts_tablet_manager.cc:1367] T 51edafdf2a9348548a494119668bfd31 P c75831edb3b44dd59c00ef151adca49d: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.001s
I20240502 21:25:48.012849 19692 raft_consensus.cc:348] T 51edafdf2a9348548a494119668bfd31 P c75831edb3b44dd59c00ef151adca49d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } }
I20240502 21:25:48.013126 19692 raft_consensus.cc:374] T 51edafdf2a9348548a494119668bfd31 P c75831edb3b44dd59c00ef151adca49d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:48.013207 19692 raft_consensus.cc:729] T 51edafdf2a9348548a494119668bfd31 P c75831edb3b44dd59c00ef151adca49d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c75831edb3b44dd59c00ef151adca49d, State: Initialized, Role: FOLLOWER
I20240502 21:25:48.013478 19692 consensus_queue.cc:260] T 51edafdf2a9348548a494119668bfd31 P c75831edb3b44dd59c00ef151adca49d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } }
I20240502 21:25:48.014195 19692 ts_tablet_manager.cc:1398] T 51edafdf2a9348548a494119668bfd31 P c75831edb3b44dd59c00ef151adca49d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:25:48.019403 19699 tablet_bootstrap.cc:492] T 693a8666bbe54ae1b774f7f4da5ba67e P 9be399440ded4e2ebe6b6e0bf486cbb1: Bootstrap starting.
I20240502 21:25:48.022446 19699 tablet_bootstrap.cc:654] T 693a8666bbe54ae1b774f7f4da5ba67e P 9be399440ded4e2ebe6b6e0bf486cbb1: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:48.025125 19698 tablet_bootstrap.cc:492] T 51edafdf2a9348548a494119668bfd31 P faadabfb099b4abf90a466395be585b5: Bootstrap starting.
I20240502 21:25:48.027560 19698 tablet_bootstrap.cc:654] T 51edafdf2a9348548a494119668bfd31 P faadabfb099b4abf90a466395be585b5: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:48.028877 19688 tablet_bootstrap.cc:492] T 693a8666bbe54ae1b774f7f4da5ba67e P 82f50f4b66f3450aa9f58a0b317f5cea: No bootstrap required, opened a new log
I20240502 21:25:48.029063 19688 ts_tablet_manager.cc:1367] T 693a8666bbe54ae1b774f7f4da5ba67e P 82f50f4b66f3450aa9f58a0b317f5cea: Time spent bootstrapping tablet: real 0.026s user 0.003s sys 0.000s
I20240502 21:25:48.029529 19699 tablet_bootstrap.cc:492] T 693a8666bbe54ae1b774f7f4da5ba67e P 9be399440ded4e2ebe6b6e0bf486cbb1: No bootstrap required, opened a new log
I20240502 21:25:48.029743 19699 ts_tablet_manager.cc:1367] T 693a8666bbe54ae1b774f7f4da5ba67e P 9be399440ded4e2ebe6b6e0bf486cbb1: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.004s
I20240502 21:25:48.030258 19688 raft_consensus.cc:348] T 693a8666bbe54ae1b774f7f4da5ba67e P 82f50f4b66f3450aa9f58a0b317f5cea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } }
I20240502 21:25:48.030552 19688 raft_consensus.cc:374] T 693a8666bbe54ae1b774f7f4da5ba67e P 82f50f4b66f3450aa9f58a0b317f5cea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:48.030619 19688 raft_consensus.cc:729] T 693a8666bbe54ae1b774f7f4da5ba67e P 82f50f4b66f3450aa9f58a0b317f5cea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82f50f4b66f3450aa9f58a0b317f5cea, State: Initialized, Role: FOLLOWER
I20240502 21:25:48.030907 19688 consensus_queue.cc:260] T 693a8666bbe54ae1b774f7f4da5ba67e P 82f50f4b66f3450aa9f58a0b317f5cea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } }
I20240502 21:25:48.031520 19688 ts_tablet_manager.cc:1398] T 693a8666bbe54ae1b774f7f4da5ba67e P 82f50f4b66f3450aa9f58a0b317f5cea: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:25:48.031884 19688 tablet_bootstrap.cc:492] T 51edafdf2a9348548a494119668bfd31 P 82f50f4b66f3450aa9f58a0b317f5cea: Bootstrap starting.
I20240502 21:25:48.032490 19699 raft_consensus.cc:348] T 693a8666bbe54ae1b774f7f4da5ba67e P 9be399440ded4e2ebe6b6e0bf486cbb1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } }
I20240502 21:25:48.032809 19699 raft_consensus.cc:374] T 693a8666bbe54ae1b774f7f4da5ba67e P 9be399440ded4e2ebe6b6e0bf486cbb1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:48.032944 19699 raft_consensus.cc:729] T 693a8666bbe54ae1b774f7f4da5ba67e P 9be399440ded4e2ebe6b6e0bf486cbb1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9be399440ded4e2ebe6b6e0bf486cbb1, State: Initialized, Role: FOLLOWER
I20240502 21:25:48.033246 19699 consensus_queue.cc:260] T 693a8666bbe54ae1b774f7f4da5ba67e P 9be399440ded4e2ebe6b6e0bf486cbb1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } }
I20240502 21:25:48.033430 19688 tablet_bootstrap.cc:654] T 51edafdf2a9348548a494119668bfd31 P 82f50f4b66f3450aa9f58a0b317f5cea: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:48.033977 19699 ts_tablet_manager.cc:1398] T 693a8666bbe54ae1b774f7f4da5ba67e P 9be399440ded4e2ebe6b6e0bf486cbb1: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 21:25:48.034423 19699 tablet_bootstrap.cc:492] T 51edafdf2a9348548a494119668bfd31 P 9be399440ded4e2ebe6b6e0bf486cbb1: Bootstrap starting.
I20240502 21:25:48.035938 19699 tablet_bootstrap.cc:654] T 51edafdf2a9348548a494119668bfd31 P 9be399440ded4e2ebe6b6e0bf486cbb1: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:48.036978 19691 tablet_bootstrap.cc:492] T 2d111096a9af46068a2476a6231d6520 P 501e3c0d44fe4d70af9ff926d9622417: Bootstrap starting.
I20240502 21:25:48.039469 19691 tablet_bootstrap.cc:654] T 2d111096a9af46068a2476a6231d6520 P 501e3c0d44fe4d70af9ff926d9622417: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:48.040927 19689 ts_tablet_manager.cc:1398] T 2d111096a9af46068a2476a6231d6520 P 2877108756304e05bd60ba1186b870da: Time spent starting tablet: real 0.064s user 0.023s sys 0.024s
I20240502 21:25:48.041275 19689 tablet_bootstrap.cc:492] T 693a8666bbe54ae1b774f7f4da5ba67e P 2877108756304e05bd60ba1186b870da: Bootstrap starting.
I20240502 21:25:48.043481 19698 tablet_bootstrap.cc:492] T 51edafdf2a9348548a494119668bfd31 P faadabfb099b4abf90a466395be585b5: No bootstrap required, opened a new log
I20240502 21:25:48.043721 19698 ts_tablet_manager.cc:1367] T 51edafdf2a9348548a494119668bfd31 P faadabfb099b4abf90a466395be585b5: Time spent bootstrapping tablet: real 0.019s user 0.002s sys 0.002s
I20240502 21:25:48.044652 19698 raft_consensus.cc:348] T 51edafdf2a9348548a494119668bfd31 P faadabfb099b4abf90a466395be585b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } }
I20240502 21:25:48.045190 19698 raft_consensus.cc:374] T 51edafdf2a9348548a494119668bfd31 P faadabfb099b4abf90a466395be585b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:48.045325 19698 raft_consensus.cc:729] T 51edafdf2a9348548a494119668bfd31 P faadabfb099b4abf90a466395be585b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: faadabfb099b4abf90a466395be585b5, State: Initialized, Role: FOLLOWER
I20240502 21:25:48.045673 19689 tablet_bootstrap.cc:654] T 693a8666bbe54ae1b774f7f4da5ba67e P 2877108756304e05bd60ba1186b870da: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:48.045805 19698 consensus_queue.cc:260] T 51edafdf2a9348548a494119668bfd31 P faadabfb099b4abf90a466395be585b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } }
I20240502 21:25:48.046351 19698 ts_tablet_manager.cc:1398] T 51edafdf2a9348548a494119668bfd31 P faadabfb099b4abf90a466395be585b5: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 21:25:48.046643 19698 tablet_bootstrap.cc:492] T 693a8666bbe54ae1b774f7f4da5ba67e P faadabfb099b4abf90a466395be585b5: Bootstrap starting.
I20240502 21:25:48.047495 19688 tablet_bootstrap.cc:492] T 51edafdf2a9348548a494119668bfd31 P 82f50f4b66f3450aa9f58a0b317f5cea: No bootstrap required, opened a new log
I20240502 21:25:48.047663 19688 ts_tablet_manager.cc:1367] T 51edafdf2a9348548a494119668bfd31 P 82f50f4b66f3450aa9f58a0b317f5cea: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.003s
I20240502 21:25:48.047896 19689 tablet_bootstrap.cc:492] T 693a8666bbe54ae1b774f7f4da5ba67e P 2877108756304e05bd60ba1186b870da: No bootstrap required, opened a new log
I20240502 21:25:48.048060 19689 ts_tablet_manager.cc:1367] T 693a8666bbe54ae1b774f7f4da5ba67e P 2877108756304e05bd60ba1186b870da: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.002s
I20240502 21:25:48.048281 19691 tablet_bootstrap.cc:492] T 2d111096a9af46068a2476a6231d6520 P 501e3c0d44fe4d70af9ff926d9622417: No bootstrap required, opened a new log
I20240502 21:25:48.048453 19691 ts_tablet_manager.cc:1367] T 2d111096a9af46068a2476a6231d6520 P 501e3c0d44fe4d70af9ff926d9622417: Time spent bootstrapping tablet: real 0.012s user 0.008s sys 0.000s
I20240502 21:25:48.048379 19688 raft_consensus.cc:348] T 51edafdf2a9348548a494119668bfd31 P 82f50f4b66f3450aa9f58a0b317f5cea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } }
I20240502 21:25:48.048790 19688 raft_consensus.cc:374] T 51edafdf2a9348548a494119668bfd31 P 82f50f4b66f3450aa9f58a0b317f5cea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:48.048889 19688 raft_consensus.cc:729] T 51edafdf2a9348548a494119668bfd31 P 82f50f4b66f3450aa9f58a0b317f5cea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82f50f4b66f3450aa9f58a0b317f5cea, State: Initialized, Role: FOLLOWER
I20240502 21:25:48.048874 19689 raft_consensus.cc:348] T 693a8666bbe54ae1b774f7f4da5ba67e P 2877108756304e05bd60ba1186b870da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } }
I20240502 21:25:48.049197 19689 raft_consensus.cc:374] T 693a8666bbe54ae1b774f7f4da5ba67e P 2877108756304e05bd60ba1186b870da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:48.049121 19688 consensus_queue.cc:260] T 51edafdf2a9348548a494119668bfd31 P 82f50f4b66f3450aa9f58a0b317f5cea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } }
I20240502 21:25:48.049317 19689 raft_consensus.cc:729] T 693a8666bbe54ae1b774f7f4da5ba67e P 2877108756304e05bd60ba1186b870da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2877108756304e05bd60ba1186b870da, State: Initialized, Role: FOLLOWER
I20240502 21:25:48.049705 19688 ts_tablet_manager.cc:1398] T 51edafdf2a9348548a494119668bfd31 P 82f50f4b66f3450aa9f58a0b317f5cea: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 21:25:48.049610 19689 consensus_queue.cc:260] T 693a8666bbe54ae1b774f7f4da5ba67e P 2877108756304e05bd60ba1186b870da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } }
I20240502 21:25:48.049590 19691 raft_consensus.cc:348] T 2d111096a9af46068a2476a6231d6520 P 501e3c0d44fe4d70af9ff926d9622417 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } }
I20240502 21:25:48.050052 19691 raft_consensus.cc:374] T 2d111096a9af46068a2476a6231d6520 P 501e3c0d44fe4d70af9ff926d9622417 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:48.050186 19689 ts_tablet_manager.cc:1398] T 693a8666bbe54ae1b774f7f4da5ba67e P 2877108756304e05bd60ba1186b870da: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:25:48.050222 19691 raft_consensus.cc:729] T 2d111096a9af46068a2476a6231d6520 P 501e3c0d44fe4d70af9ff926d9622417 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 501e3c0d44fe4d70af9ff926d9622417, State: Initialized, Role: FOLLOWER
I20240502 21:25:48.050545 19689 tablet_bootstrap.cc:492] T 51edafdf2a9348548a494119668bfd31 P 2877108756304e05bd60ba1186b870da: Bootstrap starting.
I20240502 21:25:48.050539 19691 consensus_queue.cc:260] T 2d111096a9af46068a2476a6231d6520 P 501e3c0d44fe4d70af9ff926d9622417 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } }
I20240502 21:25:48.052103 19698 tablet_bootstrap.cc:654] T 693a8666bbe54ae1b774f7f4da5ba67e P faadabfb099b4abf90a466395be585b5: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:48.052281 19689 tablet_bootstrap.cc:654] T 51edafdf2a9348548a494119668bfd31 P 2877108756304e05bd60ba1186b870da: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:48.054371 19699 tablet_bootstrap.cc:492] T 51edafdf2a9348548a494119668bfd31 P 9be399440ded4e2ebe6b6e0bf486cbb1: No bootstrap required, opened a new log
I20240502 21:25:48.054596 19699 ts_tablet_manager.cc:1367] T 51edafdf2a9348548a494119668bfd31 P 9be399440ded4e2ebe6b6e0bf486cbb1: Time spent bootstrapping tablet: real 0.020s user 0.004s sys 0.000s
I20240502 21:25:48.055394 19699 raft_consensus.cc:348] T 51edafdf2a9348548a494119668bfd31 P 9be399440ded4e2ebe6b6e0bf486cbb1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } }
I20240502 21:25:48.055706 19699 raft_consensus.cc:374] T 51edafdf2a9348548a494119668bfd31 P 9be399440ded4e2ebe6b6e0bf486cbb1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:48.055821 19699 raft_consensus.cc:729] T 51edafdf2a9348548a494119668bfd31 P 9be399440ded4e2ebe6b6e0bf486cbb1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9be399440ded4e2ebe6b6e0bf486cbb1, State: Initialized, Role: FOLLOWER
I20240502 21:25:48.056116 19699 consensus_queue.cc:260] T 51edafdf2a9348548a494119668bfd31 P 9be399440ded4e2ebe6b6e0bf486cbb1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } }
I20240502 21:25:48.057271 19699 ts_tablet_manager.cc:1398] T 51edafdf2a9348548a494119668bfd31 P 9be399440ded4e2ebe6b6e0bf486cbb1: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 21:25:48.057832 19691 ts_tablet_manager.cc:1398] T 2d111096a9af46068a2476a6231d6520 P 501e3c0d44fe4d70af9ff926d9622417: Time spent starting tablet: real 0.009s user 0.000s sys 0.007s
I20240502 21:25:48.057904 19689 tablet_bootstrap.cc:492] T 51edafdf2a9348548a494119668bfd31 P 2877108756304e05bd60ba1186b870da: No bootstrap required, opened a new log
I20240502 21:25:48.058089 19689 ts_tablet_manager.cc:1367] T 51edafdf2a9348548a494119668bfd31 P 2877108756304e05bd60ba1186b870da: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.003s
I20240502 21:25:48.058212 19691 tablet_bootstrap.cc:492] T 693a8666bbe54ae1b774f7f4da5ba67e P 501e3c0d44fe4d70af9ff926d9622417: Bootstrap starting.
I20240502 21:25:48.058933 19689 raft_consensus.cc:348] T 51edafdf2a9348548a494119668bfd31 P 2877108756304e05bd60ba1186b870da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } }
I20240502 21:25:48.059237 19689 raft_consensus.cc:374] T 51edafdf2a9348548a494119668bfd31 P 2877108756304e05bd60ba1186b870da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:48.059373 19689 raft_consensus.cc:729] T 51edafdf2a9348548a494119668bfd31 P 2877108756304e05bd60ba1186b870da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2877108756304e05bd60ba1186b870da, State: Initialized, Role: FOLLOWER
I20240502 21:25:48.059671 19689 consensus_queue.cc:260] T 51edafdf2a9348548a494119668bfd31 P 2877108756304e05bd60ba1186b870da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } }
I20240502 21:25:48.060339 19689 ts_tablet_manager.cc:1398] T 51edafdf2a9348548a494119668bfd31 P 2877108756304e05bd60ba1186b870da: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 21:25:48.060803 19691 tablet_bootstrap.cc:654] T 693a8666bbe54ae1b774f7f4da5ba67e P 501e3c0d44fe4d70af9ff926d9622417: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:48.063556 19691 tablet_bootstrap.cc:492] T 693a8666bbe54ae1b774f7f4da5ba67e P 501e3c0d44fe4d70af9ff926d9622417: No bootstrap required, opened a new log
I20240502 21:25:48.063558 19698 tablet_bootstrap.cc:492] T 693a8666bbe54ae1b774f7f4da5ba67e P faadabfb099b4abf90a466395be585b5: No bootstrap required, opened a new log
I20240502 21:25:48.063746 19698 ts_tablet_manager.cc:1367] T 693a8666bbe54ae1b774f7f4da5ba67e P faadabfb099b4abf90a466395be585b5: Time spent bootstrapping tablet: real 0.017s user 0.001s sys 0.001s
I20240502 21:25:48.063746 19691 ts_tablet_manager.cc:1367] T 693a8666bbe54ae1b774f7f4da5ba67e P 501e3c0d44fe4d70af9ff926d9622417: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.003s
I20240502 21:25:48.064550 19691 raft_consensus.cc:348] T 693a8666bbe54ae1b774f7f4da5ba67e P 501e3c0d44fe4d70af9ff926d9622417 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } }
I20240502 21:25:48.064550 19698 raft_consensus.cc:348] T 693a8666bbe54ae1b774f7f4da5ba67e P faadabfb099b4abf90a466395be585b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } }
I20240502 21:25:48.064838 19698 raft_consensus.cc:374] T 693a8666bbe54ae1b774f7f4da5ba67e P faadabfb099b4abf90a466395be585b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:48.064844 19691 raft_consensus.cc:374] T 693a8666bbe54ae1b774f7f4da5ba67e P 501e3c0d44fe4d70af9ff926d9622417 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:48.064929 19698 raft_consensus.cc:729] T 693a8666bbe54ae1b774f7f4da5ba67e P faadabfb099b4abf90a466395be585b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: faadabfb099b4abf90a466395be585b5, State: Initialized, Role: FOLLOWER
I20240502 21:25:48.064929 19691 raft_consensus.cc:729] T 693a8666bbe54ae1b774f7f4da5ba67e P 501e3c0d44fe4d70af9ff926d9622417 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 501e3c0d44fe4d70af9ff926d9622417, State: Initialized, Role: FOLLOWER
I20240502 21:25:48.065207 19698 consensus_queue.cc:260] T 693a8666bbe54ae1b774f7f4da5ba67e P faadabfb099b4abf90a466395be585b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } }
I20240502 21:25:48.065187 19691 consensus_queue.cc:260] T 693a8666bbe54ae1b774f7f4da5ba67e P 501e3c0d44fe4d70af9ff926d9622417 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } }
I20240502 21:25:48.065855 19698 ts_tablet_manager.cc:1398] T 693a8666bbe54ae1b774f7f4da5ba67e P faadabfb099b4abf90a466395be585b5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:25:48.066241 19698 tablet_bootstrap.cc:492] T 2d111096a9af46068a2476a6231d6520 P faadabfb099b4abf90a466395be585b5: Bootstrap starting.
I20240502 21:25:48.066730 19691 ts_tablet_manager.cc:1398] T 693a8666bbe54ae1b774f7f4da5ba67e P 501e3c0d44fe4d70af9ff926d9622417: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240502 21:25:48.067924 19698 tablet_bootstrap.cc:654] T 2d111096a9af46068a2476a6231d6520 P faadabfb099b4abf90a466395be585b5: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:48.069835 19698 tablet_bootstrap.cc:492] T 2d111096a9af46068a2476a6231d6520 P faadabfb099b4abf90a466395be585b5: No bootstrap required, opened a new log
I20240502 21:25:48.069973 19698 ts_tablet_manager.cc:1367] T 2d111096a9af46068a2476a6231d6520 P faadabfb099b4abf90a466395be585b5: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240502 21:25:48.070560 19698 raft_consensus.cc:348] T 2d111096a9af46068a2476a6231d6520 P faadabfb099b4abf90a466395be585b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } }
I20240502 21:25:48.070833 19698 raft_consensus.cc:374] T 2d111096a9af46068a2476a6231d6520 P faadabfb099b4abf90a466395be585b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:48.070937 19698 raft_consensus.cc:729] T 2d111096a9af46068a2476a6231d6520 P faadabfb099b4abf90a466395be585b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: faadabfb099b4abf90a466395be585b5, State: Initialized, Role: FOLLOWER
I20240502 21:25:48.071159 19698 consensus_queue.cc:260] T 2d111096a9af46068a2476a6231d6520 P faadabfb099b4abf90a466395be585b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } }
I20240502 21:25:48.071656 19698 ts_tablet_manager.cc:1398] T 2d111096a9af46068a2476a6231d6520 P faadabfb099b4abf90a466395be585b5: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
W20240502 21:25:48.117308 18406 consensus_peers.cc:485] T 40d4f61d0f6d4d31a5626c7743c88133 P faadabfb099b4abf90a466395be585b5 -> Peer 90e329b457c24caf93a9f880345eb288 (127.15.255.135:43395): Couldn't send request to peer 90e329b457c24caf93a9f880345eb288. Status: Network error: Client connection negotiation failed: client connection to 127.15.255.135:43395: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240502 21:25:48.196528 19568 raft_consensus.cc:482] T 51edafdf2a9348548a494119668bfd31 P c75831edb3b44dd59c00ef151adca49d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:25:48.196761 19568 raft_consensus.cc:504] T 51edafdf2a9348548a494119668bfd31 P c75831edb3b44dd59c00ef151adca49d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } }
I20240502 21:25:48.197640 19568 leader_election.cc:288] T 51edafdf2a9348548a494119668bfd31 P c75831edb3b44dd59c00ef151adca49d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers faadabfb099b4abf90a466395be585b5 (127.15.255.129:32983), 2877108756304e05bd60ba1186b870da (127.15.255.134:43635), 82f50f4b66f3450aa9f58a0b317f5cea (127.15.255.130:45139), 9be399440ded4e2ebe6b6e0bf486cbb1 (127.15.255.133:34217)
I20240502 21:25:48.197866 18473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51edafdf2a9348548a494119668bfd31" candidate_uuid: "c75831edb3b44dd59c00ef151adca49d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "faadabfb099b4abf90a466395be585b5" is_pre_election: true
I20240502 21:25:48.198014 19146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51edafdf2a9348548a494119668bfd31" candidate_uuid: "c75831edb3b44dd59c00ef151adca49d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2877108756304e05bd60ba1186b870da" is_pre_election: true
I20240502 21:25:48.198117 18473 raft_consensus.cc:2454] T 51edafdf2a9348548a494119668bfd31 P faadabfb099b4abf90a466395be585b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c75831edb3b44dd59c00ef151adca49d in term 0.
I20240502 21:25:48.198238 19146 raft_consensus.cc:2454] T 51edafdf2a9348548a494119668bfd31 P 2877108756304e05bd60ba1186b870da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c75831edb3b44dd59c00ef151adca49d in term 0.
I20240502 21:25:48.198323 18610 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51edafdf2a9348548a494119668bfd31" candidate_uuid: "c75831edb3b44dd59c00ef151adca49d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" is_pre_election: true
I20240502 21:25:48.198549 18610 raft_consensus.cc:2454] T 51edafdf2a9348548a494119668bfd31 P 82f50f4b66f3450aa9f58a0b317f5cea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c75831edb3b44dd59c00ef151adca49d in term 0.
I20240502 21:25:48.198726 18812 leader_election.cc:302] T 51edafdf2a9348548a494119668bfd31 P c75831edb3b44dd59c00ef151adca49d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 2877108756304e05bd60ba1186b870da, c75831edb3b44dd59c00ef151adca49d, faadabfb099b4abf90a466395be585b5; no voters:
I20240502 21:25:48.198778 19010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51edafdf2a9348548a494119668bfd31" candidate_uuid: "c75831edb3b44dd59c00ef151adca49d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" is_pre_election: true
I20240502 21:25:48.199034 19010 raft_consensus.cc:2454] T 51edafdf2a9348548a494119668bfd31 P 9be399440ded4e2ebe6b6e0bf486cbb1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c75831edb3b44dd59c00ef151adca49d in term 0.
I20240502 21:25:48.199141 19568 raft_consensus.cc:2781] T 51edafdf2a9348548a494119668bfd31 P c75831edb3b44dd59c00ef151adca49d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:25:48.199254 19568 raft_consensus.cc:482] T 51edafdf2a9348548a494119668bfd31 P c75831edb3b44dd59c00ef151adca49d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:25:48.199360 19568 raft_consensus.cc:3037] T 51edafdf2a9348548a494119668bfd31 P c75831edb3b44dd59c00ef151adca49d [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:48.201351 19568 raft_consensus.cc:504] T 51edafdf2a9348548a494119668bfd31 P c75831edb3b44dd59c00ef151adca49d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } }
I20240502 21:25:48.202165 19568 leader_election.cc:288] T 51edafdf2a9348548a494119668bfd31 P c75831edb3b44dd59c00ef151adca49d [CANDIDATE]: Term 1 election: Requested vote from peers faadabfb099b4abf90a466395be585b5 (127.15.255.129:32983), 2877108756304e05bd60ba1186b870da (127.15.255.134:43635), 82f50f4b66f3450aa9f58a0b317f5cea (127.15.255.130:45139), 9be399440ded4e2ebe6b6e0bf486cbb1 (127.15.255.133:34217)
I20240502 21:25:48.202293 18473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51edafdf2a9348548a494119668bfd31" candidate_uuid: "c75831edb3b44dd59c00ef151adca49d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "faadabfb099b4abf90a466395be585b5"
I20240502 21:25:48.202489 18473 raft_consensus.cc:3037] T 51edafdf2a9348548a494119668bfd31 P faadabfb099b4abf90a466395be585b5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:48.202462 19146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51edafdf2a9348548a494119668bfd31" candidate_uuid: "c75831edb3b44dd59c00ef151adca49d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2877108756304e05bd60ba1186b870da"
I20240502 21:25:48.202657 19146 raft_consensus.cc:3037] T 51edafdf2a9348548a494119668bfd31 P 2877108756304e05bd60ba1186b870da [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:48.202776 18610 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51edafdf2a9348548a494119668bfd31" candidate_uuid: "c75831edb3b44dd59c00ef151adca49d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82f50f4b66f3450aa9f58a0b317f5cea"
I20240502 21:25:48.202945 18610 raft_consensus.cc:3037] T 51edafdf2a9348548a494119668bfd31 P 82f50f4b66f3450aa9f58a0b317f5cea [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:48.203198 19010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51edafdf2a9348548a494119668bfd31" candidate_uuid: "c75831edb3b44dd59c00ef151adca49d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1"
I20240502 21:25:48.203399 19010 raft_consensus.cc:3037] T 51edafdf2a9348548a494119668bfd31 P 9be399440ded4e2ebe6b6e0bf486cbb1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:48.205240 18473 raft_consensus.cc:2454] T 51edafdf2a9348548a494119668bfd31 P faadabfb099b4abf90a466395be585b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c75831edb3b44dd59c00ef151adca49d in term 1.
I20240502 21:25:48.205380 19146 raft_consensus.cc:2454] T 51edafdf2a9348548a494119668bfd31 P 2877108756304e05bd60ba1186b870da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c75831edb3b44dd59c00ef151adca49d in term 1.
I20240502 21:25:48.205549 18610 raft_consensus.cc:2454] T 51edafdf2a9348548a494119668bfd31 P 82f50f4b66f3450aa9f58a0b317f5cea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c75831edb3b44dd59c00ef151adca49d in term 1.
I20240502 21:25:48.205785 18812 leader_election.cc:302] T 51edafdf2a9348548a494119668bfd31 P c75831edb3b44dd59c00ef151adca49d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 2877108756304e05bd60ba1186b870da, c75831edb3b44dd59c00ef151adca49d, faadabfb099b4abf90a466395be585b5; no voters:
I20240502 21:25:48.206009 19010 raft_consensus.cc:2454] T 51edafdf2a9348548a494119668bfd31 P 9be399440ded4e2ebe6b6e0bf486cbb1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c75831edb3b44dd59c00ef151adca49d in term 1.
I20240502 21:25:48.206049 19568 raft_consensus.cc:2781] T 51edafdf2a9348548a494119668bfd31 P c75831edb3b44dd59c00ef151adca49d [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:25:48.206470 19568 raft_consensus.cc:686] T 51edafdf2a9348548a494119668bfd31 P c75831edb3b44dd59c00ef151adca49d [term 1 LEADER]: Becoming Leader. State: Replica: c75831edb3b44dd59c00ef151adca49d, State: Running, Role: LEADER
I20240502 21:25:48.206749 19568 consensus_queue.cc:237] T 51edafdf2a9348548a494119668bfd31 P c75831edb3b44dd59c00ef151adca49d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } }
I20240502 21:25:48.209529 18322 catalog_manager.cc:5484] T 51edafdf2a9348548a494119668bfd31 P c75831edb3b44dd59c00ef151adca49d reported cstate change: term changed from 0 to 1, leader changed from <none> to c75831edb3b44dd59c00ef151adca49d (127.15.255.132). New cstate: current_term: 1 leader_uuid: "c75831edb3b44dd59c00ef151adca49d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } health_report { overall_health: UNKNOWN } } }
I20240502 21:25:48.220189 19694 raft_consensus.cc:482] T 2d111096a9af46068a2476a6231d6520 P 2877108756304e05bd60ba1186b870da [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:25:48.220360 19694 raft_consensus.cc:504] T 2d111096a9af46068a2476a6231d6520 P 2877108756304e05bd60ba1186b870da [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } }
I20240502 21:25:48.222347 19694 leader_election.cc:288] T 2d111096a9af46068a2476a6231d6520 P 2877108756304e05bd60ba1186b870da [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 82f50f4b66f3450aa9f58a0b317f5cea (127.15.255.130:45139), faadabfb099b4abf90a466395be585b5 (127.15.255.129:32983), 501e3c0d44fe4d70af9ff926d9622417 (127.15.255.131:42501), c75831edb3b44dd59c00ef151adca49d (127.15.255.132:43607)
I20240502 21:25:48.231778 19706 raft_consensus.cc:482] T 693a8666bbe54ae1b774f7f4da5ba67e P 501e3c0d44fe4d70af9ff926d9622417 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:25:48.232040 19706 raft_consensus.cc:504] T 693a8666bbe54ae1b774f7f4da5ba67e P 501e3c0d44fe4d70af9ff926d9622417 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } }
I20240502 21:25:48.232576 19640 raft_consensus.cc:482] T 2d111096a9af46068a2476a6231d6520 P 82f50f4b66f3450aa9f58a0b317f5cea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:25:48.232748 19640 raft_consensus.cc:504] T 2d111096a9af46068a2476a6231d6520 P 82f50f4b66f3450aa9f58a0b317f5cea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } }
I20240502 21:25:48.233862 19640 leader_election.cc:288] T 2d111096a9af46068a2476a6231d6520 P 82f50f4b66f3450aa9f58a0b317f5cea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2877108756304e05bd60ba1186b870da (127.15.255.134:43635), faadabfb099b4abf90a466395be585b5 (127.15.255.129:32983), 501e3c0d44fe4d70af9ff926d9622417 (127.15.255.131:42501), c75831edb3b44dd59c00ef151adca49d (127.15.255.132:43607)
I20240502 21:25:48.234544 18742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d111096a9af46068a2476a6231d6520" candidate_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "501e3c0d44fe4d70af9ff926d9622417" is_pre_election: true
I20240502 21:25:48.234530 18878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d111096a9af46068a2476a6231d6520" candidate_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c75831edb3b44dd59c00ef151adca49d" is_pre_election: true
I20240502 21:25:48.234733 18742 raft_consensus.cc:2454] T 2d111096a9af46068a2476a6231d6520 P 501e3c0d44fe4d70af9ff926d9622417 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82f50f4b66f3450aa9f58a0b317f5cea in term 0.
I20240502 21:25:48.234755 18878 raft_consensus.cc:2454] T 2d111096a9af46068a2476a6231d6520 P c75831edb3b44dd59c00ef151adca49d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82f50f4b66f3450aa9f58a0b317f5cea in term 0.
I20240502 21:25:48.235572 18543 leader_election.cc:302] T 2d111096a9af46068a2476a6231d6520 P 82f50f4b66f3450aa9f58a0b317f5cea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 501e3c0d44fe4d70af9ff926d9622417, 82f50f4b66f3450aa9f58a0b317f5cea, c75831edb3b44dd59c00ef151adca49d; no voters:
I20240502 21:25:48.236140 19640 raft_consensus.cc:2781] T 2d111096a9af46068a2476a6231d6520 P 82f50f4b66f3450aa9f58a0b317f5cea [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:25:48.236241 19640 raft_consensus.cc:482] T 2d111096a9af46068a2476a6231d6520 P 82f50f4b66f3450aa9f58a0b317f5cea [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:25:48.236318 19640 raft_consensus.cc:3037] T 2d111096a9af46068a2476a6231d6520 P 82f50f4b66f3450aa9f58a0b317f5cea [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:48.239218 19640 raft_consensus.cc:504] T 2d111096a9af46068a2476a6231d6520 P 82f50f4b66f3450aa9f58a0b317f5cea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } }
I20240502 21:25:48.239831 18473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d111096a9af46068a2476a6231d6520" candidate_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "faadabfb099b4abf90a466395be585b5" is_pre_election: true
I20240502 21:25:48.239946 19595 raft_consensus.cc:482] T 693a8666bbe54ae1b774f7f4da5ba67e P faadabfb099b4abf90a466395be585b5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:25:48.240056 18473 raft_consensus.cc:2454] T 2d111096a9af46068a2476a6231d6520 P faadabfb099b4abf90a466395be585b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82f50f4b66f3450aa9f58a0b317f5cea in term 0.
I20240502 21:25:48.240077 19640 leader_election.cc:288] T 2d111096a9af46068a2476a6231d6520 P 82f50f4b66f3450aa9f58a0b317f5cea [CANDIDATE]: Term 1 election: Requested vote from peers 2877108756304e05bd60ba1186b870da (127.15.255.134:43635), faadabfb099b4abf90a466395be585b5 (127.15.255.129:32983), 501e3c0d44fe4d70af9ff926d9622417 (127.15.255.131:42501), c75831edb3b44dd59c00ef151adca49d (127.15.255.132:43607)
I20240502 21:25:48.240105 19595 raft_consensus.cc:504] T 693a8666bbe54ae1b774f7f4da5ba67e P faadabfb099b4abf90a466395be585b5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } }
I20240502 21:25:48.241030 18742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d111096a9af46068a2476a6231d6520" candidate_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "501e3c0d44fe4d70af9ff926d9622417"
I20240502 21:25:48.241077 19595 leader_election.cc:288] T 693a8666bbe54ae1b774f7f4da5ba67e P faadabfb099b4abf90a466395be585b5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9be399440ded4e2ebe6b6e0bf486cbb1 (127.15.255.133:34217), 2877108756304e05bd60ba1186b870da (127.15.255.134:43635), 501e3c0d44fe4d70af9ff926d9622417 (127.15.255.131:42501), 82f50f4b66f3450aa9f58a0b317f5cea (127.15.255.130:45139)
I20240502 21:25:48.241222 18742 raft_consensus.cc:3037] T 2d111096a9af46068a2476a6231d6520 P 501e3c0d44fe4d70af9ff926d9622417 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:48.241439 18743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "693a8666bbe54ae1b774f7f4da5ba67e" candidate_uuid: "faadabfb099b4abf90a466395be585b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "501e3c0d44fe4d70af9ff926d9622417" is_pre_election: true
I20240502 21:25:48.241657 18743 raft_consensus.cc:2454] T 693a8666bbe54ae1b774f7f4da5ba67e P 501e3c0d44fe4d70af9ff926d9622417 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate faadabfb099b4abf90a466395be585b5 in term 0.
I20240502 21:25:48.242245 18878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d111096a9af46068a2476a6231d6520" candidate_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c75831edb3b44dd59c00ef151adca49d"
I20240502 21:25:48.242439 18878 raft_consensus.cc:3037] T 2d111096a9af46068a2476a6231d6520 P c75831edb3b44dd59c00ef151adca49d [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:48.243216 18473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d111096a9af46068a2476a6231d6520" candidate_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "faadabfb099b4abf90a466395be585b5"
I20240502 21:25:48.243405 18473 raft_consensus.cc:3037] T 2d111096a9af46068a2476a6231d6520 P faadabfb099b4abf90a466395be585b5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:48.243997 18742 raft_consensus.cc:2454] T 2d111096a9af46068a2476a6231d6520 P 501e3c0d44fe4d70af9ff926d9622417 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82f50f4b66f3450aa9f58a0b317f5cea in term 1.
I20240502 21:25:48.244544 19010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "693a8666bbe54ae1b774f7f4da5ba67e" candidate_uuid: "faadabfb099b4abf90a466395be585b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" is_pre_election: true
I20240502 21:25:48.244766 19010 raft_consensus.cc:2454] T 693a8666bbe54ae1b774f7f4da5ba67e P 9be399440ded4e2ebe6b6e0bf486cbb1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate faadabfb099b4abf90a466395be585b5 in term 0.
I20240502 21:25:48.245074 18878 raft_consensus.cc:2454] T 2d111096a9af46068a2476a6231d6520 P c75831edb3b44dd59c00ef151adca49d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82f50f4b66f3450aa9f58a0b317f5cea in term 1.
I20240502 21:25:48.245292 18405 leader_election.cc:302] T 693a8666bbe54ae1b774f7f4da5ba67e P faadabfb099b4abf90a466395be585b5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 501e3c0d44fe4d70af9ff926d9622417, 9be399440ded4e2ebe6b6e0bf486cbb1, faadabfb099b4abf90a466395be585b5; no voters:
I20240502 21:25:48.245718 19595 raft_consensus.cc:2781] T 693a8666bbe54ae1b774f7f4da5ba67e P faadabfb099b4abf90a466395be585b5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:25:48.245854 19595 raft_consensus.cc:482] T 693a8666bbe54ae1b774f7f4da5ba67e P faadabfb099b4abf90a466395be585b5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:25:48.245947 19595 raft_consensus.cc:3037] T 693a8666bbe54ae1b774f7f4da5ba67e P faadabfb099b4abf90a466395be585b5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:48.246028 18473 raft_consensus.cc:2454] T 2d111096a9af46068a2476a6231d6520 P faadabfb099b4abf90a466395be585b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82f50f4b66f3450aa9f58a0b317f5cea in term 1.
I20240502 21:25:48.246384 18544 leader_election.cc:302] T 2d111096a9af46068a2476a6231d6520 P 82f50f4b66f3450aa9f58a0b317f5cea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 82f50f4b66f3450aa9f58a0b317f5cea, c75831edb3b44dd59c00ef151adca49d, faadabfb099b4abf90a466395be585b5; no voters:
I20240502 21:25:48.246745 19640 raft_consensus.cc:2781] T 2d111096a9af46068a2476a6231d6520 P 82f50f4b66f3450aa9f58a0b317f5cea [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:25:48.247108 19640 raft_consensus.cc:686] T 2d111096a9af46068a2476a6231d6520 P 82f50f4b66f3450aa9f58a0b317f5cea [term 1 LEADER]: Becoming Leader. State: Replica: 82f50f4b66f3450aa9f58a0b317f5cea, State: Running, Role: LEADER
I20240502 21:25:48.247649 19640 consensus_queue.cc:237] T 2d111096a9af46068a2476a6231d6520 P 82f50f4b66f3450aa9f58a0b317f5cea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } }
I20240502 21:25:48.248721 19595 raft_consensus.cc:504] T 693a8666bbe54ae1b774f7f4da5ba67e P faadabfb099b4abf90a466395be585b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } }
I20240502 21:25:48.250705 19010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "693a8666bbe54ae1b774f7f4da5ba67e" candidate_uuid: "faadabfb099b4abf90a466395be585b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1"
I20240502 21:25:48.250916 19010 raft_consensus.cc:3037] T 693a8666bbe54ae1b774f7f4da5ba67e P 9be399440ded4e2ebe6b6e0bf486cbb1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:48.253762 19010 raft_consensus.cc:2454] T 693a8666bbe54ae1b774f7f4da5ba67e P 9be399440ded4e2ebe6b6e0bf486cbb1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate faadabfb099b4abf90a466395be585b5 in term 1.
I20240502 21:25:48.256690 18610 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "693a8666bbe54ae1b774f7f4da5ba67e" candidate_uuid: "faadabfb099b4abf90a466395be585b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" is_pre_election: true
I20240502 21:25:48.257858 19595 leader_election.cc:288] T 693a8666bbe54ae1b774f7f4da5ba67e P faadabfb099b4abf90a466395be585b5 [CANDIDATE]: Term 1 election: Requested vote from peers 9be399440ded4e2ebe6b6e0bf486cbb1 (127.15.255.133:34217), 2877108756304e05bd60ba1186b870da (127.15.255.134:43635), 501e3c0d44fe4d70af9ff926d9622417 (127.15.255.131:42501), 82f50f4b66f3450aa9f58a0b317f5cea (127.15.255.130:45139)
I20240502 21:25:48.257771 18611 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "693a8666bbe54ae1b774f7f4da5ba67e" candidate_uuid: "faadabfb099b4abf90a466395be585b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82f50f4b66f3450aa9f58a0b317f5cea"
I20240502 21:25:48.258052 18742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "693a8666bbe54ae1b774f7f4da5ba67e" candidate_uuid: "faadabfb099b4abf90a466395be585b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "501e3c0d44fe4d70af9ff926d9622417"
I20240502 21:25:48.258242 18611 raft_consensus.cc:3037] T 693a8666bbe54ae1b774f7f4da5ba67e P 82f50f4b66f3450aa9f58a0b317f5cea [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:48.258248 18742 raft_consensus.cc:3037] T 693a8666bbe54ae1b774f7f4da5ba67e P 501e3c0d44fe4d70af9ff926d9622417 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:48.261013 18742 raft_consensus.cc:2454] T 693a8666bbe54ae1b774f7f4da5ba67e P 501e3c0d44fe4d70af9ff926d9622417 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate faadabfb099b4abf90a466395be585b5 in term 1.
I20240502 21:25:48.261974 18611 raft_consensus.cc:2454] T 693a8666bbe54ae1b774f7f4da5ba67e P 82f50f4b66f3450aa9f58a0b317f5cea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate faadabfb099b4abf90a466395be585b5 in term 1.
I20240502 21:25:48.263032 18878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d111096a9af46068a2476a6231d6520" candidate_uuid: "2877108756304e05bd60ba1186b870da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c75831edb3b44dd59c00ef151adca49d" is_pre_election: true
I20240502 21:25:48.263285 18878 raft_consensus.cc:2379] T 2d111096a9af46068a2476a6231d6520 P c75831edb3b44dd59c00ef151adca49d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2877108756304e05bd60ba1186b870da in current term 1: Already voted for candidate 82f50f4b66f3450aa9f58a0b317f5cea in this term.
I20240502 21:25:48.264199 18610 raft_consensus.cc:2362] T 693a8666bbe54ae1b774f7f4da5ba67e P 82f50f4b66f3450aa9f58a0b317f5cea [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate faadabfb099b4abf90a466395be585b5 in term 1. Re-sending same reply.
I20240502 21:25:48.261498 18404 leader_election.cc:302] T 693a8666bbe54ae1b774f7f4da5ba67e P faadabfb099b4abf90a466395be585b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 501e3c0d44fe4d70af9ff926d9622417, 9be399440ded4e2ebe6b6e0bf486cbb1, faadabfb099b4abf90a466395be585b5; no voters:
I20240502 21:25:48.265129 19725 raft_consensus.cc:2781] T 693a8666bbe54ae1b774f7f4da5ba67e P faadabfb099b4abf90a466395be585b5 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:25:48.269560 18473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d111096a9af46068a2476a6231d6520" candidate_uuid: "2877108756304e05bd60ba1186b870da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "faadabfb099b4abf90a466395be585b5" is_pre_election: true
I20240502 21:25:48.269821 18473 raft_consensus.cc:2379] T 2d111096a9af46068a2476a6231d6520 P faadabfb099b4abf90a466395be585b5 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2877108756304e05bd60ba1186b870da in current term 1: Already voted for candidate 82f50f4b66f3450aa9f58a0b317f5cea in this term.
I20240502 21:25:48.272106 18322 catalog_manager.cc:5484] T 2d111096a9af46068a2476a6231d6520 P 82f50f4b66f3450aa9f58a0b317f5cea reported cstate change: term changed from 0 to 1, leader changed from <none> to 82f50f4b66f3450aa9f58a0b317f5cea (127.15.255.130). New cstate: current_term: 1 leader_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } health_report { overall_health: UNKNOWN } } }
I20240502 21:25:48.276659 19146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "693a8666bbe54ae1b774f7f4da5ba67e" candidate_uuid: "faadabfb099b4abf90a466395be585b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2877108756304e05bd60ba1186b870da" is_pre_election: true
I20240502 21:25:48.276963 19146 raft_consensus.cc:2454] T 693a8666bbe54ae1b774f7f4da5ba67e P 2877108756304e05bd60ba1186b870da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate faadabfb099b4abf90a466395be585b5 in term 0.
I20240502 21:25:48.277529 19706 leader_election.cc:288] T 693a8666bbe54ae1b774f7f4da5ba67e P 501e3c0d44fe4d70af9ff926d9622417 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9be399440ded4e2ebe6b6e0bf486cbb1 (127.15.255.133:34217), 2877108756304e05bd60ba1186b870da (127.15.255.134:43635), faadabfb099b4abf90a466395be585b5 (127.15.255.129:32983), 82f50f4b66f3450aa9f58a0b317f5cea (127.15.255.130:45139)
I20240502 21:25:48.278554 18610 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d111096a9af46068a2476a6231d6520" candidate_uuid: "2877108756304e05bd60ba1186b870da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" is_pre_election: true
I20240502 21:25:48.278569 19146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "693a8666bbe54ae1b774f7f4da5ba67e" candidate_uuid: "faadabfb099b4abf90a466395be585b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2877108756304e05bd60ba1186b870da"
I20240502 21:25:48.278806 19146 raft_consensus.cc:3037] T 693a8666bbe54ae1b774f7f4da5ba67e P 2877108756304e05bd60ba1186b870da [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:48.279644 19079 leader_election.cc:302] T 2d111096a9af46068a2476a6231d6520 P 2877108756304e05bd60ba1186b870da [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 5 voters: 1 yes votes; 3 no votes. yes voters: 2877108756304e05bd60ba1186b870da; no voters: 82f50f4b66f3450aa9f58a0b317f5cea, c75831edb3b44dd59c00ef151adca49d, faadabfb099b4abf90a466395be585b5
I20240502 21:25:48.280753 19694 raft_consensus.cc:3037] T 2d111096a9af46068a2476a6231d6520 P 2877108756304e05bd60ba1186b870da [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:48.282138 19146 raft_consensus.cc:2454] T 693a8666bbe54ae1b774f7f4da5ba67e P 2877108756304e05bd60ba1186b870da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate faadabfb099b4abf90a466395be585b5 in term 1.
I20240502 21:25:48.283692 19694 raft_consensus.cc:2726] T 2d111096a9af46068a2476a6231d6520 P 2877108756304e05bd60ba1186b870da [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 21:25:48.283694 18742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d111096a9af46068a2476a6231d6520" candidate_uuid: "2877108756304e05bd60ba1186b870da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "501e3c0d44fe4d70af9ff926d9622417" is_pre_election: true
I20240502 21:25:48.283970 18742 raft_consensus.cc:2379] T 2d111096a9af46068a2476a6231d6520 P 501e3c0d44fe4d70af9ff926d9622417 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2877108756304e05bd60ba1186b870da in current term 1: Already voted for candidate 82f50f4b66f3450aa9f58a0b317f5cea in this term.
I20240502 21:25:48.283938 19145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d111096a9af46068a2476a6231d6520" candidate_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2877108756304e05bd60ba1186b870da"
I20240502 21:25:48.284102 19146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d111096a9af46068a2476a6231d6520" candidate_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2877108756304e05bd60ba1186b870da" is_pre_election: true
I20240502 21:25:48.286703 19145 raft_consensus.cc:2454] T 2d111096a9af46068a2476a6231d6520 P 2877108756304e05bd60ba1186b870da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82f50f4b66f3450aa9f58a0b317f5cea in term 1.
I20240502 21:25:48.287179 19725 raft_consensus.cc:686] T 693a8666bbe54ae1b774f7f4da5ba67e P faadabfb099b4abf90a466395be585b5 [term 1 LEADER]: Becoming Leader. State: Replica: faadabfb099b4abf90a466395be585b5, State: Running, Role: LEADER
I20240502 21:25:48.288040 19725 consensus_queue.cc:237] T 693a8666bbe54ae1b774f7f4da5ba67e P faadabfb099b4abf90a466395be585b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } }
I20240502 21:25:48.294867 18322 catalog_manager.cc:5484] T 693a8666bbe54ae1b774f7f4da5ba67e P faadabfb099b4abf90a466395be585b5 reported cstate change: term changed from 0 to 1, leader changed from <none> to faadabfb099b4abf90a466395be585b5 (127.15.255.129). New cstate: current_term: 1 leader_uuid: "faadabfb099b4abf90a466395be585b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } health_report { overall_health: UNKNOWN } } }
I20240502 21:25:48.301970 18473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "693a8666bbe54ae1b774f7f4da5ba67e" candidate_uuid: "501e3c0d44fe4d70af9ff926d9622417" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "faadabfb099b4abf90a466395be585b5" is_pre_election: true
I20240502 21:25:48.302927 19145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "693a8666bbe54ae1b774f7f4da5ba67e" candidate_uuid: "501e3c0d44fe4d70af9ff926d9622417" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2877108756304e05bd60ba1186b870da" is_pre_election: true
I20240502 21:25:48.303156 19145 raft_consensus.cc:2379] T 693a8666bbe54ae1b774f7f4da5ba67e P 2877108756304e05bd60ba1186b870da [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 501e3c0d44fe4d70af9ff926d9622417 in current term 1: Already voted for candidate faadabfb099b4abf90a466395be585b5 in this term.
I20240502 21:25:48.303220 18610 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "693a8666bbe54ae1b774f7f4da5ba67e" candidate_uuid: "501e3c0d44fe4d70af9ff926d9622417" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" is_pre_election: true
I20240502 21:25:48.303460 18610 raft_consensus.cc:2379] T 693a8666bbe54ae1b774f7f4da5ba67e P 82f50f4b66f3450aa9f58a0b317f5cea [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 501e3c0d44fe4d70af9ff926d9622417 in current term 1: Already voted for candidate faadabfb099b4abf90a466395be585b5 in this term.
I20240502 21:25:48.303606 19010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "693a8666bbe54ae1b774f7f4da5ba67e" candidate_uuid: "501e3c0d44fe4d70af9ff926d9622417" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" is_pre_election: true
I20240502 21:25:48.303880 19010 raft_consensus.cc:2379] T 693a8666bbe54ae1b774f7f4da5ba67e P 9be399440ded4e2ebe6b6e0bf486cbb1 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 501e3c0d44fe4d70af9ff926d9622417 in current term 1: Already voted for candidate faadabfb099b4abf90a466395be585b5 in this term.
I20240502 21:25:48.303889 18680 leader_election.cc:302] T 693a8666bbe54ae1b774f7f4da5ba67e P 501e3c0d44fe4d70af9ff926d9622417 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 5 voters: 1 yes votes; 3 no votes. yes voters: 501e3c0d44fe4d70af9ff926d9622417; no voters: 2877108756304e05bd60ba1186b870da, 82f50f4b66f3450aa9f58a0b317f5cea, faadabfb099b4abf90a466395be585b5
I20240502 21:25:48.304282 19706 raft_consensus.cc:2726] T 693a8666bbe54ae1b774f7f4da5ba67e P 501e3c0d44fe4d70af9ff926d9622417 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 21:25:48.588961 19568 consensus_queue.cc:1035] T 51edafdf2a9348548a494119668bfd31 P c75831edb3b44dd59c00ef151adca49d [LEADER]: Connected to new peer: Peer: permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:25:48.648144 19640 consensus_queue.cc:1035] T 2d111096a9af46068a2476a6231d6520 P 82f50f4b66f3450aa9f58a0b317f5cea [LEADER]: Connected to new peer: Peer: permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 21:25:48.663118 19733 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:25:48.663475 19733 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:25:48.671293 19733 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 21:25:48.687784 19640 consensus_queue.cc:1035] T 2d111096a9af46068a2476a6231d6520 P 82f50f4b66f3450aa9f58a0b317f5cea [LEADER]: Connected to new peer: Peer: permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:25:48.692662 19640 consensus_queue.cc:1035] T 2d111096a9af46068a2476a6231d6520 P 82f50f4b66f3450aa9f58a0b317f5cea [LEADER]: Connected to new peer: Peer: permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:25:48.693092 19640 consensus_queue.cc:1035] T 2d111096a9af46068a2476a6231d6520 P 82f50f4b66f3450aa9f58a0b317f5cea [LEADER]: Connected to new peer: Peer: permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_2'
I20240502 21:25:48.732066 19725 consensus_queue.cc:1035] T 693a8666bbe54ae1b774f7f4da5ba67e P faadabfb099b4abf90a466395be585b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:25:48.742940 19568 consensus_queue.cc:1035] T 51edafdf2a9348548a494119668bfd31 P c75831edb3b44dd59c00ef151adca49d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:25:48.761193 19775 consensus_queue.cc:1035] T 51edafdf2a9348548a494119668bfd31 P c75831edb3b44dd59c00ef151adca49d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:25:48.766234 19568 consensus_queue.cc:1035] T 51edafdf2a9348548a494119668bfd31 P c75831edb3b44dd59c00ef151adca49d [LEADER]: Connected to new peer: Peer: permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:25:48.784209 19725 consensus_queue.cc:1035] T 693a8666bbe54ae1b774f7f4da5ba67e P faadabfb099b4abf90a466395be585b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:25:48.795528 19725 consensus_queue.cc:1035] T 693a8666bbe54ae1b774f7f4da5ba67e P faadabfb099b4abf90a466395be585b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:25:48.803386 19782 consensus_queue.cc:1035] T 693a8666bbe54ae1b774f7f4da5ba67e P faadabfb099b4abf90a466395be585b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 375.555 ms
time per row: 0.187777 ms
I20240502 21:25:49.260159 18320 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60616:
name: "rebalance_test_table_3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 5
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 21:25:49.270118 18857 tablet_service.cc:1467] Processing CreateTablet for tablet ebf7371ff0ef4de99ced8fec0468af28 (DEFAULT_TABLE table=rebalance_test_table_3 [id=5af23c390f054d07b5a7985c2611ca30]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:49.270581 18857 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ebf7371ff0ef4de99ced8fec0468af28. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:49.270895 18855 tablet_service.cc:1467] Processing CreateTablet for tablet 4aff104e3b6b424baa628bbe612b62e8 (DEFAULT_TABLE table=rebalance_test_table_3 [id=5af23c390f054d07b5a7985c2611ca30]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:49.271281 18855 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4aff104e3b6b424baa628bbe612b62e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:49.272874 18989 tablet_service.cc:1467] Processing CreateTablet for tablet ebf7371ff0ef4de99ced8fec0468af28 (DEFAULT_TABLE table=rebalance_test_table_3 [id=5af23c390f054d07b5a7985c2611ca30]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:49.273288 18989 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ebf7371ff0ef4de99ced8fec0468af28. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:49.273443 19124 tablet_service.cc:1467] Processing CreateTablet for tablet ebf7371ff0ef4de99ced8fec0468af28 (DEFAULT_TABLE table=rebalance_test_table_3 [id=5af23c390f054d07b5a7985c2611ca30]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:49.273902 19124 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ebf7371ff0ef4de99ced8fec0468af28. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:49.276858 18722 tablet_service.cc:1467] Processing CreateTablet for tablet ebf7371ff0ef4de99ced8fec0468af28 (DEFAULT_TABLE table=rebalance_test_table_3 [id=5af23c390f054d07b5a7985c2611ca30]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:49.276839 18588 tablet_service.cc:1467] Processing CreateTablet for tablet 9844e983dd0a403abe0ac90cd96690d6 (DEFAULT_TABLE table=rebalance_test_table_3 [id=5af23c390f054d07b5a7985c2611ca30]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:49.277261 18588 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9844e983dd0a403abe0ac90cd96690d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:49.277276 18722 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ebf7371ff0ef4de99ced8fec0468af28. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:49.280186 19125 tablet_service.cc:1467] Processing CreateTablet for tablet 4aff104e3b6b424baa628bbe612b62e8 (DEFAULT_TABLE table=rebalance_test_table_3 [id=5af23c390f054d07b5a7985c2611ca30]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:49.280553 19125 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4aff104e3b6b424baa628bbe612b62e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:49.281646 18452 tablet_service.cc:1467] Processing CreateTablet for tablet ebf7371ff0ef4de99ced8fec0468af28 (DEFAULT_TABLE table=rebalance_test_table_3 [id=5af23c390f054d07b5a7985c2611ca30]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:49.281785 18453 tablet_service.cc:1467] Processing CreateTablet for tablet 4aff104e3b6b424baa628bbe612b62e8 (DEFAULT_TABLE table=rebalance_test_table_3 [id=5af23c390f054d07b5a7985c2611ca30]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:49.282083 18452 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ebf7371ff0ef4de99ced8fec0468af28. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:49.282331 18451 tablet_service.cc:1467] Processing CreateTablet for tablet 9844e983dd0a403abe0ac90cd96690d6 (DEFAULT_TABLE table=rebalance_test_table_3 [id=5af23c390f054d07b5a7985c2611ca30]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:49.282653 18451 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9844e983dd0a403abe0ac90cd96690d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:49.285077 18856 tablet_service.cc:1467] Processing CreateTablet for tablet 9844e983dd0a403abe0ac90cd96690d6 (DEFAULT_TABLE table=rebalance_test_table_3 [id=5af23c390f054d07b5a7985c2611ca30]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:49.285328 18988 tablet_service.cc:1467] Processing CreateTablet for tablet 9844e983dd0a403abe0ac90cd96690d6 (DEFAULT_TABLE table=rebalance_test_table_3 [id=5af23c390f054d07b5a7985c2611ca30]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:49.285444 18856 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9844e983dd0a403abe0ac90cd96690d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:49.285741 18988 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9844e983dd0a403abe0ac90cd96690d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:49.288609 18453 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4aff104e3b6b424baa628bbe612b62e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:49.288760 18990 tablet_service.cc:1467] Processing CreateTablet for tablet 4aff104e3b6b424baa628bbe612b62e8 (DEFAULT_TABLE table=rebalance_test_table_3 [id=5af23c390f054d07b5a7985c2611ca30]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:49.289093 18990 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4aff104e3b6b424baa628bbe612b62e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:49.293279 18723 tablet_service.cc:1467] Processing CreateTablet for tablet 4aff104e3b6b424baa628bbe612b62e8 (DEFAULT_TABLE table=rebalance_test_table_3 [id=5af23c390f054d07b5a7985c2611ca30]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:49.293702 18723 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4aff104e3b6b424baa628bbe612b62e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:49.300205 18724 tablet_service.cc:1467] Processing CreateTablet for tablet 9844e983dd0a403abe0ac90cd96690d6 (DEFAULT_TABLE table=rebalance_test_table_3 [id=5af23c390f054d07b5a7985c2611ca30]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:49.300838 18724 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9844e983dd0a403abe0ac90cd96690d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:49.323581 19824 tablet_bootstrap.cc:492] T 9844e983dd0a403abe0ac90cd96690d6 P faadabfb099b4abf90a466395be585b5: Bootstrap starting.
I20240502 21:25:49.324144 19827 tablet_bootstrap.cc:492] T 9844e983dd0a403abe0ac90cd96690d6 P 9be399440ded4e2ebe6b6e0bf486cbb1: Bootstrap starting.
I20240502 21:25:49.325184 19828 tablet_bootstrap.cc:492] T 9844e983dd0a403abe0ac90cd96690d6 P 82f50f4b66f3450aa9f58a0b317f5cea: Bootstrap starting.
I20240502 21:25:49.325184 19826 tablet_bootstrap.cc:492] T 9844e983dd0a403abe0ac90cd96690d6 P c75831edb3b44dd59c00ef151adca49d: Bootstrap starting.
I20240502 21:25:49.326911 19827 tablet_bootstrap.cc:654] T 9844e983dd0a403abe0ac90cd96690d6 P 9be399440ded4e2ebe6b6e0bf486cbb1: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:49.328199 19826 tablet_bootstrap.cc:654] T 9844e983dd0a403abe0ac90cd96690d6 P c75831edb3b44dd59c00ef151adca49d: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:49.328617 19824 tablet_bootstrap.cc:654] T 9844e983dd0a403abe0ac90cd96690d6 P faadabfb099b4abf90a466395be585b5: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:49.328758 19825 tablet_bootstrap.cc:492] T 4aff104e3b6b424baa628bbe612b62e8 P 501e3c0d44fe4d70af9ff926d9622417: Bootstrap starting.
I20240502 21:25:49.330570 19826 tablet_bootstrap.cc:492] T 9844e983dd0a403abe0ac90cd96690d6 P c75831edb3b44dd59c00ef151adca49d: No bootstrap required, opened a new log
I20240502 21:25:49.330785 19826 ts_tablet_manager.cc:1367] T 9844e983dd0a403abe0ac90cd96690d6 P c75831edb3b44dd59c00ef151adca49d: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20240502 21:25:49.331421 19823 tablet_bootstrap.cc:492] T ebf7371ff0ef4de99ced8fec0468af28 P 2877108756304e05bd60ba1186b870da: Bootstrap starting.
I20240502 21:25:49.331770 19825 tablet_bootstrap.cc:654] T 4aff104e3b6b424baa628bbe612b62e8 P 501e3c0d44fe4d70af9ff926d9622417: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:49.331662 19826 raft_consensus.cc:348] T 9844e983dd0a403abe0ac90cd96690d6 P c75831edb3b44dd59c00ef151adca49d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } }
I20240502 21:25:49.332012 19826 raft_consensus.cc:374] T 9844e983dd0a403abe0ac90cd96690d6 P c75831edb3b44dd59c00ef151adca49d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:49.332262 19826 raft_consensus.cc:729] T 9844e983dd0a403abe0ac90cd96690d6 P c75831edb3b44dd59c00ef151adca49d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c75831edb3b44dd59c00ef151adca49d, State: Initialized, Role: FOLLOWER
I20240502 21:25:49.332614 19826 consensus_queue.cc:260] T 9844e983dd0a403abe0ac90cd96690d6 P c75831edb3b44dd59c00ef151adca49d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } }
I20240502 21:25:49.333277 19826 ts_tablet_manager.cc:1398] T 9844e983dd0a403abe0ac90cd96690d6 P c75831edb3b44dd59c00ef151adca49d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:25:49.333681 19828 tablet_bootstrap.cc:654] T 9844e983dd0a403abe0ac90cd96690d6 P 82f50f4b66f3450aa9f58a0b317f5cea: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:49.337785 19823 tablet_bootstrap.cc:654] T ebf7371ff0ef4de99ced8fec0468af28 P 2877108756304e05bd60ba1186b870da: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:49.338186 19827 tablet_bootstrap.cc:492] T 9844e983dd0a403abe0ac90cd96690d6 P 9be399440ded4e2ebe6b6e0bf486cbb1: No bootstrap required, opened a new log
I20240502 21:25:49.338423 19826 tablet_bootstrap.cc:492] T 4aff104e3b6b424baa628bbe612b62e8 P c75831edb3b44dd59c00ef151adca49d: Bootstrap starting.
I20240502 21:25:49.339692 19827 ts_tablet_manager.cc:1367] T 9844e983dd0a403abe0ac90cd96690d6 P 9be399440ded4e2ebe6b6e0bf486cbb1: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.000s
I20240502 21:25:49.340082 19826 tablet_bootstrap.cc:654] T 4aff104e3b6b424baa628bbe612b62e8 P c75831edb3b44dd59c00ef151adca49d: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:49.342306 19824 tablet_bootstrap.cc:492] T 9844e983dd0a403abe0ac90cd96690d6 P faadabfb099b4abf90a466395be585b5: No bootstrap required, opened a new log
I20240502 21:25:49.342765 19824 ts_tablet_manager.cc:1367] T 9844e983dd0a403abe0ac90cd96690d6 P faadabfb099b4abf90a466395be585b5: Time spent bootstrapping tablet: real 0.019s user 0.000s sys 0.004s
I20240502 21:25:49.343917 19824 raft_consensus.cc:348] T 9844e983dd0a403abe0ac90cd96690d6 P faadabfb099b4abf90a466395be585b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } }
I20240502 21:25:49.341532 19827 raft_consensus.cc:348] T 9844e983dd0a403abe0ac90cd96690d6 P 9be399440ded4e2ebe6b6e0bf486cbb1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } }
I20240502 21:25:49.345338 19827 raft_consensus.cc:374] T 9844e983dd0a403abe0ac90cd96690d6 P 9be399440ded4e2ebe6b6e0bf486cbb1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:49.345705 19827 raft_consensus.cc:729] T 9844e983dd0a403abe0ac90cd96690d6 P 9be399440ded4e2ebe6b6e0bf486cbb1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9be399440ded4e2ebe6b6e0bf486cbb1, State: Initialized, Role: FOLLOWER
I20240502 21:25:49.346341 19827 consensus_queue.cc:260] T 9844e983dd0a403abe0ac90cd96690d6 P 9be399440ded4e2ebe6b6e0bf486cbb1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } }
I20240502 21:25:49.348719 19825 tablet_bootstrap.cc:492] T 4aff104e3b6b424baa628bbe612b62e8 P 501e3c0d44fe4d70af9ff926d9622417: No bootstrap required, opened a new log
I20240502 21:25:49.349011 19825 ts_tablet_manager.cc:1367] T 4aff104e3b6b424baa628bbe612b62e8 P 501e3c0d44fe4d70af9ff926d9622417: Time spent bootstrapping tablet: real 0.020s user 0.007s sys 0.000s
I20240502 21:25:49.349138 19827 ts_tablet_manager.cc:1398] T 9844e983dd0a403abe0ac90cd96690d6 P 9be399440ded4e2ebe6b6e0bf486cbb1: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20240502 21:25:49.349381 19824 raft_consensus.cc:374] T 9844e983dd0a403abe0ac90cd96690d6 P faadabfb099b4abf90a466395be585b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:49.349785 19824 raft_consensus.cc:729] T 9844e983dd0a403abe0ac90cd96690d6 P faadabfb099b4abf90a466395be585b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: faadabfb099b4abf90a466395be585b5, State: Initialized, Role: FOLLOWER
I20240502 21:25:49.349938 19825 raft_consensus.cc:348] T 4aff104e3b6b424baa628bbe612b62e8 P 501e3c0d44fe4d70af9ff926d9622417 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } }
I20240502 21:25:49.350246 19825 raft_consensus.cc:374] T 4aff104e3b6b424baa628bbe612b62e8 P 501e3c0d44fe4d70af9ff926d9622417 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:49.350379 19825 raft_consensus.cc:729] T 4aff104e3b6b424baa628bbe612b62e8 P 501e3c0d44fe4d70af9ff926d9622417 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 501e3c0d44fe4d70af9ff926d9622417, State: Initialized, Role: FOLLOWER
I20240502 21:25:49.350687 19825 consensus_queue.cc:260] T 4aff104e3b6b424baa628bbe612b62e8 P 501e3c0d44fe4d70af9ff926d9622417 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } }
I20240502 21:25:49.351410 19827 tablet_bootstrap.cc:492] T 4aff104e3b6b424baa628bbe612b62e8 P 9be399440ded4e2ebe6b6e0bf486cbb1: Bootstrap starting.
I20240502 21:25:49.350392 19824 consensus_queue.cc:260] T 9844e983dd0a403abe0ac90cd96690d6 P faadabfb099b4abf90a466395be585b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } }
I20240502 21:25:49.352196 19826 tablet_bootstrap.cc:492] T 4aff104e3b6b424baa628bbe612b62e8 P c75831edb3b44dd59c00ef151adca49d: No bootstrap required, opened a new log
I20240502 21:25:49.352470 19826 ts_tablet_manager.cc:1367] T 4aff104e3b6b424baa628bbe612b62e8 P c75831edb3b44dd59c00ef151adca49d: Time spent bootstrapping tablet: real 0.019s user 0.004s sys 0.003s
I20240502 21:25:49.354255 19827 tablet_bootstrap.cc:654] T 4aff104e3b6b424baa628bbe612b62e8 P 9be399440ded4e2ebe6b6e0bf486cbb1: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:49.355908 19824 ts_tablet_manager.cc:1398] T 9844e983dd0a403abe0ac90cd96690d6 P faadabfb099b4abf90a466395be585b5: Time spent starting tablet: real 0.013s user 0.000s sys 0.002s
I20240502 21:25:49.358129 19824 tablet_bootstrap.cc:492] T 4aff104e3b6b424baa628bbe612b62e8 P faadabfb099b4abf90a466395be585b5: Bootstrap starting.
I20240502 21:25:49.356891 19823 tablet_bootstrap.cc:492] T ebf7371ff0ef4de99ced8fec0468af28 P 2877108756304e05bd60ba1186b870da: No bootstrap required, opened a new log
I20240502 21:25:49.358376 19823 ts_tablet_manager.cc:1367] T ebf7371ff0ef4de99ced8fec0468af28 P 2877108756304e05bd60ba1186b870da: Time spent bootstrapping tablet: real 0.027s user 0.005s sys 0.000s
I20240502 21:25:49.359045 19825 ts_tablet_manager.cc:1398] T 4aff104e3b6b424baa628bbe612b62e8 P 501e3c0d44fe4d70af9ff926d9622417: Time spent starting tablet: real 0.010s user 0.004s sys 0.000s
I20240502 21:25:49.360205 19823 raft_consensus.cc:348] T ebf7371ff0ef4de99ced8fec0468af28 P 2877108756304e05bd60ba1186b870da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } }
I20240502 21:25:49.362231 19828 tablet_bootstrap.cc:492] T 9844e983dd0a403abe0ac90cd96690d6 P 82f50f4b66f3450aa9f58a0b317f5cea: No bootstrap required, opened a new log
I20240502 21:25:49.362565 19828 ts_tablet_manager.cc:1367] T 9844e983dd0a403abe0ac90cd96690d6 P 82f50f4b66f3450aa9f58a0b317f5cea: Time spent bootstrapping tablet: real 0.037s user 0.006s sys 0.000s
I20240502 21:25:49.363461 19828 raft_consensus.cc:348] T 9844e983dd0a403abe0ac90cd96690d6 P 82f50f4b66f3450aa9f58a0b317f5cea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } }
I20240502 21:25:49.363790 19828 raft_consensus.cc:374] T 9844e983dd0a403abe0ac90cd96690d6 P 82f50f4b66f3450aa9f58a0b317f5cea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:49.363883 19825 tablet_bootstrap.cc:492] T 9844e983dd0a403abe0ac90cd96690d6 P 501e3c0d44fe4d70af9ff926d9622417: Bootstrap starting.
I20240502 21:25:49.364650 19824 tablet_bootstrap.cc:654] T 4aff104e3b6b424baa628bbe612b62e8 P faadabfb099b4abf90a466395be585b5: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:49.364799 19828 raft_consensus.cc:729] T 9844e983dd0a403abe0ac90cd96690d6 P 82f50f4b66f3450aa9f58a0b317f5cea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82f50f4b66f3450aa9f58a0b317f5cea, State: Initialized, Role: FOLLOWER
I20240502 21:25:49.364948 19823 raft_consensus.cc:374] T ebf7371ff0ef4de99ced8fec0468af28 P 2877108756304e05bd60ba1186b870da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:49.365190 19823 raft_consensus.cc:729] T ebf7371ff0ef4de99ced8fec0468af28 P 2877108756304e05bd60ba1186b870da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2877108756304e05bd60ba1186b870da, State: Initialized, Role: FOLLOWER
I20240502 21:25:49.365339 19826 raft_consensus.cc:348] T 4aff104e3b6b424baa628bbe612b62e8 P c75831edb3b44dd59c00ef151adca49d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } }
I20240502 21:25:49.365422 19828 consensus_queue.cc:260] T 9844e983dd0a403abe0ac90cd96690d6 P 82f50f4b66f3450aa9f58a0b317f5cea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } }
I20240502 21:25:49.365561 19825 tablet_bootstrap.cc:654] T 9844e983dd0a403abe0ac90cd96690d6 P 501e3c0d44fe4d70af9ff926d9622417: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:49.365926 19823 consensus_queue.cc:260] T ebf7371ff0ef4de99ced8fec0468af28 P 2877108756304e05bd60ba1186b870da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } }
I20240502 21:25:49.366529 19826 raft_consensus.cc:374] T 4aff104e3b6b424baa628bbe612b62e8 P c75831edb3b44dd59c00ef151adca49d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:49.366815 19826 raft_consensus.cc:729] T 4aff104e3b6b424baa628bbe612b62e8 P c75831edb3b44dd59c00ef151adca49d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c75831edb3b44dd59c00ef151adca49d, State: Initialized, Role: FOLLOWER
I20240502 21:25:49.367348 19826 consensus_queue.cc:260] T 4aff104e3b6b424baa628bbe612b62e8 P c75831edb3b44dd59c00ef151adca49d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } }
I20240502 21:25:49.369029 19828 ts_tablet_manager.cc:1398] T 9844e983dd0a403abe0ac90cd96690d6 P 82f50f4b66f3450aa9f58a0b317f5cea: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240502 21:25:49.369082 19826 ts_tablet_manager.cc:1398] T 4aff104e3b6b424baa628bbe612b62e8 P c75831edb3b44dd59c00ef151adca49d: Time spent starting tablet: real 0.016s user 0.000s sys 0.002s
I20240502 21:25:49.371522 19826 tablet_bootstrap.cc:492] T ebf7371ff0ef4de99ced8fec0468af28 P c75831edb3b44dd59c00ef151adca49d: Bootstrap starting.
I20240502 21:25:49.373461 19825 tablet_bootstrap.cc:492] T 9844e983dd0a403abe0ac90cd96690d6 P 501e3c0d44fe4d70af9ff926d9622417: No bootstrap required, opened a new log
I20240502 21:25:49.373695 19825 ts_tablet_manager.cc:1367] T 9844e983dd0a403abe0ac90cd96690d6 P 501e3c0d44fe4d70af9ff926d9622417: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.000s
I20240502 21:25:49.374480 19825 raft_consensus.cc:348] T 9844e983dd0a403abe0ac90cd96690d6 P 501e3c0d44fe4d70af9ff926d9622417 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } }
I20240502 21:25:49.374858 19825 raft_consensus.cc:374] T 9844e983dd0a403abe0ac90cd96690d6 P 501e3c0d44fe4d70af9ff926d9622417 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:49.374990 19826 tablet_bootstrap.cc:654] T ebf7371ff0ef4de99ced8fec0468af28 P c75831edb3b44dd59c00ef151adca49d: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:49.374979 19825 raft_consensus.cc:729] T 9844e983dd0a403abe0ac90cd96690d6 P 501e3c0d44fe4d70af9ff926d9622417 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 501e3c0d44fe4d70af9ff926d9622417, State: Initialized, Role: FOLLOWER
I20240502 21:25:49.375319 19825 consensus_queue.cc:260] T 9844e983dd0a403abe0ac90cd96690d6 P 501e3c0d44fe4d70af9ff926d9622417 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } }
I20240502 21:25:49.375948 19825 ts_tablet_manager.cc:1398] T 9844e983dd0a403abe0ac90cd96690d6 P 501e3c0d44fe4d70af9ff926d9622417: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:25:49.378901 19827 tablet_bootstrap.cc:492] T 4aff104e3b6b424baa628bbe612b62e8 P 9be399440ded4e2ebe6b6e0bf486cbb1: No bootstrap required, opened a new log
I20240502 21:25:49.379122 19827 ts_tablet_manager.cc:1367] T 4aff104e3b6b424baa628bbe612b62e8 P 9be399440ded4e2ebe6b6e0bf486cbb1: Time spent bootstrapping tablet: real 0.028s user 0.006s sys 0.000s
I20240502 21:25:49.380039 19827 raft_consensus.cc:348] T 4aff104e3b6b424baa628bbe612b62e8 P 9be399440ded4e2ebe6b6e0bf486cbb1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } }
I20240502 21:25:49.380338 19827 raft_consensus.cc:374] T 4aff104e3b6b424baa628bbe612b62e8 P 9be399440ded4e2ebe6b6e0bf486cbb1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:49.380488 19827 raft_consensus.cc:729] T 4aff104e3b6b424baa628bbe612b62e8 P 9be399440ded4e2ebe6b6e0bf486cbb1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9be399440ded4e2ebe6b6e0bf486cbb1, State: Initialized, Role: FOLLOWER
I20240502 21:25:49.380735 19825 tablet_bootstrap.cc:492] T ebf7371ff0ef4de99ced8fec0468af28 P 501e3c0d44fe4d70af9ff926d9622417: Bootstrap starting.
I20240502 21:25:49.380793 19827 consensus_queue.cc:260] T 4aff104e3b6b424baa628bbe612b62e8 P 9be399440ded4e2ebe6b6e0bf486cbb1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } }
I20240502 21:25:49.382084 19827 ts_tablet_manager.cc:1398] T 4aff104e3b6b424baa628bbe612b62e8 P 9be399440ded4e2ebe6b6e0bf486cbb1: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 21:25:49.382387 19827 tablet_bootstrap.cc:492] T ebf7371ff0ef4de99ced8fec0468af28 P 9be399440ded4e2ebe6b6e0bf486cbb1: Bootstrap starting.
I20240502 21:25:49.383239 19824 tablet_bootstrap.cc:492] T 4aff104e3b6b424baa628bbe612b62e8 P faadabfb099b4abf90a466395be585b5: No bootstrap required, opened a new log
I20240502 21:25:49.383459 19824 ts_tablet_manager.cc:1367] T 4aff104e3b6b424baa628bbe612b62e8 P faadabfb099b4abf90a466395be585b5: Time spent bootstrapping tablet: real 0.025s user 0.000s sys 0.003s
I20240502 21:25:49.384652 19749 raft_consensus.cc:482] T 9844e983dd0a403abe0ac90cd96690d6 P 82f50f4b66f3450aa9f58a0b317f5cea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:25:49.384613 19824 raft_consensus.cc:348] T 4aff104e3b6b424baa628bbe612b62e8 P faadabfb099b4abf90a466395be585b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } }
I20240502 21:25:49.384868 19824 raft_consensus.cc:374] T 4aff104e3b6b424baa628bbe612b62e8 P faadabfb099b4abf90a466395be585b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:49.384958 19824 raft_consensus.cc:729] T 4aff104e3b6b424baa628bbe612b62e8 P faadabfb099b4abf90a466395be585b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: faadabfb099b4abf90a466395be585b5, State: Initialized, Role: FOLLOWER
I20240502 21:25:49.384801 19749 raft_consensus.cc:504] T 9844e983dd0a403abe0ac90cd96690d6 P 82f50f4b66f3450aa9f58a0b317f5cea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } }
I20240502 21:25:49.385203 19824 consensus_queue.cc:260] T 4aff104e3b6b424baa628bbe612b62e8 P faadabfb099b4abf90a466395be585b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } }
I20240502 21:25:49.385800 19824 ts_tablet_manager.cc:1398] T 4aff104e3b6b424baa628bbe612b62e8 P faadabfb099b4abf90a466395be585b5: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 21:25:49.385764 19749 leader_election.cc:288] T 9844e983dd0a403abe0ac90cd96690d6 P 82f50f4b66f3450aa9f58a0b317f5cea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers faadabfb099b4abf90a466395be585b5 (127.15.255.129:32983), 501e3c0d44fe4d70af9ff926d9622417 (127.15.255.131:42501), 9be399440ded4e2ebe6b6e0bf486cbb1 (127.15.255.133:34217), c75831edb3b44dd59c00ef151adca49d (127.15.255.132:43607)
I20240502 21:25:49.386097 19824 tablet_bootstrap.cc:492] T ebf7371ff0ef4de99ced8fec0468af28 P faadabfb099b4abf90a466395be585b5: Bootstrap starting.
I20240502 21:25:49.387466 19010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9844e983dd0a403abe0ac90cd96690d6" candidate_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" is_pre_election: true
I20240502 21:25:49.387630 19824 tablet_bootstrap.cc:654] T ebf7371ff0ef4de99ced8fec0468af28 P faadabfb099b4abf90a466395be585b5: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:49.388226 18742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9844e983dd0a403abe0ac90cd96690d6" candidate_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "501e3c0d44fe4d70af9ff926d9622417" is_pre_election: true
I20240502 21:25:49.388492 19826 tablet_bootstrap.cc:492] T ebf7371ff0ef4de99ced8fec0468af28 P c75831edb3b44dd59c00ef151adca49d: No bootstrap required, opened a new log
I20240502 21:25:49.388453 18742 raft_consensus.cc:2454] T 9844e983dd0a403abe0ac90cd96690d6 P 501e3c0d44fe4d70af9ff926d9622417 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82f50f4b66f3450aa9f58a0b317f5cea in term 0.
I20240502 21:25:49.388643 19825 tablet_bootstrap.cc:654] T ebf7371ff0ef4de99ced8fec0468af28 P 501e3c0d44fe4d70af9ff926d9622417: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:49.389125 19826 ts_tablet_manager.cc:1367] T ebf7371ff0ef4de99ced8fec0468af28 P c75831edb3b44dd59c00ef151adca49d: Time spent bootstrapping tablet: real 0.018s user 0.003s sys 0.003s
I20240502 21:25:49.389240 18878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9844e983dd0a403abe0ac90cd96690d6" candidate_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c75831edb3b44dd59c00ef151adca49d" is_pre_election: true
I20240502 21:25:49.389494 18878 raft_consensus.cc:2454] T 9844e983dd0a403abe0ac90cd96690d6 P c75831edb3b44dd59c00ef151adca49d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82f50f4b66f3450aa9f58a0b317f5cea in term 0.
I20240502 21:25:49.389828 19826 raft_consensus.cc:348] T ebf7371ff0ef4de99ced8fec0468af28 P c75831edb3b44dd59c00ef151adca49d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } }
I20240502 21:25:49.390058 19826 raft_consensus.cc:374] T ebf7371ff0ef4de99ced8fec0468af28 P c75831edb3b44dd59c00ef151adca49d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:49.390148 19826 raft_consensus.cc:729] T ebf7371ff0ef4de99ced8fec0468af28 P c75831edb3b44dd59c00ef151adca49d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c75831edb3b44dd59c00ef151adca49d, State: Initialized, Role: FOLLOWER
I20240502 21:25:49.390233 18543 leader_election.cc:302] T 9844e983dd0a403abe0ac90cd96690d6 P 82f50f4b66f3450aa9f58a0b317f5cea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 501e3c0d44fe4d70af9ff926d9622417, 82f50f4b66f3450aa9f58a0b317f5cea, c75831edb3b44dd59c00ef151adca49d; no voters:
I20240502 21:25:49.392822 19749 raft_consensus.cc:2781] T 9844e983dd0a403abe0ac90cd96690d6 P 82f50f4b66f3450aa9f58a0b317f5cea [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:25:49.392947 19749 raft_consensus.cc:482] T 9844e983dd0a403abe0ac90cd96690d6 P 82f50f4b66f3450aa9f58a0b317f5cea [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:25:49.393028 19749 raft_consensus.cc:3037] T 9844e983dd0a403abe0ac90cd96690d6 P 82f50f4b66f3450aa9f58a0b317f5cea [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:49.393949 19826 consensus_queue.cc:260] T ebf7371ff0ef4de99ced8fec0468af28 P c75831edb3b44dd59c00ef151adca49d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } }
I20240502 21:25:49.396564 19749 raft_consensus.cc:504] T 9844e983dd0a403abe0ac90cd96690d6 P 82f50f4b66f3450aa9f58a0b317f5cea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } }
I20240502 21:25:49.398262 18472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9844e983dd0a403abe0ac90cd96690d6" candidate_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "faadabfb099b4abf90a466395be585b5" is_pre_election: true
I20240502 21:25:49.398461 18472 raft_consensus.cc:2454] T 9844e983dd0a403abe0ac90cd96690d6 P faadabfb099b4abf90a466395be585b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82f50f4b66f3450aa9f58a0b317f5cea in term 0.
I20240502 21:25:49.399148 19826 ts_tablet_manager.cc:1398] T ebf7371ff0ef4de99ced8fec0468af28 P c75831edb3b44dd59c00ef151adca49d: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20240502 21:25:49.405231 18472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9844e983dd0a403abe0ac90cd96690d6" candidate_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "faadabfb099b4abf90a466395be585b5"
I20240502 21:25:49.405414 18472 raft_consensus.cc:3037] T 9844e983dd0a403abe0ac90cd96690d6 P faadabfb099b4abf90a466395be585b5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:49.405516 18742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9844e983dd0a403abe0ac90cd96690d6" candidate_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "501e3c0d44fe4d70af9ff926d9622417"
I20240502 21:25:49.405745 18742 raft_consensus.cc:3037] T 9844e983dd0a403abe0ac90cd96690d6 P 501e3c0d44fe4d70af9ff926d9622417 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:49.408200 18472 raft_consensus.cc:2454] T 9844e983dd0a403abe0ac90cd96690d6 P faadabfb099b4abf90a466395be585b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82f50f4b66f3450aa9f58a0b317f5cea in term 1.
I20240502 21:25:49.408408 18742 raft_consensus.cc:2454] T 9844e983dd0a403abe0ac90cd96690d6 P 501e3c0d44fe4d70af9ff926d9622417 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82f50f4b66f3450aa9f58a0b317f5cea in term 1.
I20240502 21:25:49.409487 18543 leader_election.cc:302] T 9844e983dd0a403abe0ac90cd96690d6 P 82f50f4b66f3450aa9f58a0b317f5cea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 501e3c0d44fe4d70af9ff926d9622417, 82f50f4b66f3450aa9f58a0b317f5cea, faadabfb099b4abf90a466395be585b5; no voters:
I20240502 21:25:49.409754 19723 raft_consensus.cc:2781] T 9844e983dd0a403abe0ac90cd96690d6 P 82f50f4b66f3450aa9f58a0b317f5cea [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:25:49.409901 19723 raft_consensus.cc:686] T 9844e983dd0a403abe0ac90cd96690d6 P 82f50f4b66f3450aa9f58a0b317f5cea [term 1 LEADER]: Becoming Leader. State: Replica: 82f50f4b66f3450aa9f58a0b317f5cea, State: Running, Role: LEADER
I20240502 21:25:49.410251 19723 consensus_queue.cc:237] T 9844e983dd0a403abe0ac90cd96690d6 P 82f50f4b66f3450aa9f58a0b317f5cea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } }
I20240502 21:25:49.410845 18878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9844e983dd0a403abe0ac90cd96690d6" candidate_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c75831edb3b44dd59c00ef151adca49d"
I20240502 21:25:49.411026 18878 raft_consensus.cc:3037] T 9844e983dd0a403abe0ac90cd96690d6 P c75831edb3b44dd59c00ef151adca49d [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:49.414186 18878 raft_consensus.cc:2454] T 9844e983dd0a403abe0ac90cd96690d6 P c75831edb3b44dd59c00ef151adca49d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82f50f4b66f3450aa9f58a0b317f5cea in term 1.
I20240502 21:25:49.414832 19749 leader_election.cc:288] T 9844e983dd0a403abe0ac90cd96690d6 P 82f50f4b66f3450aa9f58a0b317f5cea [CANDIDATE]: Term 1 election: Requested vote from peers faadabfb099b4abf90a466395be585b5 (127.15.255.129:32983), 501e3c0d44fe4d70af9ff926d9622417 (127.15.255.131:42501), 9be399440ded4e2ebe6b6e0bf486cbb1 (127.15.255.133:34217), c75831edb3b44dd59c00ef151adca49d (127.15.255.132:43607)
I20240502 21:25:49.419472 19823 ts_tablet_manager.cc:1398] T ebf7371ff0ef4de99ced8fec0468af28 P 2877108756304e05bd60ba1186b870da: Time spent starting tablet: real 0.060s user 0.002s sys 0.003s
I20240502 21:25:49.420675 19823 tablet_bootstrap.cc:492] T 4aff104e3b6b424baa628bbe612b62e8 P 2877108756304e05bd60ba1186b870da: Bootstrap starting.
I20240502 21:25:49.422878 19823 tablet_bootstrap.cc:654] T 4aff104e3b6b424baa628bbe612b62e8 P 2877108756304e05bd60ba1186b870da: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:49.428114 19010 raft_consensus.cc:2454] T 9844e983dd0a403abe0ac90cd96690d6 P 9be399440ded4e2ebe6b6e0bf486cbb1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82f50f4b66f3450aa9f58a0b317f5cea in term 0.
I20240502 21:25:49.431881 19010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9844e983dd0a403abe0ac90cd96690d6" candidate_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1"
I20240502 21:25:49.432121 19010 raft_consensus.cc:3037] T 9844e983dd0a403abe0ac90cd96690d6 P 9be399440ded4e2ebe6b6e0bf486cbb1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:49.435068 19010 raft_consensus.cc:2454] T 9844e983dd0a403abe0ac90cd96690d6 P 9be399440ded4e2ebe6b6e0bf486cbb1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82f50f4b66f3450aa9f58a0b317f5cea in term 1.
I20240502 21:25:49.436776 19827 tablet_bootstrap.cc:654] T ebf7371ff0ef4de99ced8fec0468af28 P 9be399440ded4e2ebe6b6e0bf486cbb1: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:49.440024 19824 tablet_bootstrap.cc:492] T ebf7371ff0ef4de99ced8fec0468af28 P faadabfb099b4abf90a466395be585b5: No bootstrap required, opened a new log
I20240502 21:25:49.440248 19824 ts_tablet_manager.cc:1367] T ebf7371ff0ef4de99ced8fec0468af28 P faadabfb099b4abf90a466395be585b5: Time spent bootstrapping tablet: real 0.054s user 0.006s sys 0.021s
I20240502 21:25:49.439980 18322 catalog_manager.cc:5484] T 9844e983dd0a403abe0ac90cd96690d6 P 82f50f4b66f3450aa9f58a0b317f5cea reported cstate change: term changed from 0 to 1, leader changed from <none> to 82f50f4b66f3450aa9f58a0b317f5cea (127.15.255.130). New cstate: current_term: 1 leader_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } health_report { overall_health: UNKNOWN } } }
I20240502 21:25:49.441138 19824 raft_consensus.cc:348] T ebf7371ff0ef4de99ced8fec0468af28 P faadabfb099b4abf90a466395be585b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } }
I20240502 21:25:49.441398 19824 raft_consensus.cc:374] T ebf7371ff0ef4de99ced8fec0468af28 P faadabfb099b4abf90a466395be585b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:49.441476 19824 raft_consensus.cc:729] T ebf7371ff0ef4de99ced8fec0468af28 P faadabfb099b4abf90a466395be585b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: faadabfb099b4abf90a466395be585b5, State: Initialized, Role: FOLLOWER
I20240502 21:25:49.443239 19823 tablet_bootstrap.cc:492] T 4aff104e3b6b424baa628bbe612b62e8 P 2877108756304e05bd60ba1186b870da: No bootstrap required, opened a new log
I20240502 21:25:49.443444 19823 ts_tablet_manager.cc:1367] T 4aff104e3b6b424baa628bbe612b62e8 P 2877108756304e05bd60ba1186b870da: Time spent bootstrapping tablet: real 0.023s user 0.004s sys 0.001s
I20240502 21:25:49.444131 19825 tablet_bootstrap.cc:492] T ebf7371ff0ef4de99ced8fec0468af28 P 501e3c0d44fe4d70af9ff926d9622417: No bootstrap required, opened a new log
I20240502 21:25:49.444159 19823 raft_consensus.cc:348] T 4aff104e3b6b424baa628bbe612b62e8 P 2877108756304e05bd60ba1186b870da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } }
I20240502 21:25:49.444347 19825 ts_tablet_manager.cc:1367] T ebf7371ff0ef4de99ced8fec0468af28 P 501e3c0d44fe4d70af9ff926d9622417: Time spent bootstrapping tablet: real 0.064s user 0.012s sys 0.015s
I20240502 21:25:49.444525 19823 raft_consensus.cc:374] T 4aff104e3b6b424baa628bbe612b62e8 P 2877108756304e05bd60ba1186b870da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:49.444635 19823 raft_consensus.cc:729] T 4aff104e3b6b424baa628bbe612b62e8 P 2877108756304e05bd60ba1186b870da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2877108756304e05bd60ba1186b870da, State: Initialized, Role: FOLLOWER
I20240502 21:25:49.444929 19823 consensus_queue.cc:260] T 4aff104e3b6b424baa628bbe612b62e8 P 2877108756304e05bd60ba1186b870da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } }
I20240502 21:25:49.445153 19825 raft_consensus.cc:348] T ebf7371ff0ef4de99ced8fec0468af28 P 501e3c0d44fe4d70af9ff926d9622417 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } }
I20240502 21:25:49.445418 19825 raft_consensus.cc:374] T ebf7371ff0ef4de99ced8fec0468af28 P 501e3c0d44fe4d70af9ff926d9622417 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:49.445773 19823 ts_tablet_manager.cc:1398] T 4aff104e3b6b424baa628bbe612b62e8 P 2877108756304e05bd60ba1186b870da: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 21:25:49.446240 19825 raft_consensus.cc:729] T ebf7371ff0ef4de99ced8fec0468af28 P 501e3c0d44fe4d70af9ff926d9622417 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 501e3c0d44fe4d70af9ff926d9622417, State: Initialized, Role: FOLLOWER
I20240502 21:25:49.446697 19827 tablet_bootstrap.cc:492] T ebf7371ff0ef4de99ced8fec0468af28 P 9be399440ded4e2ebe6b6e0bf486cbb1: No bootstrap required, opened a new log
I20240502 21:25:49.446620 19825 consensus_queue.cc:260] T ebf7371ff0ef4de99ced8fec0468af28 P 501e3c0d44fe4d70af9ff926d9622417 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } }
I20240502 21:25:49.446902 19827 ts_tablet_manager.cc:1367] T ebf7371ff0ef4de99ced8fec0468af28 P 9be399440ded4e2ebe6b6e0bf486cbb1: Time spent bootstrapping tablet: real 0.065s user 0.004s sys 0.000s
I20240502 21:25:49.447662 19827 raft_consensus.cc:348] T ebf7371ff0ef4de99ced8fec0468af28 P 9be399440ded4e2ebe6b6e0bf486cbb1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } }
I20240502 21:25:49.447985 19827 raft_consensus.cc:374] T ebf7371ff0ef4de99ced8fec0468af28 P 9be399440ded4e2ebe6b6e0bf486cbb1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:49.448125 19827 raft_consensus.cc:729] T ebf7371ff0ef4de99ced8fec0468af28 P 9be399440ded4e2ebe6b6e0bf486cbb1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9be399440ded4e2ebe6b6e0bf486cbb1, State: Initialized, Role: FOLLOWER
I20240502 21:25:49.448439 19827 consensus_queue.cc:260] T ebf7371ff0ef4de99ced8fec0468af28 P 9be399440ded4e2ebe6b6e0bf486cbb1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } }
I20240502 21:25:49.449651 19825 ts_tablet_manager.cc:1398] T ebf7371ff0ef4de99ced8fec0468af28 P 501e3c0d44fe4d70af9ff926d9622417: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240502 21:25:49.452160 19827 ts_tablet_manager.cc:1398] T ebf7371ff0ef4de99ced8fec0468af28 P 9be399440ded4e2ebe6b6e0bf486cbb1: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240502 21:25:49.452750 19824 consensus_queue.cc:260] T ebf7371ff0ef4de99ced8fec0468af28 P faadabfb099b4abf90a466395be585b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } }
I20240502 21:25:49.453356 19824 ts_tablet_manager.cc:1398] T ebf7371ff0ef4de99ced8fec0468af28 P faadabfb099b4abf90a466395be585b5: Time spent starting tablet: real 0.013s user 0.002s sys 0.000s
I20240502 21:25:49.465194 19776 raft_consensus.cc:482] T ebf7371ff0ef4de99ced8fec0468af28 P c75831edb3b44dd59c00ef151adca49d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:25:49.465334 19776 raft_consensus.cc:504] T ebf7371ff0ef4de99ced8fec0468af28 P c75831edb3b44dd59c00ef151adca49d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } }
I20240502 21:25:49.466159 19776 leader_election.cc:288] T ebf7371ff0ef4de99ced8fec0468af28 P c75831edb3b44dd59c00ef151adca49d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9be399440ded4e2ebe6b6e0bf486cbb1 (127.15.255.133:34217), 501e3c0d44fe4d70af9ff926d9622417 (127.15.255.131:42501), faadabfb099b4abf90a466395be585b5 (127.15.255.129:32983), 2877108756304e05bd60ba1186b870da (127.15.255.134:43635)
I20240502 21:25:49.466697 19145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebf7371ff0ef4de99ced8fec0468af28" candidate_uuid: "c75831edb3b44dd59c00ef151adca49d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2877108756304e05bd60ba1186b870da" is_pre_election: true
I20240502 21:25:49.466814 18742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebf7371ff0ef4de99ced8fec0468af28" candidate_uuid: "c75831edb3b44dd59c00ef151adca49d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "501e3c0d44fe4d70af9ff926d9622417" is_pre_election: true
I20240502 21:25:49.466936 19145 raft_consensus.cc:2454] T ebf7371ff0ef4de99ced8fec0468af28 P 2877108756304e05bd60ba1186b870da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c75831edb3b44dd59c00ef151adca49d in term 0.
I20240502 21:25:49.467020 18742 raft_consensus.cc:2454] T ebf7371ff0ef4de99ced8fec0468af28 P 501e3c0d44fe4d70af9ff926d9622417 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c75831edb3b44dd59c00ef151adca49d in term 0.
I20240502 21:25:49.467438 19010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebf7371ff0ef4de99ced8fec0468af28" candidate_uuid: "c75831edb3b44dd59c00ef151adca49d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" is_pre_election: true
I20240502 21:25:49.467609 19010 raft_consensus.cc:2454] T ebf7371ff0ef4de99ced8fec0468af28 P 9be399440ded4e2ebe6b6e0bf486cbb1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c75831edb3b44dd59c00ef151adca49d in term 0.
I20240502 21:25:49.468060 18809 leader_election.cc:302] T ebf7371ff0ef4de99ced8fec0468af28 P c75831edb3b44dd59c00ef151adca49d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 2877108756304e05bd60ba1186b870da, 501e3c0d44fe4d70af9ff926d9622417, c75831edb3b44dd59c00ef151adca49d; no voters:
I20240502 21:25:49.468314 19776 raft_consensus.cc:2781] T ebf7371ff0ef4de99ced8fec0468af28 P c75831edb3b44dd59c00ef151adca49d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:25:49.468447 19776 raft_consensus.cc:482] T ebf7371ff0ef4de99ced8fec0468af28 P c75831edb3b44dd59c00ef151adca49d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:25:49.468536 19776 raft_consensus.cc:3037] T ebf7371ff0ef4de99ced8fec0468af28 P c75831edb3b44dd59c00ef151adca49d [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:49.468828 18472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebf7371ff0ef4de99ced8fec0468af28" candidate_uuid: "c75831edb3b44dd59c00ef151adca49d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "faadabfb099b4abf90a466395be585b5" is_pre_election: true
I20240502 21:25:49.469058 18472 raft_consensus.cc:2454] T ebf7371ff0ef4de99ced8fec0468af28 P faadabfb099b4abf90a466395be585b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c75831edb3b44dd59c00ef151adca49d in term 0.
I20240502 21:25:49.470185 19642 raft_consensus.cc:482] T 4aff104e3b6b424baa628bbe612b62e8 P 9be399440ded4e2ebe6b6e0bf486cbb1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:25:49.470315 19642 raft_consensus.cc:504] T 4aff104e3b6b424baa628bbe612b62e8 P 9be399440ded4e2ebe6b6e0bf486cbb1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } }
I20240502 21:25:49.471151 19776 raft_consensus.cc:504] T ebf7371ff0ef4de99ced8fec0468af28 P c75831edb3b44dd59c00ef151adca49d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } }
I20240502 21:25:49.471282 19642 leader_election.cc:288] T 4aff104e3b6b424baa628bbe612b62e8 P 9be399440ded4e2ebe6b6e0bf486cbb1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c75831edb3b44dd59c00ef151adca49d (127.15.255.132:43607), 501e3c0d44fe4d70af9ff926d9622417 (127.15.255.131:42501), faadabfb099b4abf90a466395be585b5 (127.15.255.129:32983), 2877108756304e05bd60ba1186b870da (127.15.255.134:43635)
I20240502 21:25:49.472175 18878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4aff104e3b6b424baa628bbe612b62e8" candidate_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c75831edb3b44dd59c00ef151adca49d" is_pre_election: true
I20240502 21:25:49.472378 18878 raft_consensus.cc:2454] T 4aff104e3b6b424baa628bbe612b62e8 P c75831edb3b44dd59c00ef151adca49d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9be399440ded4e2ebe6b6e0bf486cbb1 in term 0.
I20240502 21:25:49.473326 18742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4aff104e3b6b424baa628bbe612b62e8" candidate_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "501e3c0d44fe4d70af9ff926d9622417" is_pre_election: true
I20240502 21:25:49.473548 18742 raft_consensus.cc:2454] T 4aff104e3b6b424baa628bbe612b62e8 P 501e3c0d44fe4d70af9ff926d9622417 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9be399440ded4e2ebe6b6e0bf486cbb1 in term 0.
I20240502 21:25:49.473903 19146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebf7371ff0ef4de99ced8fec0468af28" candidate_uuid: "c75831edb3b44dd59c00ef151adca49d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2877108756304e05bd60ba1186b870da"
I20240502 21:25:49.473951 18943 leader_election.cc:302] T 4aff104e3b6b424baa628bbe612b62e8 P 9be399440ded4e2ebe6b6e0bf486cbb1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 501e3c0d44fe4d70af9ff926d9622417, 9be399440ded4e2ebe6b6e0bf486cbb1, c75831edb3b44dd59c00ef151adca49d; no voters:
I20240502 21:25:49.474086 19146 raft_consensus.cc:3037] T ebf7371ff0ef4de99ced8fec0468af28 P 2877108756304e05bd60ba1186b870da [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:49.474463 19145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4aff104e3b6b424baa628bbe612b62e8" candidate_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2877108756304e05bd60ba1186b870da" is_pre_election: true
I20240502 21:25:49.474644 19145 raft_consensus.cc:2454] T 4aff104e3b6b424baa628bbe612b62e8 P 2877108756304e05bd60ba1186b870da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9be399440ded4e2ebe6b6e0bf486cbb1 in term 0.
I20240502 21:25:49.475596 18472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4aff104e3b6b424baa628bbe612b62e8" candidate_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "faadabfb099b4abf90a466395be585b5" is_pre_election: true
I20240502 21:25:49.475809 18472 raft_consensus.cc:2454] T 4aff104e3b6b424baa628bbe612b62e8 P faadabfb099b4abf90a466395be585b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9be399440ded4e2ebe6b6e0bf486cbb1 in term 0.
I20240502 21:25:49.476661 19146 raft_consensus.cc:2454] T ebf7371ff0ef4de99ced8fec0468af28 P 2877108756304e05bd60ba1186b870da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c75831edb3b44dd59c00ef151adca49d in term 1.
I20240502 21:25:49.476933 19776 leader_election.cc:288] T ebf7371ff0ef4de99ced8fec0468af28 P c75831edb3b44dd59c00ef151adca49d [CANDIDATE]: Term 1 election: Requested vote from peers 9be399440ded4e2ebe6b6e0bf486cbb1 (127.15.255.133:34217), 501e3c0d44fe4d70af9ff926d9622417 (127.15.255.131:42501), faadabfb099b4abf90a466395be585b5 (127.15.255.129:32983), 2877108756304e05bd60ba1186b870da (127.15.255.134:43635)
I20240502 21:25:49.477948 19010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebf7371ff0ef4de99ced8fec0468af28" candidate_uuid: "c75831edb3b44dd59c00ef151adca49d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1"
I20240502 21:25:49.478106 19010 raft_consensus.cc:3037] T ebf7371ff0ef4de99ced8fec0468af28 P 9be399440ded4e2ebe6b6e0bf486cbb1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:49.479959 19010 raft_consensus.cc:2454] T ebf7371ff0ef4de99ced8fec0468af28 P 9be399440ded4e2ebe6b6e0bf486cbb1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c75831edb3b44dd59c00ef151adca49d in term 1.
I20240502 21:25:49.480641 18472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebf7371ff0ef4de99ced8fec0468af28" candidate_uuid: "c75831edb3b44dd59c00ef151adca49d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "faadabfb099b4abf90a466395be585b5"
I20240502 21:25:49.480844 18472 raft_consensus.cc:3037] T ebf7371ff0ef4de99ced8fec0468af28 P faadabfb099b4abf90a466395be585b5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:49.482795 18472 raft_consensus.cc:2454] T ebf7371ff0ef4de99ced8fec0468af28 P faadabfb099b4abf90a466395be585b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c75831edb3b44dd59c00ef151adca49d in term 1.
I20240502 21:25:49.483376 18742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebf7371ff0ef4de99ced8fec0468af28" candidate_uuid: "c75831edb3b44dd59c00ef151adca49d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "501e3c0d44fe4d70af9ff926d9622417"
I20240502 21:25:49.483547 18742 raft_consensus.cc:3037] T ebf7371ff0ef4de99ced8fec0468af28 P 501e3c0d44fe4d70af9ff926d9622417 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:49.485481 18742 raft_consensus.cc:2454] T ebf7371ff0ef4de99ced8fec0468af28 P 501e3c0d44fe4d70af9ff926d9622417 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c75831edb3b44dd59c00ef151adca49d in term 1.
I20240502 21:25:49.486104 18809 leader_election.cc:302] T ebf7371ff0ef4de99ced8fec0468af28 P c75831edb3b44dd59c00ef151adca49d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 2877108756304e05bd60ba1186b870da, 501e3c0d44fe4d70af9ff926d9622417, c75831edb3b44dd59c00ef151adca49d; no voters:
I20240502 21:25:49.486351 19776 raft_consensus.cc:2781] T ebf7371ff0ef4de99ced8fec0468af28 P c75831edb3b44dd59c00ef151adca49d [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:25:49.486631 19776 raft_consensus.cc:686] T ebf7371ff0ef4de99ced8fec0468af28 P c75831edb3b44dd59c00ef151adca49d [term 1 LEADER]: Becoming Leader. State: Replica: c75831edb3b44dd59c00ef151adca49d, State: Running, Role: LEADER
I20240502 21:25:49.486990 19642 raft_consensus.cc:2781] T 4aff104e3b6b424baa628bbe612b62e8 P 9be399440ded4e2ebe6b6e0bf486cbb1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:25:49.487098 19642 raft_consensus.cc:482] T 4aff104e3b6b424baa628bbe612b62e8 P 9be399440ded4e2ebe6b6e0bf486cbb1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:25:49.486991 19776 consensus_queue.cc:237] T ebf7371ff0ef4de99ced8fec0468af28 P c75831edb3b44dd59c00ef151adca49d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } }
I20240502 21:25:49.487174 19642 raft_consensus.cc:3037] T 4aff104e3b6b424baa628bbe612b62e8 P 9be399440ded4e2ebe6b6e0bf486cbb1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:49.489833 19642 raft_consensus.cc:504] T 4aff104e3b6b424baa628bbe612b62e8 P 9be399440ded4e2ebe6b6e0bf486cbb1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } }
I20240502 21:25:49.491195 18878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4aff104e3b6b424baa628bbe612b62e8" candidate_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c75831edb3b44dd59c00ef151adca49d"
I20240502 21:25:49.491220 18742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4aff104e3b6b424baa628bbe612b62e8" candidate_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "501e3c0d44fe4d70af9ff926d9622417"
I20240502 21:25:49.491400 18878 raft_consensus.cc:3037] T 4aff104e3b6b424baa628bbe612b62e8 P c75831edb3b44dd59c00ef151adca49d [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:49.491397 18742 raft_consensus.cc:3037] T 4aff104e3b6b424baa628bbe612b62e8 P 501e3c0d44fe4d70af9ff926d9622417 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:49.494086 19642 leader_election.cc:288] T 4aff104e3b6b424baa628bbe612b62e8 P 9be399440ded4e2ebe6b6e0bf486cbb1 [CANDIDATE]: Term 1 election: Requested vote from peers c75831edb3b44dd59c00ef151adca49d (127.15.255.132:43607), 501e3c0d44fe4d70af9ff926d9622417 (127.15.255.131:42501), faadabfb099b4abf90a466395be585b5 (127.15.255.129:32983), 2877108756304e05bd60ba1186b870da (127.15.255.134:43635)
I20240502 21:25:49.494637 18878 raft_consensus.cc:2454] T 4aff104e3b6b424baa628bbe612b62e8 P c75831edb3b44dd59c00ef151adca49d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9be399440ded4e2ebe6b6e0bf486cbb1 in term 1.
I20240502 21:25:49.494843 19145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4aff104e3b6b424baa628bbe612b62e8" candidate_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2877108756304e05bd60ba1186b870da"
I20240502 21:25:49.493929 18742 raft_consensus.cc:2454] T 4aff104e3b6b424baa628bbe612b62e8 P 501e3c0d44fe4d70af9ff926d9622417 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9be399440ded4e2ebe6b6e0bf486cbb1 in term 1.
I20240502 21:25:49.495038 19145 raft_consensus.cc:3037] T 4aff104e3b6b424baa628bbe612b62e8 P 2877108756304e05bd60ba1186b870da [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:49.494446 18472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4aff104e3b6b424baa628bbe612b62e8" candidate_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "faadabfb099b4abf90a466395be585b5"
I20240502 21:25:49.495543 18472 raft_consensus.cc:3037] T 4aff104e3b6b424baa628bbe612b62e8 P faadabfb099b4abf90a466395be585b5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:49.495687 18320 catalog_manager.cc:5484] T ebf7371ff0ef4de99ced8fec0468af28 P c75831edb3b44dd59c00ef151adca49d reported cstate change: term changed from 0 to 1, leader changed from <none> to c75831edb3b44dd59c00ef151adca49d (127.15.255.132). New cstate: current_term: 1 leader_uuid: "c75831edb3b44dd59c00ef151adca49d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } health_report { overall_health: UNKNOWN } } }
I20240502 21:25:49.496954 18943 leader_election.cc:302] T 4aff104e3b6b424baa628bbe612b62e8 P 9be399440ded4e2ebe6b6e0bf486cbb1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 501e3c0d44fe4d70af9ff926d9622417, 9be399440ded4e2ebe6b6e0bf486cbb1, c75831edb3b44dd59c00ef151adca49d; no voters:
I20240502 21:25:49.497197 19145 raft_consensus.cc:2454] T 4aff104e3b6b424baa628bbe612b62e8 P 2877108756304e05bd60ba1186b870da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9be399440ded4e2ebe6b6e0bf486cbb1 in term 1.
I20240502 21:25:49.497295 19642 raft_consensus.cc:2781] T 4aff104e3b6b424baa628bbe612b62e8 P 9be399440ded4e2ebe6b6e0bf486cbb1 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:25:49.497881 19642 raft_consensus.cc:686] T 4aff104e3b6b424baa628bbe612b62e8 P 9be399440ded4e2ebe6b6e0bf486cbb1 [term 1 LEADER]: Becoming Leader. State: Replica: 9be399440ded4e2ebe6b6e0bf486cbb1, State: Running, Role: LEADER
I20240502 21:25:49.498080 18472 raft_consensus.cc:2454] T 4aff104e3b6b424baa628bbe612b62e8 P faadabfb099b4abf90a466395be585b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9be399440ded4e2ebe6b6e0bf486cbb1 in term 1.
I20240502 21:25:49.498297 19642 consensus_queue.cc:237] T 4aff104e3b6b424baa628bbe612b62e8 P 9be399440ded4e2ebe6b6e0bf486cbb1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } }
I20240502 21:25:49.501626 18320 catalog_manager.cc:5484] T 4aff104e3b6b424baa628bbe612b62e8 P 9be399440ded4e2ebe6b6e0bf486cbb1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9be399440ded4e2ebe6b6e0bf486cbb1 (127.15.255.133). New cstate: current_term: 1 leader_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } health_report { overall_health: HEALTHY } } }
I20240502 21:25:49.810765 19723 consensus_queue.cc:1035] T 9844e983dd0a403abe0ac90cd96690d6 P 82f50f4b66f3450aa9f58a0b317f5cea [LEADER]: Connected to new peer: Peer: permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:25:49.827924 19723 consensus_queue.cc:1035] T 9844e983dd0a403abe0ac90cd96690d6 P 82f50f4b66f3450aa9f58a0b317f5cea [LEADER]: Connected to new peer: Peer: permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:25:49.834589 19723 consensus_queue.cc:1035] T 9844e983dd0a403abe0ac90cd96690d6 P 82f50f4b66f3450aa9f58a0b317f5cea [LEADER]: Connected to new peer: Peer: permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:25:49.841379 19858 consensus_queue.cc:1035] T 9844e983dd0a403abe0ac90cd96690d6 P 82f50f4b66f3450aa9f58a0b317f5cea [LEADER]: Connected to new peer: Peer: permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 21:25:49.867182 19853 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:25:49.867507 19853 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:25:49.875068 19853 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 21:25:49.879714 19642 consensus_queue.cc:1035] T 4aff104e3b6b424baa628bbe612b62e8 P 9be399440ded4e2ebe6b6e0bf486cbb1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:25:49.885164 19642 consensus_queue.cc:1035] T 4aff104e3b6b424baa628bbe612b62e8 P 9be399440ded4e2ebe6b6e0bf486cbb1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:25:49.897713 19642 consensus_queue.cc:1035] T 4aff104e3b6b424baa628bbe612b62e8 P 9be399440ded4e2ebe6b6e0bf486cbb1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:25:49.898155 19642 consensus_queue.cc:1035] T 4aff104e3b6b424baa628bbe612b62e8 P 9be399440ded4e2ebe6b6e0bf486cbb1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_3'
I20240502 21:25:50.046402 19776 consensus_queue.cc:1035] T ebf7371ff0ef4de99ced8fec0468af28 P c75831edb3b44dd59c00ef151adca49d [LEADER]: Connected to new peer: Peer: permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:25:50.050323 19776 consensus_queue.cc:1035] T ebf7371ff0ef4de99ced8fec0468af28 P c75831edb3b44dd59c00ef151adca49d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:25:50.096009 19903 consensus_queue.cc:1035] T ebf7371ff0ef4de99ced8fec0468af28 P c75831edb3b44dd59c00ef151adca49d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:25:50.105463 19903 consensus_queue.cc:1035] T ebf7371ff0ef4de99ced8fec0468af28 P c75831edb3b44dd59c00ef151adca49d [LEADER]: Connected to new peer: Peer: permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 353.516 ms
time per row: 0.176758 ms
W20240502 21:25:50.904676 18406 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.255.135:43395: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240502 21:25:50.905988 18406 consensus_peers.cc:485] T 40d4f61d0f6d4d31a5626c7743c88133 P faadabfb099b4abf90a466395be585b5 -> Peer 90e329b457c24caf93a9f880345eb288 (127.15.255.135:43395): Couldn't send request to peer 90e329b457c24caf93a9f880345eb288. Status: Network error: Client connection negotiation failed: client connection to 127.15.255.135:43395: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240502 21:25:51.269654 18321 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60616:
name: "rebalance_test_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 5
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 21:25:51.278839 18588 tablet_service.cc:1467] Processing CreateTablet for tablet 1df9d3ad5ddc438798543900b6c67b46 (DEFAULT_TABLE table=rebalance_test_table_4 [id=c79c94b660074ffc94fcea415b4caae8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:51.279305 18588 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1df9d3ad5ddc438798543900b6c67b46. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:51.279292 18451 tablet_service.cc:1467] Processing CreateTablet for tablet 1df9d3ad5ddc438798543900b6c67b46 (DEFAULT_TABLE table=rebalance_test_table_4 [id=c79c94b660074ffc94fcea415b4caae8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:51.279433 19124 tablet_service.cc:1467] Processing CreateTablet for tablet 1df9d3ad5ddc438798543900b6c67b46 (DEFAULT_TABLE table=rebalance_test_table_4 [id=c79c94b660074ffc94fcea415b4caae8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:51.279736 18451 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1df9d3ad5ddc438798543900b6c67b46. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:51.279819 19124 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1df9d3ad5ddc438798543900b6c67b46. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:51.284322 18723 tablet_service.cc:1467] Processing CreateTablet for tablet 1df9d3ad5ddc438798543900b6c67b46 (DEFAULT_TABLE table=rebalance_test_table_4 [id=c79c94b660074ffc94fcea415b4caae8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:51.284771 18723 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1df9d3ad5ddc438798543900b6c67b46. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:51.286777 19946 tablet_bootstrap.cc:492] T 1df9d3ad5ddc438798543900b6c67b46 P 82f50f4b66f3450aa9f58a0b317f5cea: Bootstrap starting.
I20240502 21:25:51.288372 19946 tablet_bootstrap.cc:654] T 1df9d3ad5ddc438798543900b6c67b46 P 82f50f4b66f3450aa9f58a0b317f5cea: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:51.288645 19947 tablet_bootstrap.cc:492] T 1df9d3ad5ddc438798543900b6c67b46 P faadabfb099b4abf90a466395be585b5: Bootstrap starting.
I20240502 21:25:51.289498 18856 tablet_service.cc:1467] Processing CreateTablet for tablet 1df9d3ad5ddc438798543900b6c67b46 (DEFAULT_TABLE table=rebalance_test_table_4 [id=c79c94b660074ffc94fcea415b4caae8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:51.289917 18856 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1df9d3ad5ddc438798543900b6c67b46. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:51.289924 18988 tablet_service.cc:1467] Processing CreateTablet for tablet b0852f54089944ce935f8ee4cbf2bc5a (DEFAULT_TABLE table=rebalance_test_table_4 [id=c79c94b660074ffc94fcea415b4caae8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:51.290290 18988 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0852f54089944ce935f8ee4cbf2bc5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:51.291159 19125 tablet_service.cc:1467] Processing CreateTablet for tablet b0852f54089944ce935f8ee4cbf2bc5a (DEFAULT_TABLE table=rebalance_test_table_4 [id=c79c94b660074ffc94fcea415b4caae8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:51.291529 19125 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0852f54089944ce935f8ee4cbf2bc5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:51.292831 18588 tablet_service.cc:1467] Processing CreateTablet for tablet b0852f54089944ce935f8ee4cbf2bc5a (DEFAULT_TABLE table=rebalance_test_table_4 [id=c79c94b660074ffc94fcea415b4caae8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:51.293174 18588 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0852f54089944ce935f8ee4cbf2bc5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:51.294834 18989 tablet_service.cc:1467] Processing CreateTablet for tablet 6f0107b0690c4c64b0a4f2472a1c8322 (DEFAULT_TABLE table=rebalance_test_table_4 [id=c79c94b660074ffc94fcea415b4caae8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:51.295477 19123 tablet_service.cc:1467] Processing CreateTablet for tablet 6f0107b0690c4c64b0a4f2472a1c8322 (DEFAULT_TABLE table=rebalance_test_table_4 [id=c79c94b660074ffc94fcea415b4caae8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:51.295833 19123 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f0107b0690c4c64b0a4f2472a1c8322. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:51.296644 18452 tablet_service.cc:1467] Processing CreateTablet for tablet 6f0107b0690c4c64b0a4f2472a1c8322 (DEFAULT_TABLE table=rebalance_test_table_4 [id=c79c94b660074ffc94fcea415b4caae8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:51.296763 18590 tablet_service.cc:1467] Processing CreateTablet for tablet 6f0107b0690c4c64b0a4f2472a1c8322 (DEFAULT_TABLE table=rebalance_test_table_4 [id=c79c94b660074ffc94fcea415b4caae8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:51.297014 18452 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f0107b0690c4c64b0a4f2472a1c8322. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:51.297091 18590 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f0107b0690c4c64b0a4f2472a1c8322. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:51.301368 18857 tablet_service.cc:1467] Processing CreateTablet for tablet 6f0107b0690c4c64b0a4f2472a1c8322 (DEFAULT_TABLE table=rebalance_test_table_4 [id=c79c94b660074ffc94fcea415b4caae8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:51.301708 18857 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f0107b0690c4c64b0a4f2472a1c8322. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:51.302304 19949 tablet_bootstrap.cc:492] T 1df9d3ad5ddc438798543900b6c67b46 P 2877108756304e05bd60ba1186b870da: Bootstrap starting.
I20240502 21:25:51.303874 19949 tablet_bootstrap.cc:654] T 1df9d3ad5ddc438798543900b6c67b46 P 2877108756304e05bd60ba1186b870da: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:51.304641 18989 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f0107b0690c4c64b0a4f2472a1c8322. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:51.309497 18722 tablet_service.cc:1467] Processing CreateTablet for tablet b0852f54089944ce935f8ee4cbf2bc5a (DEFAULT_TABLE table=rebalance_test_table_4 [id=c79c94b660074ffc94fcea415b4caae8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:51.309902 18722 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0852f54089944ce935f8ee4cbf2bc5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:51.310300 19947 tablet_bootstrap.cc:654] T 1df9d3ad5ddc438798543900b6c67b46 P faadabfb099b4abf90a466395be585b5: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:51.313397 19947 tablet_bootstrap.cc:492] T 1df9d3ad5ddc438798543900b6c67b46 P faadabfb099b4abf90a466395be585b5: No bootstrap required, opened a new log
I20240502 21:25:51.314050 19947 ts_tablet_manager.cc:1367] T 1df9d3ad5ddc438798543900b6c67b46 P faadabfb099b4abf90a466395be585b5: Time spent bootstrapping tablet: real 0.026s user 0.003s sys 0.000s
I20240502 21:25:51.314978 19947 raft_consensus.cc:348] T 1df9d3ad5ddc438798543900b6c67b46 P faadabfb099b4abf90a466395be585b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } }
I20240502 21:25:51.315821 19952 tablet_bootstrap.cc:492] T 1df9d3ad5ddc438798543900b6c67b46 P c75831edb3b44dd59c00ef151adca49d: Bootstrap starting.
I20240502 21:25:51.317839 19952 tablet_bootstrap.cc:654] T 1df9d3ad5ddc438798543900b6c67b46 P c75831edb3b44dd59c00ef151adca49d: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:51.317978 19947 raft_consensus.cc:374] T 1df9d3ad5ddc438798543900b6c67b46 P faadabfb099b4abf90a466395be585b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:51.318128 19947 raft_consensus.cc:729] T 1df9d3ad5ddc438798543900b6c67b46 P faadabfb099b4abf90a466395be585b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: faadabfb099b4abf90a466395be585b5, State: Initialized, Role: FOLLOWER
I20240502 21:25:51.318835 19947 consensus_queue.cc:260] T 1df9d3ad5ddc438798543900b6c67b46 P faadabfb099b4abf90a466395be585b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } }
I20240502 21:25:51.319489 19947 ts_tablet_manager.cc:1398] T 1df9d3ad5ddc438798543900b6c67b46 P faadabfb099b4abf90a466395be585b5: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20240502 21:25:51.319926 19950 tablet_bootstrap.cc:492] T 1df9d3ad5ddc438798543900b6c67b46 P 501e3c0d44fe4d70af9ff926d9622417: Bootstrap starting.
I20240502 21:25:51.321365 19946 tablet_bootstrap.cc:492] T 1df9d3ad5ddc438798543900b6c67b46 P 82f50f4b66f3450aa9f58a0b317f5cea: No bootstrap required, opened a new log
I20240502 21:25:51.321553 19946 ts_tablet_manager.cc:1367] T 1df9d3ad5ddc438798543900b6c67b46 P 82f50f4b66f3450aa9f58a0b317f5cea: Time spent bootstrapping tablet: real 0.035s user 0.000s sys 0.013s
I20240502 21:25:51.322250 19946 raft_consensus.cc:348] T 1df9d3ad5ddc438798543900b6c67b46 P 82f50f4b66f3450aa9f58a0b317f5cea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } }
I20240502 21:25:51.322476 19946 raft_consensus.cc:374] T 1df9d3ad5ddc438798543900b6c67b46 P 82f50f4b66f3450aa9f58a0b317f5cea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:51.322556 19946 raft_consensus.cc:729] T 1df9d3ad5ddc438798543900b6c67b46 P 82f50f4b66f3450aa9f58a0b317f5cea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82f50f4b66f3450aa9f58a0b317f5cea, State: Initialized, Role: FOLLOWER
I20240502 21:25:51.322853 19950 tablet_bootstrap.cc:654] T 1df9d3ad5ddc438798543900b6c67b46 P 501e3c0d44fe4d70af9ff926d9622417: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:51.322757 19946 consensus_queue.cc:260] T 1df9d3ad5ddc438798543900b6c67b46 P 82f50f4b66f3450aa9f58a0b317f5cea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } }
I20240502 21:25:51.324023 19946 ts_tablet_manager.cc:1398] T 1df9d3ad5ddc438798543900b6c67b46 P 82f50f4b66f3450aa9f58a0b317f5cea: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 21:25:51.324328 19946 tablet_bootstrap.cc:492] T b0852f54089944ce935f8ee4cbf2bc5a P 82f50f4b66f3450aa9f58a0b317f5cea: Bootstrap starting.
I20240502 21:25:51.325707 19949 tablet_bootstrap.cc:492] T 1df9d3ad5ddc438798543900b6c67b46 P 2877108756304e05bd60ba1186b870da: No bootstrap required, opened a new log
I20240502 21:25:51.326203 19949 ts_tablet_manager.cc:1367] T 1df9d3ad5ddc438798543900b6c67b46 P 2877108756304e05bd60ba1186b870da: Time spent bootstrapping tablet: real 0.024s user 0.000s sys 0.005s
I20240502 21:25:51.327016 19946 tablet_bootstrap.cc:654] T b0852f54089944ce935f8ee4cbf2bc5a P 82f50f4b66f3450aa9f58a0b317f5cea: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:51.327733 19949 raft_consensus.cc:348] T 1df9d3ad5ddc438798543900b6c67b46 P 2877108756304e05bd60ba1186b870da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } }
I20240502 21:25:51.328742 19949 raft_consensus.cc:374] T 1df9d3ad5ddc438798543900b6c67b46 P 2877108756304e05bd60ba1186b870da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:51.328874 19949 raft_consensus.cc:729] T 1df9d3ad5ddc438798543900b6c67b46 P 2877108756304e05bd60ba1186b870da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2877108756304e05bd60ba1186b870da, State: Initialized, Role: FOLLOWER
I20240502 21:25:51.329136 19949 consensus_queue.cc:260] T 1df9d3ad5ddc438798543900b6c67b46 P 2877108756304e05bd60ba1186b870da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } }
I20240502 21:25:51.335348 19952 tablet_bootstrap.cc:492] T 1df9d3ad5ddc438798543900b6c67b46 P c75831edb3b44dd59c00ef151adca49d: No bootstrap required, opened a new log
I20240502 21:25:51.335497 19952 ts_tablet_manager.cc:1367] T 1df9d3ad5ddc438798543900b6c67b46 P c75831edb3b44dd59c00ef151adca49d: Time spent bootstrapping tablet: real 0.020s user 0.004s sys 0.004s
I20240502 21:25:51.336167 19952 raft_consensus.cc:348] T 1df9d3ad5ddc438798543900b6c67b46 P c75831edb3b44dd59c00ef151adca49d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } }
I20240502 21:25:51.336405 19947 tablet_bootstrap.cc:492] T 6f0107b0690c4c64b0a4f2472a1c8322 P faadabfb099b4abf90a466395be585b5: Bootstrap starting.
I20240502 21:25:51.336474 19952 raft_consensus.cc:374] T 1df9d3ad5ddc438798543900b6c67b46 P c75831edb3b44dd59c00ef151adca49d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:51.336575 19952 raft_consensus.cc:729] T 1df9d3ad5ddc438798543900b6c67b46 P c75831edb3b44dd59c00ef151adca49d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c75831edb3b44dd59c00ef151adca49d, State: Initialized, Role: FOLLOWER
I20240502 21:25:51.336815 19952 consensus_queue.cc:260] T 1df9d3ad5ddc438798543900b6c67b46 P c75831edb3b44dd59c00ef151adca49d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } }
I20240502 21:25:51.337407 19952 ts_tablet_manager.cc:1398] T 1df9d3ad5ddc438798543900b6c67b46 P c75831edb3b44dd59c00ef151adca49d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:25:51.337993 19947 tablet_bootstrap.cc:654] T 6f0107b0690c4c64b0a4f2472a1c8322 P faadabfb099b4abf90a466395be585b5: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:51.338246 19946 tablet_bootstrap.cc:492] T b0852f54089944ce935f8ee4cbf2bc5a P 82f50f4b66f3450aa9f58a0b317f5cea: No bootstrap required, opened a new log
I20240502 21:25:51.338382 19946 ts_tablet_manager.cc:1367] T b0852f54089944ce935f8ee4cbf2bc5a P 82f50f4b66f3450aa9f58a0b317f5cea: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.001s
I20240502 21:25:51.339043 19946 raft_consensus.cc:348] T b0852f54089944ce935f8ee4cbf2bc5a P 82f50f4b66f3450aa9f58a0b317f5cea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } }
I20240502 21:25:51.339270 19946 raft_consensus.cc:374] T b0852f54089944ce935f8ee4cbf2bc5a P 82f50f4b66f3450aa9f58a0b317f5cea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:51.339344 19946 raft_consensus.cc:729] T b0852f54089944ce935f8ee4cbf2bc5a P 82f50f4b66f3450aa9f58a0b317f5cea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82f50f4b66f3450aa9f58a0b317f5cea, State: Initialized, Role: FOLLOWER
I20240502 21:25:51.339568 19946 consensus_queue.cc:260] T b0852f54089944ce935f8ee4cbf2bc5a P 82f50f4b66f3450aa9f58a0b317f5cea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } }
I20240502 21:25:51.340102 19946 ts_tablet_manager.cc:1398] T b0852f54089944ce935f8ee4cbf2bc5a P 82f50f4b66f3450aa9f58a0b317f5cea: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 21:25:51.340391 19946 tablet_bootstrap.cc:492] T 6f0107b0690c4c64b0a4f2472a1c8322 P 82f50f4b66f3450aa9f58a0b317f5cea: Bootstrap starting.
I20240502 21:25:51.341858 19946 tablet_bootstrap.cc:654] T 6f0107b0690c4c64b0a4f2472a1c8322 P 82f50f4b66f3450aa9f58a0b317f5cea: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:51.342967 19949 ts_tablet_manager.cc:1398] T 1df9d3ad5ddc438798543900b6c67b46 P 2877108756304e05bd60ba1186b870da: Time spent starting tablet: real 0.016s user 0.000s sys 0.007s
I20240502 21:25:51.343246 19949 tablet_bootstrap.cc:492] T b0852f54089944ce935f8ee4cbf2bc5a P 2877108756304e05bd60ba1186b870da: Bootstrap starting.
I20240502 21:25:51.344816 19949 tablet_bootstrap.cc:654] T b0852f54089944ce935f8ee4cbf2bc5a P 2877108756304e05bd60ba1186b870da: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:51.349960 19952 tablet_bootstrap.cc:492] T 6f0107b0690c4c64b0a4f2472a1c8322 P c75831edb3b44dd59c00ef151adca49d: Bootstrap starting.
I20240502 21:25:51.351452 19952 tablet_bootstrap.cc:654] T 6f0107b0690c4c64b0a4f2472a1c8322 P c75831edb3b44dd59c00ef151adca49d: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:51.352499 18451 tablet_service.cc:1467] Processing CreateTablet for tablet b0852f54089944ce935f8ee4cbf2bc5a (DEFAULT_TABLE table=rebalance_test_table_4 [id=c79c94b660074ffc94fcea415b4caae8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:25:51.352908 18451 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0852f54089944ce935f8ee4cbf2bc5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:25:51.352983 19947 tablet_bootstrap.cc:492] T 6f0107b0690c4c64b0a4f2472a1c8322 P faadabfb099b4abf90a466395be585b5: No bootstrap required, opened a new log
I20240502 21:25:51.353158 19947 ts_tablet_manager.cc:1367] T 6f0107b0690c4c64b0a4f2472a1c8322 P faadabfb099b4abf90a466395be585b5: Time spent bootstrapping tablet: real 0.017s user 0.010s sys 0.004s
I20240502 21:25:51.353909 19947 raft_consensus.cc:348] T 6f0107b0690c4c64b0a4f2472a1c8322 P faadabfb099b4abf90a466395be585b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } }
I20240502 21:25:51.354259 19947 raft_consensus.cc:374] T 6f0107b0690c4c64b0a4f2472a1c8322 P faadabfb099b4abf90a466395be585b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:51.354339 19947 raft_consensus.cc:729] T 6f0107b0690c4c64b0a4f2472a1c8322 P faadabfb099b4abf90a466395be585b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: faadabfb099b4abf90a466395be585b5, State: Initialized, Role: FOLLOWER
I20240502 21:25:51.354605 19947 consensus_queue.cc:260] T 6f0107b0690c4c64b0a4f2472a1c8322 P faadabfb099b4abf90a466395be585b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } }
I20240502 21:25:51.355183 19947 ts_tablet_manager.cc:1398] T 6f0107b0690c4c64b0a4f2472a1c8322 P faadabfb099b4abf90a466395be585b5: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 21:25:51.363920 19958 tablet_bootstrap.cc:492] T 6f0107b0690c4c64b0a4f2472a1c8322 P 9be399440ded4e2ebe6b6e0bf486cbb1: Bootstrap starting.
I20240502 21:25:51.363982 19947 tablet_bootstrap.cc:492] T b0852f54089944ce935f8ee4cbf2bc5a P faadabfb099b4abf90a466395be585b5: Bootstrap starting.
I20240502 21:25:51.365200 19952 tablet_bootstrap.cc:492] T 6f0107b0690c4c64b0a4f2472a1c8322 P c75831edb3b44dd59c00ef151adca49d: No bootstrap required, opened a new log
I20240502 21:25:51.365873 19947 tablet_bootstrap.cc:654] T b0852f54089944ce935f8ee4cbf2bc5a P faadabfb099b4abf90a466395be585b5: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:51.365993 19958 tablet_bootstrap.cc:654] T 6f0107b0690c4c64b0a4f2472a1c8322 P 9be399440ded4e2ebe6b6e0bf486cbb1: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:51.366945 19952 ts_tablet_manager.cc:1367] T 6f0107b0690c4c64b0a4f2472a1c8322 P c75831edb3b44dd59c00ef151adca49d: Time spent bootstrapping tablet: real 0.017s user 0.001s sys 0.008s
I20240502 21:25:51.367748 19952 raft_consensus.cc:348] T 6f0107b0690c4c64b0a4f2472a1c8322 P c75831edb3b44dd59c00ef151adca49d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } }
I20240502 21:25:51.368096 19952 raft_consensus.cc:374] T 6f0107b0690c4c64b0a4f2472a1c8322 P c75831edb3b44dd59c00ef151adca49d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:51.368213 19952 raft_consensus.cc:729] T 6f0107b0690c4c64b0a4f2472a1c8322 P c75831edb3b44dd59c00ef151adca49d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c75831edb3b44dd59c00ef151adca49d, State: Initialized, Role: FOLLOWER
I20240502 21:25:51.368610 19946 tablet_bootstrap.cc:492] T 6f0107b0690c4c64b0a4f2472a1c8322 P 82f50f4b66f3450aa9f58a0b317f5cea: No bootstrap required, opened a new log
I20240502 21:25:51.368537 19952 consensus_queue.cc:260] T 6f0107b0690c4c64b0a4f2472a1c8322 P c75831edb3b44dd59c00ef151adca49d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } }
I20240502 21:25:51.368831 19946 ts_tablet_manager.cc:1367] T 6f0107b0690c4c64b0a4f2472a1c8322 P 82f50f4b66f3450aa9f58a0b317f5cea: Time spent bootstrapping tablet: real 0.029s user 0.017s sys 0.006s
I20240502 21:25:51.369287 19952 ts_tablet_manager.cc:1398] T 6f0107b0690c4c64b0a4f2472a1c8322 P c75831edb3b44dd59c00ef151adca49d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:25:51.369582 19946 raft_consensus.cc:348] T 6f0107b0690c4c64b0a4f2472a1c8322 P 82f50f4b66f3450aa9f58a0b317f5cea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } }
I20240502 21:25:51.369931 19946 raft_consensus.cc:374] T 6f0107b0690c4c64b0a4f2472a1c8322 P 82f50f4b66f3450aa9f58a0b317f5cea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:51.370060 19946 raft_consensus.cc:729] T 6f0107b0690c4c64b0a4f2472a1c8322 P 82f50f4b66f3450aa9f58a0b317f5cea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82f50f4b66f3450aa9f58a0b317f5cea, State: Initialized, Role: FOLLOWER
I20240502 21:25:51.370380 19946 consensus_queue.cc:260] T 6f0107b0690c4c64b0a4f2472a1c8322 P 82f50f4b66f3450aa9f58a0b317f5cea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } }
I20240502 21:25:51.372026 19946 ts_tablet_manager.cc:1398] T 6f0107b0690c4c64b0a4f2472a1c8322 P 82f50f4b66f3450aa9f58a0b317f5cea: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 21:25:51.372598 19950 tablet_bootstrap.cc:492] T 1df9d3ad5ddc438798543900b6c67b46 P 501e3c0d44fe4d70af9ff926d9622417: No bootstrap required, opened a new log
I20240502 21:25:51.372782 19950 ts_tablet_manager.cc:1367] T 1df9d3ad5ddc438798543900b6c67b46 P 501e3c0d44fe4d70af9ff926d9622417: Time spent bootstrapping tablet: real 0.053s user 0.011s sys 0.010s
I20240502 21:25:51.373283 19947 tablet_bootstrap.cc:492] T b0852f54089944ce935f8ee4cbf2bc5a P faadabfb099b4abf90a466395be585b5: No bootstrap required, opened a new log
I20240502 21:25:51.373416 19947 ts_tablet_manager.cc:1367] T b0852f54089944ce935f8ee4cbf2bc5a P faadabfb099b4abf90a466395be585b5: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.004s
I20240502 21:25:51.373551 19950 raft_consensus.cc:348] T 1df9d3ad5ddc438798543900b6c67b46 P 501e3c0d44fe4d70af9ff926d9622417 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } }
I20240502 21:25:51.373793 19950 raft_consensus.cc:374] T 1df9d3ad5ddc438798543900b6c67b46 P 501e3c0d44fe4d70af9ff926d9622417 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:51.373889 19950 raft_consensus.cc:729] T 1df9d3ad5ddc438798543900b6c67b46 P 501e3c0d44fe4d70af9ff926d9622417 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 501e3c0d44fe4d70af9ff926d9622417, State: Initialized, Role: FOLLOWER
I20240502 21:25:51.373979 19947 raft_consensus.cc:348] T b0852f54089944ce935f8ee4cbf2bc5a P faadabfb099b4abf90a466395be585b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } }
I20240502 21:25:51.374164 19947 raft_consensus.cc:374] T b0852f54089944ce935f8ee4cbf2bc5a P faadabfb099b4abf90a466395be585b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:51.374240 19947 raft_consensus.cc:729] T b0852f54089944ce935f8ee4cbf2bc5a P faadabfb099b4abf90a466395be585b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: faadabfb099b4abf90a466395be585b5, State: Initialized, Role: FOLLOWER
I20240502 21:25:51.374105 19950 consensus_queue.cc:260] T 1df9d3ad5ddc438798543900b6c67b46 P 501e3c0d44fe4d70af9ff926d9622417 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } }
I20240502 21:25:51.374475 19947 consensus_queue.cc:260] T b0852f54089944ce935f8ee4cbf2bc5a P faadabfb099b4abf90a466395be585b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } }
I20240502 21:25:51.374970 19958 tablet_bootstrap.cc:492] T 6f0107b0690c4c64b0a4f2472a1c8322 P 9be399440ded4e2ebe6b6e0bf486cbb1: No bootstrap required, opened a new log
I20240502 21:25:51.375193 19947 ts_tablet_manager.cc:1398] T b0852f54089944ce935f8ee4cbf2bc5a P faadabfb099b4abf90a466395be585b5: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 21:25:51.375591 19958 ts_tablet_manager.cc:1367] T 6f0107b0690c4c64b0a4f2472a1c8322 P 9be399440ded4e2ebe6b6e0bf486cbb1: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.003s
I20240502 21:25:51.376462 19958 raft_consensus.cc:348] T 6f0107b0690c4c64b0a4f2472a1c8322 P 9be399440ded4e2ebe6b6e0bf486cbb1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } }
I20240502 21:25:51.376739 19958 raft_consensus.cc:374] T 6f0107b0690c4c64b0a4f2472a1c8322 P 9be399440ded4e2ebe6b6e0bf486cbb1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:51.376838 19958 raft_consensus.cc:729] T 6f0107b0690c4c64b0a4f2472a1c8322 P 9be399440ded4e2ebe6b6e0bf486cbb1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9be399440ded4e2ebe6b6e0bf486cbb1, State: Initialized, Role: FOLLOWER
I20240502 21:25:51.376907 19949 tablet_bootstrap.cc:492] T b0852f54089944ce935f8ee4cbf2bc5a P 2877108756304e05bd60ba1186b870da: No bootstrap required, opened a new log
I20240502 21:25:51.377081 19949 ts_tablet_manager.cc:1367] T b0852f54089944ce935f8ee4cbf2bc5a P 2877108756304e05bd60ba1186b870da: Time spent bootstrapping tablet: real 0.034s user 0.004s sys 0.000s
I20240502 21:25:51.377096 19958 consensus_queue.cc:260] T 6f0107b0690c4c64b0a4f2472a1c8322 P 9be399440ded4e2ebe6b6e0bf486cbb1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } }
I20240502 21:25:51.377760 19958 ts_tablet_manager.cc:1398] T 6f0107b0690c4c64b0a4f2472a1c8322 P 9be399440ded4e2ebe6b6e0bf486cbb1: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 21:25:51.377777 19949 raft_consensus.cc:348] T b0852f54089944ce935f8ee4cbf2bc5a P 2877108756304e05bd60ba1186b870da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } }
I20240502 21:25:51.378048 19949 raft_consensus.cc:374] T b0852f54089944ce935f8ee4cbf2bc5a P 2877108756304e05bd60ba1186b870da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:51.378130 19949 raft_consensus.cc:729] T b0852f54089944ce935f8ee4cbf2bc5a P 2877108756304e05bd60ba1186b870da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2877108756304e05bd60ba1186b870da, State: Initialized, Role: FOLLOWER
I20240502 21:25:51.378362 19949 consensus_queue.cc:260] T b0852f54089944ce935f8ee4cbf2bc5a P 2877108756304e05bd60ba1186b870da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } }
I20240502 21:25:51.378886 19949 ts_tablet_manager.cc:1398] T b0852f54089944ce935f8ee4cbf2bc5a P 2877108756304e05bd60ba1186b870da: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20240502 21:25:51.379179 19949 tablet_bootstrap.cc:492] T 6f0107b0690c4c64b0a4f2472a1c8322 P 2877108756304e05bd60ba1186b870da: Bootstrap starting.
I20240502 21:25:51.380652 19949 tablet_bootstrap.cc:654] T 6f0107b0690c4c64b0a4f2472a1c8322 P 2877108756304e05bd60ba1186b870da: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:51.382926 19949 tablet_bootstrap.cc:492] T 6f0107b0690c4c64b0a4f2472a1c8322 P 2877108756304e05bd60ba1186b870da: No bootstrap required, opened a new log
I20240502 21:25:51.383034 19949 ts_tablet_manager.cc:1367] T 6f0107b0690c4c64b0a4f2472a1c8322 P 2877108756304e05bd60ba1186b870da: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20240502 21:25:51.383123 19950 ts_tablet_manager.cc:1398] T 1df9d3ad5ddc438798543900b6c67b46 P 501e3c0d44fe4d70af9ff926d9622417: Time spent starting tablet: real 0.010s user 0.006s sys 0.002s
I20240502 21:25:51.383381 19950 tablet_bootstrap.cc:492] T b0852f54089944ce935f8ee4cbf2bc5a P 501e3c0d44fe4d70af9ff926d9622417: Bootstrap starting.
I20240502 21:25:51.381106 19958 tablet_bootstrap.cc:492] T b0852f54089944ce935f8ee4cbf2bc5a P 9be399440ded4e2ebe6b6e0bf486cbb1: Bootstrap starting.
I20240502 21:25:51.383543 19949 raft_consensus.cc:348] T 6f0107b0690c4c64b0a4f2472a1c8322 P 2877108756304e05bd60ba1186b870da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } }
I20240502 21:25:51.383803 19949 raft_consensus.cc:374] T 6f0107b0690c4c64b0a4f2472a1c8322 P 2877108756304e05bd60ba1186b870da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:51.383909 19949 raft_consensus.cc:729] T 6f0107b0690c4c64b0a4f2472a1c8322 P 2877108756304e05bd60ba1186b870da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2877108756304e05bd60ba1186b870da, State: Initialized, Role: FOLLOWER
I20240502 21:25:51.384120 19949 consensus_queue.cc:260] T 6f0107b0690c4c64b0a4f2472a1c8322 P 2877108756304e05bd60ba1186b870da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } }
I20240502 21:25:51.384630 19949 ts_tablet_manager.cc:1398] T 6f0107b0690c4c64b0a4f2472a1c8322 P 2877108756304e05bd60ba1186b870da: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:25:51.384991 19950 tablet_bootstrap.cc:654] T b0852f54089944ce935f8ee4cbf2bc5a P 501e3c0d44fe4d70af9ff926d9622417: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:51.385601 19958 tablet_bootstrap.cc:654] T b0852f54089944ce935f8ee4cbf2bc5a P 9be399440ded4e2ebe6b6e0bf486cbb1: Neither blocks nor log segments found. Creating new log.
I20240502 21:25:51.387825 19950 tablet_bootstrap.cc:492] T b0852f54089944ce935f8ee4cbf2bc5a P 501e3c0d44fe4d70af9ff926d9622417: No bootstrap required, opened a new log
I20240502 21:25:51.387984 19950 ts_tablet_manager.cc:1367] T b0852f54089944ce935f8ee4cbf2bc5a P 501e3c0d44fe4d70af9ff926d9622417: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240502 21:25:51.388837 19958 tablet_bootstrap.cc:492] T b0852f54089944ce935f8ee4cbf2bc5a P 9be399440ded4e2ebe6b6e0bf486cbb1: No bootstrap required, opened a new log
I20240502 21:25:51.389007 19958 ts_tablet_manager.cc:1367] T b0852f54089944ce935f8ee4cbf2bc5a P 9be399440ded4e2ebe6b6e0bf486cbb1: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240502 21:25:51.388864 19950 raft_consensus.cc:348] T b0852f54089944ce935f8ee4cbf2bc5a P 501e3c0d44fe4d70af9ff926d9622417 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } }
I20240502 21:25:51.389103 19950 raft_consensus.cc:374] T b0852f54089944ce935f8ee4cbf2bc5a P 501e3c0d44fe4d70af9ff926d9622417 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:51.389185 19950 raft_consensus.cc:729] T b0852f54089944ce935f8ee4cbf2bc5a P 501e3c0d44fe4d70af9ff926d9622417 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 501e3c0d44fe4d70af9ff926d9622417, State: Initialized, Role: FOLLOWER
I20240502 21:25:51.389406 19950 consensus_queue.cc:260] T b0852f54089944ce935f8ee4cbf2bc5a P 501e3c0d44fe4d70af9ff926d9622417 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } }
I20240502 21:25:51.389660 19958 raft_consensus.cc:348] T b0852f54089944ce935f8ee4cbf2bc5a P 9be399440ded4e2ebe6b6e0bf486cbb1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } }
I20240502 21:25:51.389902 19950 ts_tablet_manager.cc:1398] T b0852f54089944ce935f8ee4cbf2bc5a P 501e3c0d44fe4d70af9ff926d9622417: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 21:25:51.389906 19958 raft_consensus.cc:374] T b0852f54089944ce935f8ee4cbf2bc5a P 9be399440ded4e2ebe6b6e0bf486cbb1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:25:51.389978 19958 raft_consensus.cc:729] T b0852f54089944ce935f8ee4cbf2bc5a P 9be399440ded4e2ebe6b6e0bf486cbb1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9be399440ded4e2ebe6b6e0bf486cbb1, State: Initialized, Role: FOLLOWER
I20240502 21:25:51.390201 19958 consensus_queue.cc:260] T b0852f54089944ce935f8ee4cbf2bc5a P 9be399440ded4e2ebe6b6e0bf486cbb1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } }
I20240502 21:25:51.390709 19958 ts_tablet_manager.cc:1398] T b0852f54089944ce935f8ee4cbf2bc5a P 9be399440ded4e2ebe6b6e0bf486cbb1: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 21:25:51.468595 19902 raft_consensus.cc:482] T 6f0107b0690c4c64b0a4f2472a1c8322 P faadabfb099b4abf90a466395be585b5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:25:51.468797 19902 raft_consensus.cc:504] T 6f0107b0690c4c64b0a4f2472a1c8322 P faadabfb099b4abf90a466395be585b5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } }
I20240502 21:25:51.469689 19902 leader_election.cc:288] T 6f0107b0690c4c64b0a4f2472a1c8322 P faadabfb099b4abf90a466395be585b5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 82f50f4b66f3450aa9f58a0b317f5cea (127.15.255.130:45139), c75831edb3b44dd59c00ef151adca49d (127.15.255.132:43607), 9be399440ded4e2ebe6b6e0bf486cbb1 (127.15.255.133:34217), 2877108756304e05bd60ba1186b870da (127.15.255.134:43635)
I20240502 21:25:51.470057 19009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f0107b0690c4c64b0a4f2472a1c8322" candidate_uuid: "faadabfb099b4abf90a466395be585b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" is_pre_election: true
I20240502 21:25:51.470117 18878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f0107b0690c4c64b0a4f2472a1c8322" candidate_uuid: "faadabfb099b4abf90a466395be585b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c75831edb3b44dd59c00ef151adca49d" is_pre_election: true
I20240502 21:25:51.470290 19009 raft_consensus.cc:2454] T 6f0107b0690c4c64b0a4f2472a1c8322 P 9be399440ded4e2ebe6b6e0bf486cbb1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate faadabfb099b4abf90a466395be585b5 in term 0.
I20240502 21:25:51.470257 18611 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f0107b0690c4c64b0a4f2472a1c8322" candidate_uuid: "faadabfb099b4abf90a466395be585b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" is_pre_election: true
I20240502 21:25:51.470351 18878 raft_consensus.cc:2454] T 6f0107b0690c4c64b0a4f2472a1c8322 P c75831edb3b44dd59c00ef151adca49d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate faadabfb099b4abf90a466395be585b5 in term 0.
I20240502 21:25:51.470454 18611 raft_consensus.cc:2454] T 6f0107b0690c4c64b0a4f2472a1c8322 P 82f50f4b66f3450aa9f58a0b317f5cea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate faadabfb099b4abf90a466395be585b5 in term 0.
I20240502 21:25:51.470793 18406 leader_election.cc:302] T 6f0107b0690c4c64b0a4f2472a1c8322 P faadabfb099b4abf90a466395be585b5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 9be399440ded4e2ebe6b6e0bf486cbb1, c75831edb3b44dd59c00ef151adca49d, faadabfb099b4abf90a466395be585b5; no voters:
I20240502 21:25:51.471087 19902 raft_consensus.cc:2781] T 6f0107b0690c4c64b0a4f2472a1c8322 P faadabfb099b4abf90a466395be585b5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:25:51.471199 19902 raft_consensus.cc:482] T 6f0107b0690c4c64b0a4f2472a1c8322 P faadabfb099b4abf90a466395be585b5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:25:51.471280 19902 raft_consensus.cc:3037] T 6f0107b0690c4c64b0a4f2472a1c8322 P faadabfb099b4abf90a466395be585b5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:51.471422 19146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f0107b0690c4c64b0a4f2472a1c8322" candidate_uuid: "faadabfb099b4abf90a466395be585b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2877108756304e05bd60ba1186b870da" is_pre_election: true
I20240502 21:25:51.471649 19146 raft_consensus.cc:2454] T 6f0107b0690c4c64b0a4f2472a1c8322 P 2877108756304e05bd60ba1186b870da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate faadabfb099b4abf90a466395be585b5 in term 0.
I20240502 21:25:51.473469 19902 raft_consensus.cc:504] T 6f0107b0690c4c64b0a4f2472a1c8322 P faadabfb099b4abf90a466395be585b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } }
I20240502 21:25:51.474331 19902 leader_election.cc:288] T 6f0107b0690c4c64b0a4f2472a1c8322 P faadabfb099b4abf90a466395be585b5 [CANDIDATE]: Term 1 election: Requested vote from peers 82f50f4b66f3450aa9f58a0b317f5cea (127.15.255.130:45139), c75831edb3b44dd59c00ef151adca49d (127.15.255.132:43607), 9be399440ded4e2ebe6b6e0bf486cbb1 (127.15.255.133:34217), 2877108756304e05bd60ba1186b870da (127.15.255.134:43635)
I20240502 21:25:51.474450 19009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f0107b0690c4c64b0a4f2472a1c8322" candidate_uuid: "faadabfb099b4abf90a466395be585b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1"
I20240502 21:25:51.474507 18611 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f0107b0690c4c64b0a4f2472a1c8322" candidate_uuid: "faadabfb099b4abf90a466395be585b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82f50f4b66f3450aa9f58a0b317f5cea"
I20240502 21:25:51.474623 19009 raft_consensus.cc:3037] T 6f0107b0690c4c64b0a4f2472a1c8322 P 9be399440ded4e2ebe6b6e0bf486cbb1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:51.474689 18611 raft_consensus.cc:3037] T 6f0107b0690c4c64b0a4f2472a1c8322 P 82f50f4b66f3450aa9f58a0b317f5cea [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:51.474787 19146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f0107b0690c4c64b0a4f2472a1c8322" candidate_uuid: "faadabfb099b4abf90a466395be585b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2877108756304e05bd60ba1186b870da"
I20240502 21:25:51.474946 19146 raft_consensus.cc:3037] T 6f0107b0690c4c64b0a4f2472a1c8322 P 2877108756304e05bd60ba1186b870da [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:51.476524 19009 raft_consensus.cc:2454] T 6f0107b0690c4c64b0a4f2472a1c8322 P 9be399440ded4e2ebe6b6e0bf486cbb1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate faadabfb099b4abf90a466395be585b5 in term 1.
I20240502 21:25:51.477331 18611 raft_consensus.cc:2454] T 6f0107b0690c4c64b0a4f2472a1c8322 P 82f50f4b66f3450aa9f58a0b317f5cea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate faadabfb099b4abf90a466395be585b5 in term 1.
I20240502 21:25:51.477861 18878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f0107b0690c4c64b0a4f2472a1c8322" candidate_uuid: "faadabfb099b4abf90a466395be585b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c75831edb3b44dd59c00ef151adca49d"
I20240502 21:25:51.478049 18878 raft_consensus.cc:3037] T 6f0107b0690c4c64b0a4f2472a1c8322 P c75831edb3b44dd59c00ef151adca49d [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:51.478163 19146 raft_consensus.cc:2454] T 6f0107b0690c4c64b0a4f2472a1c8322 P 2877108756304e05bd60ba1186b870da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate faadabfb099b4abf90a466395be585b5 in term 1.
I20240502 21:25:51.479878 18878 raft_consensus.cc:2454] T 6f0107b0690c4c64b0a4f2472a1c8322 P c75831edb3b44dd59c00ef151adca49d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate faadabfb099b4abf90a466395be585b5 in term 1.
I20240502 21:25:51.480190 18407 leader_election.cc:302] T 6f0107b0690c4c64b0a4f2472a1c8322 P faadabfb099b4abf90a466395be585b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 2877108756304e05bd60ba1186b870da, 9be399440ded4e2ebe6b6e0bf486cbb1, faadabfb099b4abf90a466395be585b5; no voters:
I20240502 21:25:51.480520 19902 raft_consensus.cc:2781] T 6f0107b0690c4c64b0a4f2472a1c8322 P faadabfb099b4abf90a466395be585b5 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:25:51.480700 19902 raft_consensus.cc:686] T 6f0107b0690c4c64b0a4f2472a1c8322 P faadabfb099b4abf90a466395be585b5 [term 1 LEADER]: Becoming Leader. State: Replica: faadabfb099b4abf90a466395be585b5, State: Running, Role: LEADER
I20240502 21:25:51.480952 19902 consensus_queue.cc:237] T 6f0107b0690c4c64b0a4f2472a1c8322 P faadabfb099b4abf90a466395be585b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } }
I20240502 21:25:51.483729 18321 catalog_manager.cc:5484] T 6f0107b0690c4c64b0a4f2472a1c8322 P faadabfb099b4abf90a466395be585b5 reported cstate change: term changed from 0 to 1, leader changed from <none> to faadabfb099b4abf90a466395be585b5 (127.15.255.129). New cstate: current_term: 1 leader_uuid: "faadabfb099b4abf90a466395be585b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } health_report { overall_health: UNKNOWN } } }
I20240502 21:25:51.494592 19967 raft_consensus.cc:482] T 1df9d3ad5ddc438798543900b6c67b46 P 501e3c0d44fe4d70af9ff926d9622417 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:25:51.494756 19967 raft_consensus.cc:504] T 1df9d3ad5ddc438798543900b6c67b46 P 501e3c0d44fe4d70af9ff926d9622417 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } }
I20240502 21:25:51.495719 19967 leader_election.cc:288] T 1df9d3ad5ddc438798543900b6c67b46 P 501e3c0d44fe4d70af9ff926d9622417 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 82f50f4b66f3450aa9f58a0b317f5cea (127.15.255.130:45139), faadabfb099b4abf90a466395be585b5 (127.15.255.129:32983), 2877108756304e05bd60ba1186b870da (127.15.255.134:43635), c75831edb3b44dd59c00ef151adca49d (127.15.255.132:43607)
I20240502 21:25:51.496047 18473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1df9d3ad5ddc438798543900b6c67b46" candidate_uuid: "501e3c0d44fe4d70af9ff926d9622417" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "faadabfb099b4abf90a466395be585b5" is_pre_election: true
I20240502 21:25:51.496155 18611 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1df9d3ad5ddc438798543900b6c67b46" candidate_uuid: "501e3c0d44fe4d70af9ff926d9622417" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" is_pre_election: true
I20240502 21:25:51.496263 18473 raft_consensus.cc:2454] T 1df9d3ad5ddc438798543900b6c67b46 P faadabfb099b4abf90a466395be585b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 501e3c0d44fe4d70af9ff926d9622417 in term 0.
I20240502 21:25:51.496369 18611 raft_consensus.cc:2454] T 1df9d3ad5ddc438798543900b6c67b46 P 82f50f4b66f3450aa9f58a0b317f5cea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 501e3c0d44fe4d70af9ff926d9622417 in term 0.
I20240502 21:25:51.496904 18678 leader_election.cc:302] T 1df9d3ad5ddc438798543900b6c67b46 P 501e3c0d44fe4d70af9ff926d9622417 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 501e3c0d44fe4d70af9ff926d9622417, 82f50f4b66f3450aa9f58a0b317f5cea, faadabfb099b4abf90a466395be585b5; no voters:
I20240502 21:25:51.497277 19967 raft_consensus.cc:2781] T 1df9d3ad5ddc438798543900b6c67b46 P 501e3c0d44fe4d70af9ff926d9622417 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:25:51.497401 19967 raft_consensus.cc:482] T 1df9d3ad5ddc438798543900b6c67b46 P 501e3c0d44fe4d70af9ff926d9622417 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:25:51.496042 19146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1df9d3ad5ddc438798543900b6c67b46" candidate_uuid: "501e3c0d44fe4d70af9ff926d9622417" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2877108756304e05bd60ba1186b870da" is_pre_election: true
I20240502 21:25:51.497488 19967 raft_consensus.cc:3037] T 1df9d3ad5ddc438798543900b6c67b46 P 501e3c0d44fe4d70af9ff926d9622417 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:51.497560 19146 raft_consensus.cc:2454] T 1df9d3ad5ddc438798543900b6c67b46 P 2877108756304e05bd60ba1186b870da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 501e3c0d44fe4d70af9ff926d9622417 in term 0.
I20240502 21:25:51.500160 19967 raft_consensus.cc:504] T 1df9d3ad5ddc438798543900b6c67b46 P 501e3c0d44fe4d70af9ff926d9622417 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } }
I20240502 21:25:51.501387 19967 leader_election.cc:288] T 1df9d3ad5ddc438798543900b6c67b46 P 501e3c0d44fe4d70af9ff926d9622417 [CANDIDATE]: Term 1 election: Requested vote from peers 82f50f4b66f3450aa9f58a0b317f5cea (127.15.255.130:45139), faadabfb099b4abf90a466395be585b5 (127.15.255.129:32983), 2877108756304e05bd60ba1186b870da (127.15.255.134:43635), c75831edb3b44dd59c00ef151adca49d (127.15.255.132:43607)
I20240502 21:25:51.501358 18611 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1df9d3ad5ddc438798543900b6c67b46" candidate_uuid: "501e3c0d44fe4d70af9ff926d9622417" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82f50f4b66f3450aa9f58a0b317f5cea"
I20240502 21:25:51.501566 18611 raft_consensus.cc:3037] T 1df9d3ad5ddc438798543900b6c67b46 P 82f50f4b66f3450aa9f58a0b317f5cea [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:51.501601 18473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1df9d3ad5ddc438798543900b6c67b46" candidate_uuid: "501e3c0d44fe4d70af9ff926d9622417" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "faadabfb099b4abf90a466395be585b5"
I20240502 21:25:51.501708 19146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1df9d3ad5ddc438798543900b6c67b46" candidate_uuid: "501e3c0d44fe4d70af9ff926d9622417" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2877108756304e05bd60ba1186b870da"
I20240502 21:25:51.501791 18473 raft_consensus.cc:3037] T 1df9d3ad5ddc438798543900b6c67b46 P faadabfb099b4abf90a466395be585b5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:51.501899 19146 raft_consensus.cc:3037] T 1df9d3ad5ddc438798543900b6c67b46 P 2877108756304e05bd60ba1186b870da [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:51.504235 18611 raft_consensus.cc:2454] T 1df9d3ad5ddc438798543900b6c67b46 P 82f50f4b66f3450aa9f58a0b317f5cea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 501e3c0d44fe4d70af9ff926d9622417 in term 1.
I20240502 21:25:51.504343 18473 raft_consensus.cc:2454] T 1df9d3ad5ddc438798543900b6c67b46 P faadabfb099b4abf90a466395be585b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 501e3c0d44fe4d70af9ff926d9622417 in term 1.
I20240502 21:25:51.504459 19146 raft_consensus.cc:2454] T 1df9d3ad5ddc438798543900b6c67b46 P 2877108756304e05bd60ba1186b870da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 501e3c0d44fe4d70af9ff926d9622417 in term 1.
I20240502 21:25:51.504838 18877 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1df9d3ad5ddc438798543900b6c67b46" candidate_uuid: "501e3c0d44fe4d70af9ff926d9622417" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c75831edb3b44dd59c00ef151adca49d"
I20240502 21:25:51.505031 18877 raft_consensus.cc:3037] T 1df9d3ad5ddc438798543900b6c67b46 P c75831edb3b44dd59c00ef151adca49d [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:51.505092 18680 leader_election.cc:302] T 1df9d3ad5ddc438798543900b6c67b46 P 501e3c0d44fe4d70af9ff926d9622417 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 2877108756304e05bd60ba1186b870da, 501e3c0d44fe4d70af9ff926d9622417, 82f50f4b66f3450aa9f58a0b317f5cea; no voters:
I20240502 21:25:51.505290 18878 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1df9d3ad5ddc438798543900b6c67b46" candidate_uuid: "501e3c0d44fe4d70af9ff926d9622417" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c75831edb3b44dd59c00ef151adca49d" is_pre_election: true
I20240502 21:25:51.505776 19967 raft_consensus.cc:2781] T 1df9d3ad5ddc438798543900b6c67b46 P 501e3c0d44fe4d70af9ff926d9622417 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:25:51.506242 19967 raft_consensus.cc:686] T 1df9d3ad5ddc438798543900b6c67b46 P 501e3c0d44fe4d70af9ff926d9622417 [term 1 LEADER]: Becoming Leader. State: Replica: 501e3c0d44fe4d70af9ff926d9622417, State: Running, Role: LEADER
I20240502 21:25:51.507017 19967 consensus_queue.cc:237] T 1df9d3ad5ddc438798543900b6c67b46 P 501e3c0d44fe4d70af9ff926d9622417 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } }
I20240502 21:25:51.507548 18877 raft_consensus.cc:2454] T 1df9d3ad5ddc438798543900b6c67b46 P c75831edb3b44dd59c00ef151adca49d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 501e3c0d44fe4d70af9ff926d9622417 in term 1.
I20240502 21:25:51.510625 18321 catalog_manager.cc:5484] T 1df9d3ad5ddc438798543900b6c67b46 P 501e3c0d44fe4d70af9ff926d9622417 reported cstate change: term changed from 0 to 1, leader changed from <none> to 501e3c0d44fe4d70af9ff926d9622417 (127.15.255.131). New cstate: current_term: 1 leader_uuid: "501e3c0d44fe4d70af9ff926d9622417" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } health_report { overall_health: UNKNOWN } } }
I20240502 21:25:51.574913 19957 raft_consensus.cc:482] T b0852f54089944ce935f8ee4cbf2bc5a P 2877108756304e05bd60ba1186b870da [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:25:51.575076 19957 raft_consensus.cc:504] T b0852f54089944ce935f8ee4cbf2bc5a P 2877108756304e05bd60ba1186b870da [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } }
I20240502 21:25:51.576025 19957 leader_election.cc:288] T b0852f54089944ce935f8ee4cbf2bc5a P 2877108756304e05bd60ba1186b870da [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9be399440ded4e2ebe6b6e0bf486cbb1 (127.15.255.133:34217), 82f50f4b66f3450aa9f58a0b317f5cea (127.15.255.130:45139), faadabfb099b4abf90a466395be585b5 (127.15.255.129:32983), 501e3c0d44fe4d70af9ff926d9622417 (127.15.255.131:42501)
I20240502 21:25:51.576583 18743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0852f54089944ce935f8ee4cbf2bc5a" candidate_uuid: "2877108756304e05bd60ba1186b870da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "501e3c0d44fe4d70af9ff926d9622417" is_pre_election: true
I20240502 21:25:51.576380 18611 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0852f54089944ce935f8ee4cbf2bc5a" candidate_uuid: "2877108756304e05bd60ba1186b870da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" is_pre_election: true
I20240502 21:25:51.576815 18743 raft_consensus.cc:2454] T b0852f54089944ce935f8ee4cbf2bc5a P 501e3c0d44fe4d70af9ff926d9622417 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2877108756304e05bd60ba1186b870da in term 0.
I20240502 21:25:51.576906 18611 raft_consensus.cc:2454] T b0852f54089944ce935f8ee4cbf2bc5a P 82f50f4b66f3450aa9f58a0b317f5cea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2877108756304e05bd60ba1186b870da in term 0.
I20240502 21:25:51.577365 18473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0852f54089944ce935f8ee4cbf2bc5a" candidate_uuid: "2877108756304e05bd60ba1186b870da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "faadabfb099b4abf90a466395be585b5" is_pre_election: true
I20240502 21:25:51.577431 19076 leader_election.cc:302] T b0852f54089944ce935f8ee4cbf2bc5a P 2877108756304e05bd60ba1186b870da [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 5 voters: 3 yes votes; 0 no votes. yes voters: 2877108756304e05bd60ba1186b870da, 501e3c0d44fe4d70af9ff926d9622417, 82f50f4b66f3450aa9f58a0b317f5cea; no voters:
I20240502 21:25:51.577661 18473 raft_consensus.cc:2454] T b0852f54089944ce935f8ee4cbf2bc5a P faadabfb099b4abf90a466395be585b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2877108756304e05bd60ba1186b870da in term 0.
I20240502 21:25:51.577737 19957 raft_consensus.cc:2781] T b0852f54089944ce935f8ee4cbf2bc5a P 2877108756304e05bd60ba1186b870da [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:25:51.577875 19957 raft_consensus.cc:482] T b0852f54089944ce935f8ee4cbf2bc5a P 2877108756304e05bd60ba1186b870da [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:25:51.577980 19957 raft_consensus.cc:3037] T b0852f54089944ce935f8ee4cbf2bc5a P 2877108756304e05bd60ba1186b870da [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:51.579931 19957 raft_consensus.cc:504] T b0852f54089944ce935f8ee4cbf2bc5a P 2877108756304e05bd60ba1186b870da [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } }
I20240502 21:25:51.580828 19957 leader_election.cc:288] T b0852f54089944ce935f8ee4cbf2bc5a P 2877108756304e05bd60ba1186b870da [CANDIDATE]: Term 1 election: Requested vote from peers 9be399440ded4e2ebe6b6e0bf486cbb1 (127.15.255.133:34217), 82f50f4b66f3450aa9f58a0b317f5cea (127.15.255.130:45139), faadabfb099b4abf90a466395be585b5 (127.15.255.129:32983), 501e3c0d44fe4d70af9ff926d9622417 (127.15.255.131:42501)
I20240502 21:25:51.581321 18611 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0852f54089944ce935f8ee4cbf2bc5a" candidate_uuid: "2877108756304e05bd60ba1186b870da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82f50f4b66f3450aa9f58a0b317f5cea"
I20240502 21:25:51.581312 18473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0852f54089944ce935f8ee4cbf2bc5a" candidate_uuid: "2877108756304e05bd60ba1186b870da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "faadabfb099b4abf90a466395be585b5"
I20240502 21:25:51.581339 18743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0852f54089944ce935f8ee4cbf2bc5a" candidate_uuid: "2877108756304e05bd60ba1186b870da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "501e3c0d44fe4d70af9ff926d9622417"
I20240502 21:25:51.581496 18611 raft_consensus.cc:3037] T b0852f54089944ce935f8ee4cbf2bc5a P 82f50f4b66f3450aa9f58a0b317f5cea [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:51.581526 18743 raft_consensus.cc:3037] T b0852f54089944ce935f8ee4cbf2bc5a P 501e3c0d44fe4d70af9ff926d9622417 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:51.581528 18473 raft_consensus.cc:3037] T b0852f54089944ce935f8ee4cbf2bc5a P faadabfb099b4abf90a466395be585b5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:51.583038 19009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0852f54089944ce935f8ee4cbf2bc5a" candidate_uuid: "2877108756304e05bd60ba1186b870da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" is_pre_election: true
I20240502 21:25:51.583065 19010 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0852f54089944ce935f8ee4cbf2bc5a" candidate_uuid: "2877108756304e05bd60ba1186b870da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1"
I20240502 21:25:51.583312 19009 raft_consensus.cc:2454] T b0852f54089944ce935f8ee4cbf2bc5a P 9be399440ded4e2ebe6b6e0bf486cbb1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2877108756304e05bd60ba1186b870da in term 0.
I20240502 21:25:51.584108 18743 raft_consensus.cc:2454] T b0852f54089944ce935f8ee4cbf2bc5a P 501e3c0d44fe4d70af9ff926d9622417 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2877108756304e05bd60ba1186b870da in term 1.
I20240502 21:25:51.584137 18611 raft_consensus.cc:2454] T b0852f54089944ce935f8ee4cbf2bc5a P 82f50f4b66f3450aa9f58a0b317f5cea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2877108756304e05bd60ba1186b870da in term 1.
I20240502 21:25:51.584769 19079 leader_election.cc:302] T b0852f54089944ce935f8ee4cbf2bc5a P 2877108756304e05bd60ba1186b870da [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 5 voters: 3 yes votes; 1 no votes. yes voters: 2877108756304e05bd60ba1186b870da, 501e3c0d44fe4d70af9ff926d9622417, 82f50f4b66f3450aa9f58a0b317f5cea; no voters: 9be399440ded4e2ebe6b6e0bf486cbb1
I20240502 21:25:51.584920 18473 raft_consensus.cc:2454] T b0852f54089944ce935f8ee4cbf2bc5a P faadabfb099b4abf90a466395be585b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2877108756304e05bd60ba1186b870da in term 1.
I20240502 21:25:51.585044 19957 raft_consensus.cc:2781] T b0852f54089944ce935f8ee4cbf2bc5a P 2877108756304e05bd60ba1186b870da [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:25:51.585592 19957 raft_consensus.cc:686] T b0852f54089944ce935f8ee4cbf2bc5a P 2877108756304e05bd60ba1186b870da [term 1 LEADER]: Becoming Leader. State: Replica: 2877108756304e05bd60ba1186b870da, State: Running, Role: LEADER
I20240502 21:25:51.586144 19957 consensus_queue.cc:237] T b0852f54089944ce935f8ee4cbf2bc5a P 2877108756304e05bd60ba1186b870da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } }
I20240502 21:25:51.589999 18321 catalog_manager.cc:5484] T b0852f54089944ce935f8ee4cbf2bc5a P 2877108756304e05bd60ba1186b870da reported cstate change: term changed from 0 to 1, leader changed from <none> to 2877108756304e05bd60ba1186b870da (127.15.255.134). New cstate: current_term: 1 leader_uuid: "2877108756304e05bd60ba1186b870da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } health_report { overall_health: UNKNOWN } } }
I20240502 21:25:51.881419 19902 consensus_queue.cc:1035] T 6f0107b0690c4c64b0a4f2472a1c8322 P faadabfb099b4abf90a466395be585b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:25:51.893245 19902 consensus_queue.cc:1035] T 6f0107b0690c4c64b0a4f2472a1c8322 P faadabfb099b4abf90a466395be585b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:25:51.897112 19902 consensus_queue.cc:1035] T 6f0107b0690c4c64b0a4f2472a1c8322 P faadabfb099b4abf90a466395be585b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:25:51.901347 19936 consensus_queue.cc:1035] T 6f0107b0690c4c64b0a4f2472a1c8322 P faadabfb099b4abf90a466395be585b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:25:51.927320 19967 consensus_queue.cc:1035] T 1df9d3ad5ddc438798543900b6c67b46 P 501e3c0d44fe4d70af9ff926d9622417 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 21:25:51.934286 19980 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:25:51.934576 19980 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240502 21:25:51.938359 19967 consensus_queue.cc:1035] T 1df9d3ad5ddc438798543900b6c67b46 P 501e3c0d44fe4d70af9ff926d9622417 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 21:25:51.943830 19980 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 21:25:51.944975 19974 consensus_queue.cc:1035] T 1df9d3ad5ddc438798543900b6c67b46 P 501e3c0d44fe4d70af9ff926d9622417 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:25:51.948601 19974 consensus_queue.cc:1035] T 1df9d3ad5ddc438798543900b6c67b46 P 501e3c0d44fe4d70af9ff926d9622417 [LEADER]: Connected to new peer: Peer: permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_4'
I20240502 21:25:52.098508 18743 raft_consensus.cc:1261] T b0852f54089944ce935f8ee4cbf2bc5a P 501e3c0d44fe4d70af9ff926d9622417 [term 1 FOLLOWER]: Refusing update from remote peer 2877108756304e05bd60ba1186b870da: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:25:52.099695 19957 consensus_queue.cc:1035] T b0852f54089944ce935f8ee4cbf2bc5a P 2877108756304e05bd60ba1186b870da [LEADER]: Connected to new peer: Peer: permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:25:52.103442 18473 raft_consensus.cc:1261] T b0852f54089944ce935f8ee4cbf2bc5a P faadabfb099b4abf90a466395be585b5 [term 1 FOLLOWER]: Refusing update from remote peer 2877108756304e05bd60ba1186b870da: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:25:52.105276 19009 raft_consensus.cc:3037] T b0852f54089944ce935f8ee4cbf2bc5a P 9be399440ded4e2ebe6b6e0bf486cbb1 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:25:52.111927 19957 consensus_queue.cc:1035] T b0852f54089944ce935f8ee4cbf2bc5a P 2877108756304e05bd60ba1186b870da [LEADER]: Connected to new peer: Peer: permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:25:52.125536 19009 raft_consensus.cc:1261] T b0852f54089944ce935f8ee4cbf2bc5a P 9be399440ded4e2ebe6b6e0bf486cbb1 [term 1 FOLLOWER]: Refusing update from remote peer 2877108756304e05bd60ba1186b870da: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:25:52.127125 19957 consensus_queue.cc:1035] T b0852f54089944ce935f8ee4cbf2bc5a P 2877108756304e05bd60ba1186b870da [LEADER]: Connected to new peer: Peer: permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:25:52.143602 18611 raft_consensus.cc:1261] T b0852f54089944ce935f8ee4cbf2bc5a P 82f50f4b66f3450aa9f58a0b317f5cea [term 1 FOLLOWER]: Refusing update from remote peer 2877108756304e05bd60ba1186b870da: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:25:52.144338 19957 consensus_queue.cc:1035] T b0852f54089944ce935f8ee4cbf2bc5a P 2877108756304e05bd60ba1186b870da [LEADER]: Connected to new peer: Peer: permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:25:52.159482 19985 mvcc.cc:205] Tried to move back new op lower bound from 7023350382985854976 to 7023350380901335040. Current Snapshot: MvccSnapshot[applied={T|T < 7023350382985854976}]
I20240502 21:25:52.334888 20045 mvcc.cc:205] Tried to move back new op lower bound from 7023350382985854976 to 7023350380901335040. Current Snapshot: MvccSnapshot[applied={T|T < 7023350382985854976}]
INSERT report
rows total: 2000
time total: 433.193 ms
time per row: 0.216597 ms
W20240502 21:25:52.450227 18406 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.135:43395, user_credentials={real_user=slave}} blocked reactor thread for 67046us
W20240502 21:25:53.245746 18406 consensus_peers.cc:485] T 40d4f61d0f6d4d31a5626c7743c88133 P faadabfb099b4abf90a466395be585b5 -> Peer 90e329b457c24caf93a9f880345eb288 (127.15.255.135:43395): Couldn't send request to peer 90e329b457c24caf93a9f880345eb288. Status: Network error: Client connection negotiation failed: client connection to 127.15.255.135:43395: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240502 21:25:53.430723 16382 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvNciQZ/build/asan/bin/kudu
/tmp/dist-test-taskvNciQZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.255.135:43395
--local_ip_for_outbound_sockets=127.15.255.135
--tserver_master_addrs=127.15.255.190:36947
--webserver_port=46157
--webserver_interface=127.15.255.135
--builtin_ntp_servers=127.15.255.148:40185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240502 21:25:53.746995 20077 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:25:53.747244 20077 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:25:53.747352 20077 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:25:53.754719 20077 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 21:25:53.754823 20077 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:25:53.754966 20077 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.255.135
I20240502 21:25:53.763430 20077 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.255.148:40185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.255.135:43395
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.15.255.135
--webserver_port=46157
--tserver_master_addrs=127.15.255.190:36947
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.255.135
--log_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:25:53.764878 20077 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:25:53.766497 20077 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:25:53.778518 20086 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:53.778998 20084 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:25:53.779719 20083 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:25:53.780326 20077 server_base.cc:1034] running on GCE node
I20240502 21:25:53.780887 20077 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:25:53.781708 20077 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:25:53.783010 20077 hybrid_clock.cc:648] HybridClock initialized: now 1714685153782953 us; error 54 us; skew 500 ppm
I20240502 21:25:53.783325 20077 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:25:53.785588 20077 webserver.cc:460] Webserver started at http://127.15.255.135:46157/ using document root <none> and password file <none>
I20240502 21:25:53.786384 20077 fs_manager.cc:359] Metadata directory not provided
I20240502 21:25:53.786489 20077 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:25:53.791816 20077 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.001s
I20240502 21:25:53.794808 20093 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:25:53.796296 20077 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20240502 21:25:53.796511 20077 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "90e329b457c24caf93a9f880345eb288"
format_stamp: "Formatted at 2024-05-02 21:25:39 on dist-test-slave-jdfx"
I20240502 21:25:53.797523 20077 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:25:53.821326 20077 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:25:53.822033 20077 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:25:53.822297 20077 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:25:53.823133 20077 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:25:53.824892 20101 ts_tablet_manager.cc:522] Loading tablet metadata (0/1 complete)
I20240502 21:25:53.829674 20077 ts_tablet_manager.cc:559] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20240502 21:25:53.829757 20077 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20240502 21:25:53.829872 20077 ts_tablet_manager.cc:574] Registering tablets (0/1 complete)
I20240502 21:25:53.832629 20077 ts_tablet_manager.cc:590] Registered 1 tablets
I20240502 21:25:53.832744 20077 ts_tablet_manager.cc:569] Time spent register tablets: real 0.003s user 0.003s sys 0.000s
I20240502 21:25:53.833108 20101 tablet_bootstrap.cc:492] T 40d4f61d0f6d4d31a5626c7743c88133 P 90e329b457c24caf93a9f880345eb288: Bootstrap starting.
I20240502 21:25:53.853183 20101 log.cc:824] T 40d4f61d0f6d4d31a5626c7743c88133 P 90e329b457c24caf93a9f880345eb288: Log is configured to *not* fsync() on all Append() calls
I20240502 21:25:53.856329 20101 tablet_bootstrap.cc:492] T 40d4f61d0f6d4d31a5626c7743c88133 P 90e329b457c24caf93a9f880345eb288: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:25:53.857216 20101 tablet_bootstrap.cc:492] T 40d4f61d0f6d4d31a5626c7743c88133 P 90e329b457c24caf93a9f880345eb288: Bootstrap complete.
I20240502 21:25:53.858844 20101 ts_tablet_manager.cc:1367] T 40d4f61d0f6d4d31a5626c7743c88133 P 90e329b457c24caf93a9f880345eb288: Time spent bootstrapping tablet: real 0.026s user 0.000s sys 0.025s
I20240502 21:25:53.872714 20077 rpc_server.cc:307] RPC server started. Bound to: 127.15.255.135:43395
I20240502 21:25:53.872783 20214 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.255.135:43395 every 8 connection(s)
I20240502 21:25:53.874218 20077 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240502 21:25:53.878152 16382 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvNciQZ/build/asan/bin/kudu as pid 20077
I20240502 21:25:53.878453 16382 test_util.cc:268] Using random seed: 1065329054
I20240502 21:25:53.883543 20101 raft_consensus.cc:348] T 40d4f61d0f6d4d31a5626c7743c88133 P 90e329b457c24caf93a9f880345eb288 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } }
I20240502 21:25:53.888599 20101 raft_consensus.cc:729] T 40d4f61d0f6d4d31a5626c7743c88133 P 90e329b457c24caf93a9f880345eb288 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90e329b457c24caf93a9f880345eb288, State: Initialized, Role: FOLLOWER
I20240502 21:25:53.889513 20101 consensus_queue.cc:260] T 40d4f61d0f6d4d31a5626c7743c88133 P 90e329b457c24caf93a9f880345eb288 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } }
I20240502 21:25:53.894979 20215 heartbeater.cc:347] Connected to a master server at 127.15.255.190:36947
I20240502 21:25:53.895581 20215 heartbeater.cc:464] Registering TS with master...
I20240502 21:25:53.896907 20101 ts_tablet_manager.cc:1398] T 40d4f61d0f6d4d31a5626c7743c88133 P 90e329b457c24caf93a9f880345eb288: Time spent starting tablet: real 0.038s user 0.014s sys 0.007s
I20240502 21:25:53.896940 20215 heartbeater.cc:511] Master 127.15.255.190:36947 requested a full tablet report, sending...
I20240502 21:25:53.899791 18324 ts_manager.cc:194] Re-registered known tserver with Master: 90e329b457c24caf93a9f880345eb288 (127.15.255.135:43395)
I20240502 21:25:53.901124 18324 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.255.135:44661
I20240502 21:25:53.903656 20215 heartbeater.cc:503] Master 127.15.255.190:36947 was elected leader, sending a full tablet report...
I20240502 21:25:53.906208 16382 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_0 already exists
I20240502 21:25:53.906637 16382 test_util.cc:268] Using random seed: 1065357239
I20240502 21:25:53.922048 16382 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_1 already exists
I20240502 21:25:53.922520 16382 test_util.cc:268] Using random seed: 1065373120
I20240502 21:25:53.940030 16382 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_2 already exists
I20240502 21:25:53.940968 16382 test_util.cc:268] Using random seed: 1065391556
W20240502 21:25:54.127430 20216 tablet.cc:2369] T 40d4f61d0f6d4d31a5626c7743c88133 P 90e329b457c24caf93a9f880345eb288: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240502 21:25:54.182462 18544 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.133:34217, user_credentials={real_user=slave}} blocked reactor thread for 57878.8us
W20240502 21:25:54.360679 18810 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.133:34217, user_credentials={real_user=slave}} blocked reactor thread for 67611.6us
I20240502 21:25:54.387642 16382 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_3 already exists
I20240502 21:25:54.405160 16382 test_util.cc:268] Using random seed: 1065855756
I20240502 21:25:54.890537 16382 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_4 already exists
W20240502 21:25:55.208328 18812 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.134:43635, user_credentials={real_user=slave}} blocked reactor thread for 66457.8us
W20240502 21:25:55.381510 18679 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.132:43607, user_credentials={real_user=slave}} blocked reactor thread for 50529.1us
W20240502 21:25:55.441694 18407 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.134:43635, user_credentials={real_user=slave}} blocked reactor thread for 59862us
W20240502 21:25:55.549875 18407 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.134:43635, user_credentials={real_user=slave}} blocked reactor thread for 59419.2us
W20240502 21:25:55.550318 18543 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.131:42501, user_credentials={real_user=slave}} blocked reactor thread for 53575us
W20240502 21:25:55.550590 18544 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.129:32983, user_credentials={real_user=slave}} blocked reactor thread for 55123.2us
W20240502 21:25:56.861068 18543 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.131:42501, user_credentials={real_user=slave}} blocked reactor thread for 51671.4us
W20240502 21:25:56.961558 19077 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.129:32983, user_credentials={real_user=slave}} blocked reactor thread for 49388.1us
W20240502 21:25:56.990474 18407 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.134:43635, user_credentials={real_user=slave}} blocked reactor thread for 82305.2us
W20240502 21:25:56.995858 19076 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.131:42501, user_credentials={real_user=slave}} blocked reactor thread for 87613.2us
W20240502 21:25:57.679944 19076 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.131:42501, user_credentials={real_user=slave}} blocked reactor thread for 98716us
W20240502 21:25:57.832566 18544 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.133:34217, user_credentials={real_user=slave}} blocked reactor thread for 140037us
I20240502 21:25:58.014914 19124 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:25:58.274371 18857 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:25:58.278081 18590 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:25:58.283313 18989 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:25:58.298040 20147 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 21:25:58.331557 18544 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.133:34217, user_credentials={real_user=slave}} blocked reactor thread for 72347.9us
I20240502 21:25:58.356786 18722 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:25:58.444991 18451 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:25:58.618280 19077 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.133:34217, user_credentials={real_user=slave}} blocked reactor thread for 86979.2us
W20240502 21:25:59.701090 18543 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.131:42501, user_credentials={real_user=slave}} blocked reactor thread for 150394us
W20240502 21:25:59.892069 18543 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.131:42501, user_credentials={real_user=slave}} blocked reactor thread for 77501.8us
W20240502 21:26:01.314406 18812 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.130:45139, user_credentials={real_user=slave}} blocked reactor thread for 79059.9us
W20240502 21:26:01.797451 18406 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.132:43607, user_credentials={real_user=slave}} blocked reactor thread for 63580.9us
W20240502 21:26:01.963425 18544 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.133:34217, user_credentials={real_user=slave}} blocked reactor thread for 67629.1us
W20240502 21:26:02.163033 18544 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.129:32983, user_credentials={real_user=slave}} blocked reactor thread for 64809.7us
W20240502 21:26:02.243378 18543 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.131:42501, user_credentials={real_user=slave}} blocked reactor thread for 261511us
W20240502 21:26:02.925715 18812 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.130:45139, user_credentials={real_user=slave}} blocked reactor thread for 193120us
W20240502 21:26:02.957691 18544 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.133:34217, user_credentials={real_user=slave}} blocked reactor thread for 47403.5us
I20240502 21:26:03.280915 20147 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:26:03.289700 18989 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:26:03.339453 18406 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.132:43607, user_credentials={real_user=slave}} blocked reactor thread for 121817us
I20240502 21:26:03.347348 18722 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:26:03.356750 19124 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:26:03.382663 18857 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 21:26:03.538951 18407 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.130:45139, user_credentials={real_user=slave}} blocked reactor thread for 99788us
I20240502 21:26:03.675619 18590 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:26:03.721139 18451 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:26:03.864785 18543 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.131:42501, user_credentials={real_user=slave}} blocked reactor thread for 131643us
W20240502 21:26:03.893776 18406 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.132:43607, user_credentials={real_user=slave}} blocked reactor thread for 48195.6us
W20240502 21:26:04.124404 18407 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.134:43635, user_credentials={real_user=slave}} blocked reactor thread for 45212.4us
W20240502 21:26:04.245680 18406 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.132:43607, user_credentials={real_user=slave}} blocked reactor thread for 66364.1us
I20240502 21:26:04.506435 18877 consensus_queue.cc:237] T ebf7371ff0ef4de99ced8fec0468af28 P c75831edb3b44dd59c00ef151adca49d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 95, Committed index: 95, Last appended: 1.95, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 96 OBSOLETE_local: false peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } attrs { replace: true } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: NON_VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: true } }
I20240502 21:26:04.519272 19010 raft_consensus.cc:1261] T ebf7371ff0ef4de99ced8fec0468af28 P 9be399440ded4e2ebe6b6e0bf486cbb1 [term 1 FOLLOWER]: Refusing update from remote peer c75831edb3b44dd59c00ef151adca49d: Log matching property violated. Preceding OpId in replica: term: 1 index: 95. Preceding OpId from leader: term: 1 index: 96. (index mismatch)
I20240502 21:26:04.520015 20605 consensus_queue.cc:1035] T ebf7371ff0ef4de99ced8fec0468af28 P c75831edb3b44dd59c00ef151adca49d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 96, Last known committed idx: 95, Time since last communication: 0.000s
I20240502 21:26:04.521250 18471 raft_consensus.cc:1261] T ebf7371ff0ef4de99ced8fec0468af28 P faadabfb099b4abf90a466395be585b5 [term 1 FOLLOWER]: Refusing update from remote peer c75831edb3b44dd59c00ef151adca49d: Log matching property violated. Preceding OpId in replica: term: 1 index: 95. Preceding OpId from leader: term: 1 index: 96. (index mismatch)
I20240502 21:26:04.523586 20605 consensus_queue.cc:1035] T ebf7371ff0ef4de99ced8fec0468af28 P c75831edb3b44dd59c00ef151adca49d [LEADER]: Connected to new peer: Peer: permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 96, Last known committed idx: 95, Time since last communication: 0.000s
I20240502 21:26:04.524127 18741 raft_consensus.cc:1261] T ebf7371ff0ef4de99ced8fec0468af28 P 501e3c0d44fe4d70af9ff926d9622417 [term 1 FOLLOWER]: Refusing update from remote peer c75831edb3b44dd59c00ef151adca49d: Log matching property violated. Preceding OpId in replica: term: 1 index: 95. Preceding OpId from leader: term: 1 index: 96. (index mismatch)
I20240502 21:26:04.527210 20605 consensus_queue.cc:1035] T ebf7371ff0ef4de99ced8fec0468af28 P c75831edb3b44dd59c00ef151adca49d [LEADER]: Connected to new peer: Peer: permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 96, Last known committed idx: 95, Time since last communication: 0.000s
I20240502 21:26:04.530840 20304 raft_consensus.cc:2932] T ebf7371ff0ef4de99ced8fec0468af28 P c75831edb3b44dd59c00ef151adca49d [term 1 LEADER]: Committing config change with OpId 1.96: config changed from index -1 to 96, NON_VOTER 90e329b457c24caf93a9f880345eb288 (127.15.255.135) added. New config: { opid_index: 96 OBSOLETE_local: false peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } attrs { replace: true } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: NON_VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: true } } }
I20240502 21:26:04.532595 18471 raft_consensus.cc:2932] T ebf7371ff0ef4de99ced8fec0468af28 P faadabfb099b4abf90a466395be585b5 [term 1 FOLLOWER]: Committing config change with OpId 1.96: config changed from index -1 to 96, NON_VOTER 90e329b457c24caf93a9f880345eb288 (127.15.255.135) added. New config: { opid_index: 96 OBSOLETE_local: false peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } attrs { replace: true } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: NON_VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: true } } }
I20240502 21:26:04.537756 19010 raft_consensus.cc:2932] T ebf7371ff0ef4de99ced8fec0468af28 P 9be399440ded4e2ebe6b6e0bf486cbb1 [term 1 FOLLOWER]: Committing config change with OpId 1.96: config changed from index -1 to 96, NON_VOTER 90e329b457c24caf93a9f880345eb288 (127.15.255.135) added. New config: { opid_index: 96 OBSOLETE_local: false peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } attrs { replace: true } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: NON_VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: true } } }
I20240502 21:26:04.545737 18741 raft_consensus.cc:2932] T ebf7371ff0ef4de99ced8fec0468af28 P 501e3c0d44fe4d70af9ff926d9622417 [term 1 FOLLOWER]: Committing config change with OpId 1.96: config changed from index -1 to 96, NON_VOTER 90e329b457c24caf93a9f880345eb288 (127.15.255.135) added. New config: { opid_index: 96 OBSOLETE_local: false peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } attrs { replace: true } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: NON_VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: true } } }
I20240502 21:26:04.546897 18321 catalog_manager.cc:5484] T ebf7371ff0ef4de99ced8fec0468af28 P c75831edb3b44dd59c00ef151adca49d reported cstate change: config changed from index -1 to 96, NON_VOTER 90e329b457c24caf93a9f880345eb288 (127.15.255.135) added. New cstate: current_term: 1 leader_uuid: "c75831edb3b44dd59c00ef151adca49d" committed_config { opid_index: 96 OBSOLETE_local: false peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: NON_VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240502 21:26:04.584298 18811 consensus_peers.cc:485] T ebf7371ff0ef4de99ced8fec0468af28 P c75831edb3b44dd59c00ef151adca49d -> Peer 90e329b457c24caf93a9f880345eb288 (127.15.255.135:43395): Couldn't send request to peer 90e329b457c24caf93a9f880345eb288. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ebf7371ff0ef4de99ced8fec0468af28. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:26:04.659736 19141 raft_consensus.cc:1261] T ebf7371ff0ef4de99ced8fec0468af28 P 2877108756304e05bd60ba1186b870da [term 1 FOLLOWER]: Refusing update from remote peer c75831edb3b44dd59c00ef151adca49d: Log matching property violated. Preceding OpId in replica: term: 1 index: 95. Preceding OpId from leader: term: 1 index: 96. (index mismatch)
I20240502 21:26:04.704852 20605 consensus_queue.cc:1035] T ebf7371ff0ef4de99ced8fec0468af28 P c75831edb3b44dd59c00ef151adca49d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 96, Last known committed idx: 95, Time since last communication: 0.000s
I20240502 21:26:04.790382 19008 consensus_queue.cc:237] T e0e69b88ad014363b97d1f650df693a8 P 9be399440ded4e2ebe6b6e0bf486cbb1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 93, Committed index: 93, Last appended: 1.93, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 94 OBSOLETE_local: false peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } attrs { replace: true } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: NON_VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: true } }
I20240502 21:26:04.795014 19141 raft_consensus.cc:1261] T e0e69b88ad014363b97d1f650df693a8 P 2877108756304e05bd60ba1186b870da [term 1 FOLLOWER]: Refusing update from remote peer 9be399440ded4e2ebe6b6e0bf486cbb1: Log matching property violated. Preceding OpId in replica: term: 1 index: 93. Preceding OpId from leader: term: 1 index: 94. (index mismatch)
I20240502 21:26:04.795687 20662 consensus_queue.cc:1035] T e0e69b88ad014363b97d1f650df693a8 P 9be399440ded4e2ebe6b6e0bf486cbb1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 94, Last known committed idx: 93, Time since last communication: 0.000s
I20240502 21:26:04.808923 18876 raft_consensus.cc:1261] T e0e69b88ad014363b97d1f650df693a8 P c75831edb3b44dd59c00ef151adca49d [term 1 FOLLOWER]: Refusing update from remote peer 9be399440ded4e2ebe6b6e0bf486cbb1: Log matching property violated. Preceding OpId in replica: term: 1 index: 93. Preceding OpId from leader: term: 1 index: 94. (index mismatch)
I20240502 21:26:04.809453 20662 consensus_queue.cc:1035] T e0e69b88ad014363b97d1f650df693a8 P 9be399440ded4e2ebe6b6e0bf486cbb1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 94, Last known committed idx: 93, Time since last communication: 0.000s
I20240502 21:26:04.810477 18740 raft_consensus.cc:1261] T e0e69b88ad014363b97d1f650df693a8 P 501e3c0d44fe4d70af9ff926d9622417 [term 1 FOLLOWER]: Refusing update from remote peer 9be399440ded4e2ebe6b6e0bf486cbb1: Log matching property violated. Preceding OpId in replica: term: 1 index: 93. Preceding OpId from leader: term: 1 index: 94. (index mismatch)
I20240502 21:26:04.813578 20662 consensus_queue.cc:1035] T e0e69b88ad014363b97d1f650df693a8 P 9be399440ded4e2ebe6b6e0bf486cbb1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 94, Last known committed idx: 93, Time since last communication: 0.001s
I20240502 21:26:04.814847 20368 raft_consensus.cc:2932] T e0e69b88ad014363b97d1f650df693a8 P 9be399440ded4e2ebe6b6e0bf486cbb1 [term 1 LEADER]: Committing config change with OpId 1.94: config changed from index -1 to 94, NON_VOTER 90e329b457c24caf93a9f880345eb288 (127.15.255.135) added. New config: { opid_index: 94 OBSOLETE_local: false peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } attrs { replace: true } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: NON_VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: true } } }
I20240502 21:26:04.816363 18878 raft_consensus.cc:2932] T e0e69b88ad014363b97d1f650df693a8 P c75831edb3b44dd59c00ef151adca49d [term 1 FOLLOWER]: Committing config change with OpId 1.94: config changed from index -1 to 94, NON_VOTER 90e329b457c24caf93a9f880345eb288 (127.15.255.135) added. New config: { opid_index: 94 OBSOLETE_local: false peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } attrs { replace: true } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: NON_VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: true } } }
W20240502 21:26:04.820020 18543 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.131:42501, user_credentials={real_user=slave}} blocked reactor thread for 99308.5us
W20240502 21:26:04.820663 18544 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.133:34217, user_credentials={real_user=slave}} blocked reactor thread for 73949.6us
I20240502 21:26:04.821301 18608 raft_consensus.cc:1261] T e0e69b88ad014363b97d1f650df693a8 P 82f50f4b66f3450aa9f58a0b317f5cea [term 1 FOLLOWER]: Refusing update from remote peer 9be399440ded4e2ebe6b6e0bf486cbb1: Log matching property violated. Preceding OpId in replica: term: 1 index: 93. Preceding OpId from leader: term: 1 index: 94. (index mismatch)
I20240502 21:26:04.835933 19141 raft_consensus.cc:2932] T ebf7371ff0ef4de99ced8fec0468af28 P 2877108756304e05bd60ba1186b870da [term 1 FOLLOWER]: Committing config change with OpId 1.96: config changed from index -1 to 96, NON_VOTER 90e329b457c24caf93a9f880345eb288 (127.15.255.135) added. New config: { opid_index: 96 OBSOLETE_local: false peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } attrs { replace: true } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: NON_VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: true } } }
I20240502 21:26:04.846323 19143 raft_consensus.cc:2932] T e0e69b88ad014363b97d1f650df693a8 P 2877108756304e05bd60ba1186b870da [term 1 FOLLOWER]: Committing config change with OpId 1.94: config changed from index -1 to 94, NON_VOTER 90e329b457c24caf93a9f880345eb288 (127.15.255.135) added. New config: { opid_index: 94 OBSOLETE_local: false peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } attrs { replace: true } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: NON_VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: true } } }
W20240502 21:26:04.992251 18406 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.132:43607, user_credentials={real_user=slave}} blocked reactor thread for 52971.8us
I20240502 21:26:05.027345 18739 raft_consensus.cc:2932] T e0e69b88ad014363b97d1f650df693a8 P 501e3c0d44fe4d70af9ff926d9622417 [term 1 FOLLOWER]: Committing config change with OpId 1.94: config changed from index -1 to 94, NON_VOTER 90e329b457c24caf93a9f880345eb288 (127.15.255.135) added. New config: { opid_index: 94 OBSOLETE_local: false peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } attrs { replace: true } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: NON_VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: true } } }
I20240502 21:26:05.029532 20646 consensus_queue.cc:1035] T e0e69b88ad014363b97d1f650df693a8 P 9be399440ded4e2ebe6b6e0bf486cbb1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 94, Last known committed idx: 93, Time since last communication: 0.000s
I20240502 21:26:05.074306 20689 ts_tablet_manager.cc:902] T ebf7371ff0ef4de99ced8fec0468af28 P 90e329b457c24caf93a9f880345eb288: Initiating tablet copy from peer c75831edb3b44dd59c00ef151adca49d (127.15.255.132:43607)
W20240502 21:26:05.177292 18945 consensus_peers.cc:485] T e0e69b88ad014363b97d1f650df693a8 P 9be399440ded4e2ebe6b6e0bf486cbb1 -> Peer 90e329b457c24caf93a9f880345eb288 (127.15.255.135:43395): Couldn't send request to peer 90e329b457c24caf93a9f880345eb288. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e0e69b88ad014363b97d1f650df693a8. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:26:05.190340 18323 catalog_manager.cc:5484] T e0e69b88ad014363b97d1f650df693a8 P c75831edb3b44dd59c00ef151adca49d reported cstate change: config changed from index -1 to 94, NON_VOTER 90e329b457c24caf93a9f880345eb288 (127.15.255.135) added. New cstate: current_term: 1 leader_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" committed_config { opid_index: 94 OBSOLETE_local: false peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } attrs { replace: true } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: NON_VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: true } } }
I20240502 21:26:05.220906 20689 tablet_copy_client.cc:308] T ebf7371ff0ef4de99ced8fec0468af28 P 90e329b457c24caf93a9f880345eb288: tablet copy: Beginning tablet copy session from remote peer at address 127.15.255.132:43607
W20240502 21:26:05.236478 18543 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.131:42501, user_credentials={real_user=slave}} blocked reactor thread for 62828.6us
I20240502 21:26:05.259645 18609 raft_consensus.cc:2932] T e0e69b88ad014363b97d1f650df693a8 P 82f50f4b66f3450aa9f58a0b317f5cea [term 1 FOLLOWER]: Committing config change with OpId 1.94: config changed from index -1 to 94, NON_VOTER 90e329b457c24caf93a9f880345eb288 (127.15.255.135) added. New config: { opid_index: 94 OBSOLETE_local: false peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } attrs { replace: true } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: NON_VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: true } } }
I20240502 21:26:05.355001 20702 ts_tablet_manager.cc:902] T e0e69b88ad014363b97d1f650df693a8 P 90e329b457c24caf93a9f880345eb288: Initiating tablet copy from peer 9be399440ded4e2ebe6b6e0bf486cbb1 (127.15.255.133:34217)
I20240502 21:26:05.518388 18898 tablet_copy_service.cc:140] P c75831edb3b44dd59c00ef151adca49d: Received BeginTabletCopySession request for tablet ebf7371ff0ef4de99ced8fec0468af28 from peer 90e329b457c24caf93a9f880345eb288 ({username='slave'} at 127.15.255.135:46619)
I20240502 21:26:05.518654 18898 tablet_copy_service.cc:161] P c75831edb3b44dd59c00ef151adca49d: Beginning new tablet copy session on tablet ebf7371ff0ef4de99ced8fec0468af28 from peer 90e329b457c24caf93a9f880345eb288 at {username='slave'} at 127.15.255.135:46619: session id = 90e329b457c24caf93a9f880345eb288-ebf7371ff0ef4de99ced8fec0468af28
I20240502 21:26:05.522854 18898 tablet_copy_source_session.cc:205] T ebf7371ff0ef4de99ced8fec0468af28 P c75831edb3b44dd59c00ef151adca49d: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:26:05.526187 20689 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ebf7371ff0ef4de99ced8fec0468af28. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:05.536706 20689 tablet_copy_client.cc:783] T ebf7371ff0ef4de99ced8fec0468af28 P 90e329b457c24caf93a9f880345eb288: tablet copy: Starting download of 0 data blocks...
I20240502 21:26:05.537076 20689 tablet_copy_client.cc:647] T ebf7371ff0ef4de99ced8fec0468af28 P 90e329b457c24caf93a9f880345eb288: tablet copy: Starting download of 1 WAL segments...
I20240502 21:26:05.542027 20689 tablet_copy_client.cc:523] T ebf7371ff0ef4de99ced8fec0468af28 P 90e329b457c24caf93a9f880345eb288: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:26:05.545713 20689 tablet_bootstrap.cc:492] T ebf7371ff0ef4de99ced8fec0468af28 P 90e329b457c24caf93a9f880345eb288: Bootstrap starting.
I20240502 21:26:05.647884 20702 tablet_copy_client.cc:308] T e0e69b88ad014363b97d1f650df693a8 P 90e329b457c24caf93a9f880345eb288: tablet copy: Beginning tablet copy session from remote peer at address 127.15.255.133:34217
I20240502 21:26:05.711123 19033 tablet_copy_service.cc:140] P 9be399440ded4e2ebe6b6e0bf486cbb1: Received BeginTabletCopySession request for tablet e0e69b88ad014363b97d1f650df693a8 from peer 90e329b457c24caf93a9f880345eb288 ({username='slave'} at 127.15.255.135:33929)
I20240502 21:26:05.711378 19033 tablet_copy_service.cc:161] P 9be399440ded4e2ebe6b6e0bf486cbb1: Beginning new tablet copy session on tablet e0e69b88ad014363b97d1f650df693a8 from peer 90e329b457c24caf93a9f880345eb288 at {username='slave'} at 127.15.255.135:33929: session id = 90e329b457c24caf93a9f880345eb288-e0e69b88ad014363b97d1f650df693a8
I20240502 21:26:05.730852 19033 tablet_copy_source_session.cc:205] T e0e69b88ad014363b97d1f650df693a8 P 9be399440ded4e2ebe6b6e0bf486cbb1: Tablet Copy: opened 0 blocks and 1 log segments
W20240502 21:26:05.732986 18946 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.134:43635, user_credentials={real_user=slave}} blocked reactor thread for 65367.8us
I20240502 21:26:05.735656 18474 consensus_queue.cc:237] T 6f0107b0690c4c64b0a4f2472a1c8322 P faadabfb099b4abf90a466395be585b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 96, Committed index: 96, Last appended: 1.97, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 98 OBSOLETE_local: false peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } attrs { replace: true } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: NON_VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: true } }
I20240502 21:26:05.744643 18609 raft_consensus.cc:1261] T 6f0107b0690c4c64b0a4f2472a1c8322 P 82f50f4b66f3450aa9f58a0b317f5cea [term 1 FOLLOWER]: Refusing update from remote peer faadabfb099b4abf90a466395be585b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 96. Preceding OpId from leader: term: 1 index: 98. (index mismatch)
W20240502 21:26:05.748330 18406 consensus_peers.cc:485] T 6f0107b0690c4c64b0a4f2472a1c8322 P faadabfb099b4abf90a466395be585b5 -> Peer 90e329b457c24caf93a9f880345eb288 (127.15.255.135:43395): Couldn't send request to peer 90e329b457c24caf93a9f880345eb288. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6f0107b0690c4c64b0a4f2472a1c8322. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:26:05.750303 20636 consensus_queue.cc:1035] T 6f0107b0690c4c64b0a4f2472a1c8322 P faadabfb099b4abf90a466395be585b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 98, Last known committed idx: 96, Time since last communication: 0.004s
I20240502 21:26:05.777407 20702 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0e69b88ad014363b97d1f650df693a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:05.788300 20702 tablet_copy_client.cc:783] T e0e69b88ad014363b97d1f650df693a8 P 90e329b457c24caf93a9f880345eb288: tablet copy: Starting download of 0 data blocks...
I20240502 21:26:05.788414 18877 raft_consensus.cc:1261] T 6f0107b0690c4c64b0a4f2472a1c8322 P c75831edb3b44dd59c00ef151adca49d [term 1 FOLLOWER]: Refusing update from remote peer faadabfb099b4abf90a466395be585b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 96. Preceding OpId from leader: term: 1 index: 98. (index mismatch)
I20240502 21:26:05.788645 20702 tablet_copy_client.cc:647] T e0e69b88ad014363b97d1f650df693a8 P 90e329b457c24caf93a9f880345eb288: tablet copy: Starting download of 1 WAL segments...
I20240502 21:26:05.790397 20058 consensus_queue.cc:1035] T 6f0107b0690c4c64b0a4f2472a1c8322 P faadabfb099b4abf90a466395be585b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 98, Last known committed idx: 96, Time since last communication: 0.000s
I20240502 21:26:05.792450 19144 raft_consensus.cc:1261] T 6f0107b0690c4c64b0a4f2472a1c8322 P 2877108756304e05bd60ba1186b870da [term 1 FOLLOWER]: Refusing update from remote peer faadabfb099b4abf90a466395be585b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 96. Preceding OpId from leader: term: 1 index: 98. (index mismatch)
I20240502 21:26:05.795466 20702 tablet_copy_client.cc:523] T e0e69b88ad014363b97d1f650df693a8 P 90e329b457c24caf93a9f880345eb288: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:26:05.796272 20636 consensus_queue.cc:1035] T 6f0107b0690c4c64b0a4f2472a1c8322 P faadabfb099b4abf90a466395be585b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 98, Last known committed idx: 96, Time since last communication: 0.000s
I20240502 21:26:05.798687 20702 tablet_bootstrap.cc:492] T e0e69b88ad014363b97d1f650df693a8 P 90e329b457c24caf93a9f880345eb288: Bootstrap starting.
I20240502 21:26:05.805581 20636 raft_consensus.cc:2932] T 6f0107b0690c4c64b0a4f2472a1c8322 P faadabfb099b4abf90a466395be585b5 [term 1 LEADER]: Committing config change with OpId 1.98: config changed from index -1 to 98, NON_VOTER 90e329b457c24caf93a9f880345eb288 (127.15.255.135) added. New config: { opid_index: 98 OBSOLETE_local: false peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } attrs { replace: true } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: NON_VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: true } } }
I20240502 21:26:05.821867 18876 raft_consensus.cc:2932] T 6f0107b0690c4c64b0a4f2472a1c8322 P c75831edb3b44dd59c00ef151adca49d [term 1 FOLLOWER]: Committing config change with OpId 1.98: config changed from index -1 to 98, NON_VOTER 90e329b457c24caf93a9f880345eb288 (127.15.255.135) added. New config: { opid_index: 98 OBSOLETE_local: false peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } attrs { replace: true } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: NON_VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: true } } }
I20240502 21:26:05.826195 18609 raft_consensus.cc:2932] T 6f0107b0690c4c64b0a4f2472a1c8322 P 82f50f4b66f3450aa9f58a0b317f5cea [term 1 FOLLOWER]: Committing config change with OpId 1.98: config changed from index -1 to 98, NON_VOTER 90e329b457c24caf93a9f880345eb288 (127.15.255.135) added. New config: { opid_index: 98 OBSOLETE_local: false peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } attrs { replace: true } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: NON_VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: true } } }
W20240502 21:26:05.828073 18946 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.130:45139, user_credentials={real_user=slave}} blocked reactor thread for 46539.8us
I20240502 21:26:05.830155 18327 catalog_manager.cc:5484] T 6f0107b0690c4c64b0a4f2472a1c8322 P faadabfb099b4abf90a466395be585b5 reported cstate change: config changed from index -1 to 98, NON_VOTER 90e329b457c24caf93a9f880345eb288 (127.15.255.135) added. New cstate: current_term: 1 leader_uuid: "faadabfb099b4abf90a466395be585b5" committed_config { opid_index: 98 OBSOLETE_local: false peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: NON_VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 21:26:05.884080 19143 raft_consensus.cc:2932] T 6f0107b0690c4c64b0a4f2472a1c8322 P 2877108756304e05bd60ba1186b870da [term 1 FOLLOWER]: Committing config change with OpId 1.98: config changed from index -1 to 98, NON_VOTER 90e329b457c24caf93a9f880345eb288 (127.15.255.135) added. New config: { opid_index: 98 OBSOLETE_local: false peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } attrs { replace: true } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: NON_VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: true } } }
I20240502 21:26:05.921139 19006 raft_consensus.cc:1261] T 6f0107b0690c4c64b0a4f2472a1c8322 P 9be399440ded4e2ebe6b6e0bf486cbb1 [term 1 FOLLOWER]: Refusing update from remote peer faadabfb099b4abf90a466395be585b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 96. Preceding OpId from leader: term: 1 index: 98. (index mismatch)
W20240502 21:26:05.960713 19077 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.129:32983, user_credentials={real_user=slave}} blocked reactor thread for 46908.1us
I20240502 21:26:05.980541 20712 consensus_queue.cc:1035] T 6f0107b0690c4c64b0a4f2472a1c8322 P faadabfb099b4abf90a466395be585b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 98, Last known committed idx: 96, Time since last communication: 0.000s
I20240502 21:26:06.005331 19008 raft_consensus.cc:2932] T 6f0107b0690c4c64b0a4f2472a1c8322 P 9be399440ded4e2ebe6b6e0bf486cbb1 [term 1 FOLLOWER]: Committing config change with OpId 1.98: config changed from index -1 to 98, NON_VOTER 90e329b457c24caf93a9f880345eb288 (127.15.255.135) added. New config: { opid_index: 98 OBSOLETE_local: false peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } attrs { replace: true } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: NON_VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: true } } }
I20240502 21:26:06.391467 20702 tablet_bootstrap.cc:492] T e0e69b88ad014363b97d1f650df693a8 P 90e329b457c24caf93a9f880345eb288: Bootstrap replayed 1/1 log segments. Stats: ops{read=107 overwritten=0 applied=107 ignored=0} inserts{seen=738 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:26:06.395345 20702 tablet_bootstrap.cc:492] T e0e69b88ad014363b97d1f650df693a8 P 90e329b457c24caf93a9f880345eb288: Bootstrap complete.
I20240502 21:26:06.395792 20702 ts_tablet_manager.cc:1367] T e0e69b88ad014363b97d1f650df693a8 P 90e329b457c24caf93a9f880345eb288: Time spent bootstrapping tablet: real 0.597s user 0.162s sys 0.040s
I20240502 21:26:06.396992 20702 raft_consensus.cc:348] T e0e69b88ad014363b97d1f650df693a8 P 90e329b457c24caf93a9f880345eb288 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 94 OBSOLETE_local: false peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } attrs { replace: true } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: NON_VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: true } }
I20240502 21:26:06.400169 20702 raft_consensus.cc:729] T e0e69b88ad014363b97d1f650df693a8 P 90e329b457c24caf93a9f880345eb288 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 90e329b457c24caf93a9f880345eb288, State: Initialized, Role: LEARNER
I20240502 21:26:06.400569 20702 consensus_queue.cc:260] T e0e69b88ad014363b97d1f650df693a8 P 90e329b457c24caf93a9f880345eb288 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 107, Last appended: 1.107, Last appended by leader: 107, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 94 OBSOLETE_local: false peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } attrs { replace: true } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: NON_VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: true } }
I20240502 21:26:06.417459 20737 ts_tablet_manager.cc:902] T 6f0107b0690c4c64b0a4f2472a1c8322 P 90e329b457c24caf93a9f880345eb288: Initiating tablet copy from peer faadabfb099b4abf90a466395be585b5 (127.15.255.129:32983)
I20240502 21:26:06.449517 18472 consensus_queue.cc:237] T 84b235fd89634168938c0dddb534462a P faadabfb099b4abf90a466395be585b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 116, Committed index: 116, Last appended: 1.117, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 118 OBSOLETE_local: false peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } attrs { replace: true } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: NON_VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: true } }
I20240502 21:26:06.452790 19143 raft_consensus.cc:1261] T 84b235fd89634168938c0dddb534462a P 2877108756304e05bd60ba1186b870da [term 1 FOLLOWER]: Refusing update from remote peer faadabfb099b4abf90a466395be585b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 117. Preceding OpId from leader: term: 1 index: 118. (index mismatch)
I20240502 21:26:06.453701 20636 consensus_queue.cc:1035] T 84b235fd89634168938c0dddb534462a P faadabfb099b4abf90a466395be585b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 118, Last known committed idx: 116, Time since last communication: 0.000s
W20240502 21:26:06.454752 18406 consensus_peers.cc:485] T 84b235fd89634168938c0dddb534462a P faadabfb099b4abf90a466395be585b5 -> Peer 90e329b457c24caf93a9f880345eb288 (127.15.255.135:43395): Couldn't send request to peer 90e329b457c24caf93a9f880345eb288. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 84b235fd89634168938c0dddb534462a. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:26:06.455066 18611 raft_consensus.cc:1261] T 84b235fd89634168938c0dddb534462a P 82f50f4b66f3450aa9f58a0b317f5cea [term 1 FOLLOWER]: Refusing update from remote peer faadabfb099b4abf90a466395be585b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 117. Preceding OpId from leader: term: 1 index: 118. (index mismatch)
I20240502 21:26:06.455765 20636 consensus_queue.cc:1035] T 84b235fd89634168938c0dddb534462a P faadabfb099b4abf90a466395be585b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 118, Last known committed idx: 116, Time since last communication: 0.000s
I20240502 21:26:06.462075 19007 raft_consensus.cc:1261] T 84b235fd89634168938c0dddb534462a P 9be399440ded4e2ebe6b6e0bf486cbb1 [term 1 FOLLOWER]: Refusing update from remote peer faadabfb099b4abf90a466395be585b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 116. Preceding OpId from leader: term: 1 index: 118. (index mismatch)
I20240502 21:26:06.462935 20569 raft_consensus.cc:2932] T 84b235fd89634168938c0dddb534462a P faadabfb099b4abf90a466395be585b5 [term 1 LEADER]: Committing config change with OpId 1.118: config changed from index -1 to 118, NON_VOTER 90e329b457c24caf93a9f880345eb288 (127.15.255.135) added. New config: { opid_index: 118 OBSOLETE_local: false peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } attrs { replace: true } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: NON_VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: true } } }
I20240502 21:26:06.463491 20446 consensus_queue.cc:1035] T 84b235fd89634168938c0dddb534462a P faadabfb099b4abf90a466395be585b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 118, Last known committed idx: 115, Time since last communication: 0.000s
I20240502 21:26:06.467141 18611 raft_consensus.cc:2932] T 84b235fd89634168938c0dddb534462a P 82f50f4b66f3450aa9f58a0b317f5cea [term 1 FOLLOWER]: Committing config change with OpId 1.118: config changed from index -1 to 118, NON_VOTER 90e329b457c24caf93a9f880345eb288 (127.15.255.135) added. New config: { opid_index: 118 OBSOLETE_local: false peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } attrs { replace: true } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: NON_VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: true } } }
I20240502 21:26:06.467615 19007 raft_consensus.cc:2932] T 84b235fd89634168938c0dddb534462a P 9be399440ded4e2ebe6b6e0bf486cbb1 [term 1 FOLLOWER]: Committing config change with OpId 1.118: config changed from index -1 to 118, NON_VOTER 90e329b457c24caf93a9f880345eb288 (127.15.255.135) added. New config: { opid_index: 118 OBSOLETE_local: false peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } attrs { replace: true } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: NON_VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: true } } }
I20240502 21:26:06.472781 18323 catalog_manager.cc:5484] T 84b235fd89634168938c0dddb534462a P 82f50f4b66f3450aa9f58a0b317f5cea reported cstate change: config changed from index -1 to 118, NON_VOTER 90e329b457c24caf93a9f880345eb288 (127.15.255.135) added. New cstate: current_term: 1 leader_uuid: "faadabfb099b4abf90a466395be585b5" committed_config { opid_index: 118 OBSOLETE_local: false peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } attrs { replace: true } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: NON_VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: true } } }
I20240502 21:26:06.480202 19143 raft_consensus.cc:2932] T 84b235fd89634168938c0dddb534462a P 2877108756304e05bd60ba1186b870da [term 1 FOLLOWER]: Committing config change with OpId 1.118: config changed from index -1 to 118, NON_VOTER 90e329b457c24caf93a9f880345eb288 (127.15.255.135) added. New config: { opid_index: 118 OBSOLETE_local: false peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } attrs { replace: true } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: NON_VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: true } } }
I20240502 21:26:06.539537 20689 tablet_bootstrap.cc:492] T ebf7371ff0ef4de99ced8fec0468af28 P 90e329b457c24caf93a9f880345eb288: Bootstrap replayed 1/1 log segments. Stats: ops{read=106 overwritten=0 applied=105 ignored=0} inserts{seen=790 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240502 21:26:06.540096 20689 tablet_bootstrap.cc:492] T ebf7371ff0ef4de99ced8fec0468af28 P 90e329b457c24caf93a9f880345eb288: Bootstrap complete.
I20240502 21:26:06.540627 20737 tablet_copy_client.cc:308] T 6f0107b0690c4c64b0a4f2472a1c8322 P 90e329b457c24caf93a9f880345eb288: tablet copy: Beginning tablet copy session from remote peer at address 127.15.255.129:32983
I20240502 21:26:06.558472 20702 ts_tablet_manager.cc:1398] T e0e69b88ad014363b97d1f650df693a8 P 90e329b457c24caf93a9f880345eb288: Time spent starting tablet: real 0.162s user 0.010s sys 0.004s
I20240502 21:26:06.584930 18878 raft_consensus.cc:1261] T 84b235fd89634168938c0dddb534462a P c75831edb3b44dd59c00ef151adca49d [term 1 FOLLOWER]: Refusing update from remote peer faadabfb099b4abf90a466395be585b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 117. Preceding OpId from leader: term: 1 index: 118. (index mismatch)
I20240502 21:26:06.656195 19033 tablet_copy_service.cc:342] P 9be399440ded4e2ebe6b6e0bf486cbb1: Request end of tablet copy session 90e329b457c24caf93a9f880345eb288-e0e69b88ad014363b97d1f650df693a8 received from {username='slave'} at 127.15.255.135:33929
I20240502 21:26:06.656389 19033 tablet_copy_service.cc:433] P 9be399440ded4e2ebe6b6e0bf486cbb1: ending tablet copy session 90e329b457c24caf93a9f880345eb288-e0e69b88ad014363b97d1f650df693a8 on tablet e0e69b88ad014363b97d1f650df693a8 with peer 90e329b457c24caf93a9f880345eb288
I20240502 21:26:06.671387 20689 ts_tablet_manager.cc:1367] T ebf7371ff0ef4de99ced8fec0468af28 P 90e329b457c24caf93a9f880345eb288: Time spent bootstrapping tablet: real 0.995s user 0.169s sys 0.008s
I20240502 21:26:06.672528 20689 raft_consensus.cc:348] T ebf7371ff0ef4de99ced8fec0468af28 P 90e329b457c24caf93a9f880345eb288 [term 1 LEARNER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 96 OBSOLETE_local: false peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } attrs { replace: true } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: NON_VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: true } }
I20240502 21:26:06.681475 18496 tablet_copy_service.cc:140] P faadabfb099b4abf90a466395be585b5: Received BeginTabletCopySession request for tablet 6f0107b0690c4c64b0a4f2472a1c8322 from peer 90e329b457c24caf93a9f880345eb288 ({username='slave'} at 127.15.255.135:49783)
I20240502 21:26:06.681767 18496 tablet_copy_service.cc:161] P faadabfb099b4abf90a466395be585b5: Beginning new tablet copy session on tablet 6f0107b0690c4c64b0a4f2472a1c8322 from peer 90e329b457c24caf93a9f880345eb288 at {username='slave'} at 127.15.255.135:49783: session id = 90e329b457c24caf93a9f880345eb288-6f0107b0690c4c64b0a4f2472a1c8322
I20240502 21:26:06.686316 18496 tablet_copy_source_session.cc:205] T 6f0107b0690c4c64b0a4f2472a1c8322 P faadabfb099b4abf90a466395be585b5: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:26:06.692404 20737 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f0107b0690c4c64b0a4f2472a1c8322. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:06.699385 20737 tablet_copy_client.cc:783] T 6f0107b0690c4c64b0a4f2472a1c8322 P 90e329b457c24caf93a9f880345eb288: tablet copy: Starting download of 0 data blocks...
I20240502 21:26:06.699774 20737 tablet_copy_client.cc:647] T 6f0107b0690c4c64b0a4f2472a1c8322 P 90e329b457c24caf93a9f880345eb288: tablet copy: Starting download of 1 WAL segments...
I20240502 21:26:06.702790 20737 tablet_copy_client.cc:523] T 6f0107b0690c4c64b0a4f2472a1c8322 P 90e329b457c24caf93a9f880345eb288: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:26:06.737284 20737 tablet_bootstrap.cc:492] T 6f0107b0690c4c64b0a4f2472a1c8322 P 90e329b457c24caf93a9f880345eb288: Bootstrap starting.
I20240502 21:26:06.744248 20689 raft_consensus.cc:729] T ebf7371ff0ef4de99ced8fec0468af28 P 90e329b457c24caf93a9f880345eb288 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 90e329b457c24caf93a9f880345eb288, State: Initialized, Role: LEARNER
I20240502 21:26:06.744676 20689 consensus_queue.cc:260] T ebf7371ff0ef4de99ced8fec0468af28 P 90e329b457c24caf93a9f880345eb288 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 105, Last appended: 1.106, Last appended by leader: 106, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 96 OBSOLETE_local: false peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } attrs { replace: true } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: NON_VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: true } }
I20240502 21:26:06.748958 20689 ts_tablet_manager.cc:1398] T ebf7371ff0ef4de99ced8fec0468af28 P 90e329b457c24caf93a9f880345eb288: Time spent starting tablet: real 0.077s user 0.004s sys 0.000s
I20240502 21:26:06.752506 18898 tablet_copy_service.cc:342] P c75831edb3b44dd59c00ef151adca49d: Request end of tablet copy session 90e329b457c24caf93a9f880345eb288-ebf7371ff0ef4de99ced8fec0468af28 received from {username='slave'} at 127.15.255.135:46619
I20240502 21:26:06.752714 18898 tablet_copy_service.cc:433] P c75831edb3b44dd59c00ef151adca49d: ending tablet copy session 90e329b457c24caf93a9f880345eb288-ebf7371ff0ef4de99ced8fec0468af28 on tablet ebf7371ff0ef4de99ced8fec0468af28 with peer 90e329b457c24caf93a9f880345eb288
I20240502 21:26:06.777385 20636 consensus_queue.cc:1035] T 84b235fd89634168938c0dddb534462a P faadabfb099b4abf90a466395be585b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 118, Last known committed idx: 116, Time since last communication: 0.000s
I20240502 21:26:06.834223 18878 raft_consensus.cc:2932] T 84b235fd89634168938c0dddb534462a P c75831edb3b44dd59c00ef151adca49d [term 1 FOLLOWER]: Committing config change with OpId 1.118: config changed from index -1 to 118, NON_VOTER 90e329b457c24caf93a9f880345eb288 (127.15.255.135) added. New config: { opid_index: 118 OBSOLETE_local: false peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } attrs { replace: true } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: NON_VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: true } } }
I20240502 21:26:06.854072 20689 ts_tablet_manager.cc:902] T 84b235fd89634168938c0dddb534462a P 90e329b457c24caf93a9f880345eb288: Initiating tablet copy from peer faadabfb099b4abf90a466395be585b5 (127.15.255.129:32983)
I20240502 21:26:06.856792 18876 consensus_queue.cc:237] T 37e9dcc6191c42d3a5c1503dedc5511e P c75831edb3b44dd59c00ef151adca49d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 115, Committed index: 115, Last appended: 1.116, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 117 OBSOLETE_local: false peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } attrs { replace: true } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: NON_VOTER last_known_addr { host: "127.15.255.129" port: 32983 } attrs { promote: true } }
I20240502 21:26:06.861394 18738 raft_consensus.cc:1261] T 37e9dcc6191c42d3a5c1503dedc5511e P 501e3c0d44fe4d70af9ff926d9622417 [term 1 FOLLOWER]: Refusing update from remote peer c75831edb3b44dd59c00ef151adca49d: Log matching property violated. Preceding OpId in replica: term: 1 index: 116. Preceding OpId from leader: term: 1 index: 117. (index mismatch)
I20240502 21:26:06.864264 19007 raft_consensus.cc:1261] T 37e9dcc6191c42d3a5c1503dedc5511e P 9be399440ded4e2ebe6b6e0bf486cbb1 [term 1 FOLLOWER]: Refusing update from remote peer c75831edb3b44dd59c00ef151adca49d: Log matching property violated. Preceding OpId in replica: term: 1 index: 116. Preceding OpId from leader: term: 1 index: 117. (index mismatch)
W20240502 21:26:06.865064 18810 consensus_peers.cc:485] T 37e9dcc6191c42d3a5c1503dedc5511e P c75831edb3b44dd59c00ef151adca49d -> Peer faadabfb099b4abf90a466395be585b5 (127.15.255.129:32983): Couldn't send request to peer faadabfb099b4abf90a466395be585b5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 37e9dcc6191c42d3a5c1503dedc5511e. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:26:06.868283 20606 consensus_queue.cc:1035] T 37e9dcc6191c42d3a5c1503dedc5511e P c75831edb3b44dd59c00ef151adca49d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 117, Last known committed idx: 115, Time since last communication: 0.000s
I20240502 21:26:06.873669 20606 consensus_queue.cc:1035] T 37e9dcc6191c42d3a5c1503dedc5511e P c75831edb3b44dd59c00ef151adca49d [LEADER]: Connected to new peer: Peer: permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 117, Last known committed idx: 115, Time since last communication: 0.000s
I20240502 21:26:06.881315 20169 raft_consensus.cc:1203] T e0e69b88ad014363b97d1f650df693a8 P 90e329b457c24caf93a9f880345eb288 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.93->[1.94-1.121] Dedup: 1.107->[1.108-1.121]
I20240502 21:26:06.883349 20304 raft_consensus.cc:2932] T 37e9dcc6191c42d3a5c1503dedc5511e P c75831edb3b44dd59c00ef151adca49d [term 1 LEADER]: Committing config change with OpId 1.117: config changed from index -1 to 117, NON_VOTER faadabfb099b4abf90a466395be585b5 (127.15.255.129) added. New config: { opid_index: 117 OBSOLETE_local: false peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } attrs { replace: true } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: NON_VOTER last_known_addr { host: "127.15.255.129" port: 32983 } attrs { promote: true } } }
I20240502 21:26:06.894093 18738 raft_consensus.cc:2932] T 37e9dcc6191c42d3a5c1503dedc5511e P 501e3c0d44fe4d70af9ff926d9622417 [term 1 FOLLOWER]: Committing config change with OpId 1.117: config changed from index -1 to 117, NON_VOTER faadabfb099b4abf90a466395be585b5 (127.15.255.129) added. New config: { opid_index: 117 OBSOLETE_local: false peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } attrs { replace: true } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: NON_VOTER last_known_addr { host: "127.15.255.129" port: 32983 } attrs { promote: true } } }
I20240502 21:26:06.904289 18321 catalog_manager.cc:5484] T 37e9dcc6191c42d3a5c1503dedc5511e P 501e3c0d44fe4d70af9ff926d9622417 reported cstate change: config changed from index -1 to 117, NON_VOTER faadabfb099b4abf90a466395be585b5 (127.15.255.129) added. New cstate: current_term: 1 leader_uuid: "c75831edb3b44dd59c00ef151adca49d" committed_config { opid_index: 117 OBSOLETE_local: false peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } attrs { replace: true } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: NON_VOTER last_known_addr { host: "127.15.255.129" port: 32983 } attrs { promote: true } } }
I20240502 21:26:06.904371 19005 raft_consensus.cc:2932] T 37e9dcc6191c42d3a5c1503dedc5511e P 9be399440ded4e2ebe6b6e0bf486cbb1 [term 1 FOLLOWER]: Committing config change with OpId 1.117: config changed from index -1 to 117, NON_VOTER faadabfb099b4abf90a466395be585b5 (127.15.255.129) added. New config: { opid_index: 117 OBSOLETE_local: false peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } attrs { replace: true } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: NON_VOTER last_known_addr { host: "127.15.255.129" port: 32983 } attrs { promote: true } } }
I20240502 21:26:06.940217 20167 raft_consensus.cc:1203] T ebf7371ff0ef4de99ced8fec0468af28 P 90e329b457c24caf93a9f880345eb288 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.95->[1.96-1.113] Dedup: 1.106->[1.107-1.113]
I20240502 21:26:07.004535 19140 raft_consensus.cc:1261] T 37e9dcc6191c42d3a5c1503dedc5511e P 2877108756304e05bd60ba1186b870da [term 1 FOLLOWER]: Refusing update from remote peer c75831edb3b44dd59c00ef151adca49d: Log matching property violated. Preceding OpId in replica: term: 1 index: 112. Preceding OpId from leader: term: 1 index: 117. (index mismatch)
I20240502 21:26:07.114077 18608 consensus_queue.cc:237] T a7a9256f2b784abcb780b190e2ff3983 P 82f50f4b66f3450aa9f58a0b317f5cea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 120, Committed index: 120, Last appended: 1.120, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 121 OBSOLETE_local: false peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } attrs { replace: true } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: NON_VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: true } }
I20240502 21:26:07.134907 18472 raft_consensus.cc:1261] T a7a9256f2b784abcb780b190e2ff3983 P faadabfb099b4abf90a466395be585b5 [term 1 FOLLOWER]: Refusing update from remote peer 82f50f4b66f3450aa9f58a0b317f5cea: Log matching property violated. Preceding OpId in replica: term: 1 index: 120. Preceding OpId from leader: term: 1 index: 121. (index mismatch)
I20240502 21:26:07.135989 18737 raft_consensus.cc:1261] T a7a9256f2b784abcb780b190e2ff3983 P 501e3c0d44fe4d70af9ff926d9622417 [term 1 FOLLOWER]: Refusing update from remote peer 82f50f4b66f3450aa9f58a0b317f5cea: Log matching property violated. Preceding OpId in replica: term: 1 index: 120. Preceding OpId from leader: term: 1 index: 121. (index mismatch)
I20240502 21:26:07.136900 20763 consensus_queue.cc:1035] T a7a9256f2b784abcb780b190e2ff3983 P 82f50f4b66f3450aa9f58a0b317f5cea [LEADER]: Connected to new peer: Peer: permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 121, Last known committed idx: 120, Time since last communication: 0.000s
I20240502 21:26:07.137261 20672 consensus_queue.cc:1035] T a7a9256f2b784abcb780b190e2ff3983 P 82f50f4b66f3450aa9f58a0b317f5cea [LEADER]: Connected to new peer: Peer: permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 121, Last known committed idx: 120, Time since last communication: 0.000s
I20240502 21:26:07.137457 19010 raft_consensus.cc:1261] T a7a9256f2b784abcb780b190e2ff3983 P 9be399440ded4e2ebe6b6e0bf486cbb1 [term 1 FOLLOWER]: Refusing update from remote peer 82f50f4b66f3450aa9f58a0b317f5cea: Log matching property violated. Preceding OpId in replica: term: 1 index: 120. Preceding OpId from leader: term: 1 index: 121. (index mismatch)
I20240502 21:26:07.141290 20763 consensus_queue.cc:1035] T a7a9256f2b784abcb780b190e2ff3983 P 82f50f4b66f3450aa9f58a0b317f5cea [LEADER]: Connected to new peer: Peer: permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 121, Last known committed idx: 120, Time since last communication: 0.000s
I20240502 21:26:07.141984 20672 raft_consensus.cc:2932] T a7a9256f2b784abcb780b190e2ff3983 P 82f50f4b66f3450aa9f58a0b317f5cea [term 1 LEADER]: Committing config change with OpId 1.121: config changed from index -1 to 121, NON_VOTER 90e329b457c24caf93a9f880345eb288 (127.15.255.135) added. New config: { opid_index: 121 OBSOLETE_local: false peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } attrs { replace: true } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: NON_VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: true } } }
I20240502 21:26:07.142377 18472 raft_consensus.cc:2932] T a7a9256f2b784abcb780b190e2ff3983 P faadabfb099b4abf90a466395be585b5 [term 1 FOLLOWER]: Committing config change with OpId 1.121: config changed from index -1 to 121, NON_VOTER 90e329b457c24caf93a9f880345eb288 (127.15.255.135) added. New config: { opid_index: 121 OBSOLETE_local: false peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } attrs { replace: true } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: NON_VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: true } } }
I20240502 21:26:07.147711 18738 raft_consensus.cc:2932] T a7a9256f2b784abcb780b190e2ff3983 P 501e3c0d44fe4d70af9ff926d9622417 [term 1 FOLLOWER]: Committing config change with OpId 1.121: config changed from index -1 to 121, NON_VOTER 90e329b457c24caf93a9f880345eb288 (127.15.255.135) added. New config: { opid_index: 121 OBSOLETE_local: false peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } attrs { replace: true } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: NON_VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: true } } }
I20240502 21:26:07.148780 18321 catalog_manager.cc:5484] T a7a9256f2b784abcb780b190e2ff3983 P faadabfb099b4abf90a466395be585b5 reported cstate change: config changed from index -1 to 121, NON_VOTER 90e329b457c24caf93a9f880345eb288 (127.15.255.135) added. New cstate: current_term: 1 leader_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" committed_config { opid_index: 121 OBSOLETE_local: false peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } attrs { replace: true } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: NON_VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: true } } }
I20240502 21:26:07.154281 19010 raft_consensus.cc:2932] T a7a9256f2b784abcb780b190e2ff3983 P 9be399440ded4e2ebe6b6e0bf486cbb1 [term 1 FOLLOWER]: Committing config change with OpId 1.121: config changed from index -1 to 121, NON_VOTER 90e329b457c24caf93a9f880345eb288 (127.15.255.135) added. New config: { opid_index: 121 OBSOLETE_local: false peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } attrs { replace: true } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: NON_VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: true } } }
I20240502 21:26:07.182740 18608 raft_consensus.cc:1261] T 37e9dcc6191c42d3a5c1503dedc5511e P 82f50f4b66f3450aa9f58a0b317f5cea [term 1 FOLLOWER]: Refusing update from remote peer c75831edb3b44dd59c00ef151adca49d: Log matching property violated. Preceding OpId in replica: term: 1 index: 114. Preceding OpId from leader: term: 1 index: 117. (index mismatch)
W20240502 21:26:07.316476 18812 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.134:43635, user_credentials={real_user=slave}} blocked reactor thread for 67387.6us
I20240502 21:26:07.328353 20662 raft_consensus.cc:1050] T e0e69b88ad014363b97d1f650df693a8 P 9be399440ded4e2ebe6b6e0bf486cbb1: attempting to promote NON_VOTER 90e329b457c24caf93a9f880345eb288 to VOTER
I20240502 21:26:07.328760 20689 tablet_copy_client.cc:308] T 84b235fd89634168938c0dddb534462a P 90e329b457c24caf93a9f880345eb288: tablet copy: Beginning tablet copy session from remote peer at address 127.15.255.129:32983
I20240502 21:26:07.330796 18496 tablet_copy_service.cc:140] P faadabfb099b4abf90a466395be585b5: Received BeginTabletCopySession request for tablet 84b235fd89634168938c0dddb534462a from peer 90e329b457c24caf93a9f880345eb288 ({username='slave'} at 127.15.255.135:49783)
I20240502 21:26:07.331002 18496 tablet_copy_service.cc:161] P faadabfb099b4abf90a466395be585b5: Beginning new tablet copy session on tablet 84b235fd89634168938c0dddb534462a from peer 90e329b457c24caf93a9f880345eb288 at {username='slave'} at 127.15.255.135:49783: session id = 90e329b457c24caf93a9f880345eb288-84b235fd89634168938c0dddb534462a
I20240502 21:26:07.334440 18496 tablet_copy_source_session.cc:205] T 84b235fd89634168938c0dddb534462a P faadabfb099b4abf90a466395be585b5: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:26:07.335680 20689 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84b235fd89634168938c0dddb534462a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:07.336401 20662 consensus_queue.cc:237] T e0e69b88ad014363b97d1f650df693a8 P 9be399440ded4e2ebe6b6e0bf486cbb1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 128, Committed index: 128, Last appended: 1.128, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 129 OBSOLETE_local: false peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } attrs { replace: true } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } }
I20240502 21:26:07.339705 18738 raft_consensus.cc:1261] T e0e69b88ad014363b97d1f650df693a8 P 501e3c0d44fe4d70af9ff926d9622417 [term 1 FOLLOWER]: Refusing update from remote peer 9be399440ded4e2ebe6b6e0bf486cbb1: Log matching property violated. Preceding OpId in replica: term: 1 index: 128. Preceding OpId from leader: term: 1 index: 129. (index mismatch)
I20240502 21:26:07.340276 20429 consensus_queue.cc:1035] T e0e69b88ad014363b97d1f650df693a8 P 9be399440ded4e2ebe6b6e0bf486cbb1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 129, Last known committed idx: 128, Time since last communication: 0.000s
I20240502 21:26:07.341805 18608 raft_consensus.cc:1261] T e0e69b88ad014363b97d1f650df693a8 P 82f50f4b66f3450aa9f58a0b317f5cea [term 1 FOLLOWER]: Refusing update from remote peer 9be399440ded4e2ebe6b6e0bf486cbb1: Log matching property violated. Preceding OpId in replica: term: 1 index: 128. Preceding OpId from leader: term: 1 index: 129. (index mismatch)
I20240502 21:26:07.342509 20429 consensus_queue.cc:1035] T e0e69b88ad014363b97d1f650df693a8 P 9be399440ded4e2ebe6b6e0bf486cbb1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 129, Last known committed idx: 128, Time since last communication: 0.000s
I20240502 21:26:07.347138 20167 raft_consensus.cc:1261] T e0e69b88ad014363b97d1f650df693a8 P 90e329b457c24caf93a9f880345eb288 [term 1 LEARNER]: Refusing update from remote peer 9be399440ded4e2ebe6b6e0bf486cbb1: Log matching property violated. Preceding OpId in replica: term: 1 index: 128. Preceding OpId from leader: term: 1 index: 129. (index mismatch)
I20240502 21:26:07.347347 18878 raft_consensus.cc:1261] T e0e69b88ad014363b97d1f650df693a8 P c75831edb3b44dd59c00ef151adca49d [term 1 FOLLOWER]: Refusing update from remote peer 9be399440ded4e2ebe6b6e0bf486cbb1: Log matching property violated. Preceding OpId in replica: term: 1 index: 128. Preceding OpId from leader: term: 1 index: 129. (index mismatch)
I20240502 21:26:07.348018 20429 consensus_queue.cc:1035] T e0e69b88ad014363b97d1f650df693a8 P 9be399440ded4e2ebe6b6e0bf486cbb1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 129, Last known committed idx: 128, Time since last communication: 0.000s
I20240502 21:26:07.348785 20429 consensus_queue.cc:1035] T e0e69b88ad014363b97d1f650df693a8 P 9be399440ded4e2ebe6b6e0bf486cbb1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 129, Last known committed idx: 128, Time since last communication: 0.000s
I20240502 21:26:07.352342 20662 raft_consensus.cc:2932] T e0e69b88ad014363b97d1f650df693a8 P 9be399440ded4e2ebe6b6e0bf486cbb1 [term 1 LEADER]: Committing config change with OpId 1.129: config changed from index 94 to 129, 90e329b457c24caf93a9f880345eb288 (127.15.255.135) changed from NON_VOTER to VOTER. New config: { opid_index: 129 OBSOLETE_local: false peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } attrs { replace: true } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } } }
I20240502 21:26:07.356587 20689 tablet_copy_client.cc:783] T 84b235fd89634168938c0dddb534462a P 90e329b457c24caf93a9f880345eb288: tablet copy: Starting download of 0 data blocks...
I20240502 21:26:07.356984 20689 tablet_copy_client.cc:647] T 84b235fd89634168938c0dddb534462a P 90e329b457c24caf93a9f880345eb288: tablet copy: Starting download of 1 WAL segments...
I20240502 21:26:07.358053 18738 raft_consensus.cc:2932] T e0e69b88ad014363b97d1f650df693a8 P 501e3c0d44fe4d70af9ff926d9622417 [term 1 FOLLOWER]: Committing config change with OpId 1.129: config changed from index 94 to 129, 90e329b457c24caf93a9f880345eb288 (127.15.255.135) changed from NON_VOTER to VOTER. New config: { opid_index: 129 OBSOLETE_local: false peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } attrs { replace: true } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } } }
I20240502 21:26:07.363188 20689 tablet_copy_client.cc:523] T 84b235fd89634168938c0dddb534462a P 90e329b457c24caf93a9f880345eb288: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:26:07.364923 18321 catalog_manager.cc:5484] T e0e69b88ad014363b97d1f650df693a8 P 501e3c0d44fe4d70af9ff926d9622417 reported cstate change: config changed from index 94 to 129, 90e329b457c24caf93a9f880345eb288 (127.15.255.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" committed_config { opid_index: 129 OBSOLETE_local: false peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } attrs { replace: true } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } } }
I20240502 21:26:07.366382 18878 raft_consensus.cc:2932] T e0e69b88ad014363b97d1f650df693a8 P c75831edb3b44dd59c00ef151adca49d [term 1 FOLLOWER]: Committing config change with OpId 1.129: config changed from index 94 to 129, 90e329b457c24caf93a9f880345eb288 (127.15.255.135) changed from NON_VOTER to VOTER. New config: { opid_index: 129 OBSOLETE_local: false peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } attrs { replace: true } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } } }
I20240502 21:26:07.367177 20167 raft_consensus.cc:2932] T e0e69b88ad014363b97d1f650df693a8 P 90e329b457c24caf93a9f880345eb288 [term 1 FOLLOWER]: Committing config change with OpId 1.129: config changed from index 94 to 129, 90e329b457c24caf93a9f880345eb288 (127.15.255.135) changed from NON_VOTER to VOTER. New config: { opid_index: 129 OBSOLETE_local: false peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } attrs { replace: true } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } } }
I20240502 21:26:07.371687 18608 raft_consensus.cc:2932] T e0e69b88ad014363b97d1f650df693a8 P 82f50f4b66f3450aa9f58a0b317f5cea [term 1 FOLLOWER]: Committing config change with OpId 1.129: config changed from index 94 to 129, 90e329b457c24caf93a9f880345eb288 (127.15.255.135) changed from NON_VOTER to VOTER. New config: { opid_index: 129 OBSOLETE_local: false peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } attrs { replace: true } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } } }
I20240502 21:26:07.379806 19143 raft_consensus.cc:1261] T e0e69b88ad014363b97d1f650df693a8 P 2877108756304e05bd60ba1186b870da [term 1 FOLLOWER]: Refusing update from remote peer 9be399440ded4e2ebe6b6e0bf486cbb1: Log matching property violated. Preceding OpId in replica: term: 1 index: 127. Preceding OpId from leader: term: 1 index: 129. (index mismatch)
I20240502 21:26:07.380769 20429 consensus_queue.cc:1035] T e0e69b88ad014363b97d1f650df693a8 P 9be399440ded4e2ebe6b6e0bf486cbb1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 129, Last known committed idx: 126, Time since last communication: 0.000s
I20240502 21:26:07.385557 19143 raft_consensus.cc:2932] T e0e69b88ad014363b97d1f650df693a8 P 2877108756304e05bd60ba1186b870da [term 1 FOLLOWER]: Committing config change with OpId 1.129: config changed from index 94 to 129, 90e329b457c24caf93a9f880345eb288 (127.15.255.135) changed from NON_VOTER to VOTER. New config: { opid_index: 129 OBSOLETE_local: false peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } attrs { replace: true } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } } }
I20240502 21:26:07.396247 20644 consensus_queue.cc:1035] T 37e9dcc6191c42d3a5c1503dedc5511e P c75831edb3b44dd59c00ef151adca49d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 117, Last known committed idx: 112, Time since last communication: 0.000s
W20240502 21:26:07.396891 18810 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.133:34217, user_credentials={real_user=slave}} blocked reactor thread for 72209.1us
I20240502 21:26:07.423146 20689 tablet_bootstrap.cc:492] T 84b235fd89634168938c0dddb534462a P 90e329b457c24caf93a9f880345eb288: Bootstrap starting.
W20240502 21:26:07.466992 18812 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.134:43635, user_credentials={real_user=slave}} blocked reactor thread for 150216us
I20240502 21:26:07.475482 20806 ts_tablet_manager.cc:902] T 37e9dcc6191c42d3a5c1503dedc5511e P faadabfb099b4abf90a466395be585b5: Initiating tablet copy from peer c75831edb3b44dd59c00ef151adca49d (127.15.255.132:43607)
I20240502 21:26:07.489068 20605 raft_consensus.cc:1050] T ebf7371ff0ef4de99ced8fec0468af28 P c75831edb3b44dd59c00ef151adca49d: attempting to promote NON_VOTER 90e329b457c24caf93a9f880345eb288 to VOTER
I20240502 21:26:07.490942 20605 consensus_queue.cc:237] T ebf7371ff0ef4de99ced8fec0468af28 P c75831edb3b44dd59c00ef151adca49d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 113, Committed index: 113, Last appended: 1.113, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 114 OBSOLETE_local: false peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } attrs { replace: true } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } }
I20240502 21:26:07.493800 20169 raft_consensus.cc:1261] T ebf7371ff0ef4de99ced8fec0468af28 P 90e329b457c24caf93a9f880345eb288 [term 1 LEARNER]: Refusing update from remote peer c75831edb3b44dd59c00ef151adca49d: Log matching property violated. Preceding OpId in replica: term: 1 index: 113. Preceding OpId from leader: term: 1 index: 114. (index mismatch)
I20240502 21:26:07.494365 20304 consensus_queue.cc:1035] T ebf7371ff0ef4de99ced8fec0468af28 P c75831edb3b44dd59c00ef151adca49d [LEADER]: Connected to new peer: Peer: permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 114, Last known committed idx: 113, Time since last communication: 0.000s
I20240502 21:26:07.495879 18472 raft_consensus.cc:1261] T ebf7371ff0ef4de99ced8fec0468af28 P faadabfb099b4abf90a466395be585b5 [term 1 FOLLOWER]: Refusing update from remote peer c75831edb3b44dd59c00ef151adca49d: Log matching property violated. Preceding OpId in replica: term: 1 index: 113. Preceding OpId from leader: term: 1 index: 114. (index mismatch)
I20240502 21:26:07.495954 19010 raft_consensus.cc:1261] T ebf7371ff0ef4de99ced8fec0468af28 P 9be399440ded4e2ebe6b6e0bf486cbb1 [term 1 FOLLOWER]: Refusing update from remote peer c75831edb3b44dd59c00ef151adca49d: Log matching property violated. Preceding OpId in replica: term: 1 index: 113. Preceding OpId from leader: term: 1 index: 114. (index mismatch)
I20240502 21:26:07.496672 20605 consensus_queue.cc:1035] T ebf7371ff0ef4de99ced8fec0468af28 P c75831edb3b44dd59c00ef151adca49d [LEADER]: Connected to new peer: Peer: permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 114, Last known committed idx: 113, Time since last communication: 0.000s
I20240502 21:26:07.497185 20605 consensus_queue.cc:1035] T ebf7371ff0ef4de99ced8fec0468af28 P c75831edb3b44dd59c00ef151adca49d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 114, Last known committed idx: 113, Time since last communication: 0.000s
I20240502 21:26:07.497475 18738 raft_consensus.cc:1261] T ebf7371ff0ef4de99ced8fec0468af28 P 501e3c0d44fe4d70af9ff926d9622417 [term 1 FOLLOWER]: Refusing update from remote peer c75831edb3b44dd59c00ef151adca49d: Log matching property violated. Preceding OpId in replica: term: 1 index: 113. Preceding OpId from leader: term: 1 index: 114. (index mismatch)
I20240502 21:26:07.498946 20605 consensus_queue.cc:1035] T ebf7371ff0ef4de99ced8fec0468af28 P c75831edb3b44dd59c00ef151adca49d [LEADER]: Connected to new peer: Peer: permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 114, Last known committed idx: 113, Time since last communication: 0.001s
I20240502 21:26:07.498617 19143 raft_consensus.cc:2932] T 37e9dcc6191c42d3a5c1503dedc5511e P 2877108756304e05bd60ba1186b870da [term 1 FOLLOWER]: Committing config change with OpId 1.117: config changed from index -1 to 117, NON_VOTER faadabfb099b4abf90a466395be585b5 (127.15.255.129) added. New config: { opid_index: 117 OBSOLETE_local: false peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } attrs { replace: true } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: NON_VOTER last_known_addr { host: "127.15.255.129" port: 32983 } attrs { promote: true } } }
I20240502 21:26:07.501749 20304 raft_consensus.cc:2932] T ebf7371ff0ef4de99ced8fec0468af28 P c75831edb3b44dd59c00ef151adca49d [term 1 LEADER]: Committing config change with OpId 1.114: config changed from index 96 to 114, 90e329b457c24caf93a9f880345eb288 (127.15.255.135) changed from NON_VOTER to VOTER. New config: { opid_index: 114 OBSOLETE_local: false peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } attrs { replace: true } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } } }
I20240502 21:26:07.502377 18737 raft_consensus.cc:2932] T ebf7371ff0ef4de99ced8fec0468af28 P 501e3c0d44fe4d70af9ff926d9622417 [term 1 FOLLOWER]: Committing config change with OpId 1.114: config changed from index 96 to 114, 90e329b457c24caf93a9f880345eb288 (127.15.255.135) changed from NON_VOTER to VOTER. New config: { opid_index: 114 OBSOLETE_local: false peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } attrs { replace: true } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } } }
I20240502 21:26:07.505610 19010 raft_consensus.cc:2932] T ebf7371ff0ef4de99ced8fec0468af28 P 9be399440ded4e2ebe6b6e0bf486cbb1 [term 1 FOLLOWER]: Committing config change with OpId 1.114: config changed from index 96 to 114, 90e329b457c24caf93a9f880345eb288 (127.15.255.135) changed from NON_VOTER to VOTER. New config: { opid_index: 114 OBSOLETE_local: false peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } attrs { replace: true } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } } }
I20240502 21:26:07.509629 20806 tablet_copy_client.cc:308] T 37e9dcc6191c42d3a5c1503dedc5511e P faadabfb099b4abf90a466395be585b5: tablet copy: Beginning tablet copy session from remote peer at address 127.15.255.132:43607
I20240502 21:26:07.509693 18321 catalog_manager.cc:5484] T ebf7371ff0ef4de99ced8fec0468af28 P 501e3c0d44fe4d70af9ff926d9622417 reported cstate change: config changed from index 96 to 114, 90e329b457c24caf93a9f880345eb288 (127.15.255.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c75831edb3b44dd59c00ef151adca49d" committed_config { opid_index: 114 OBSOLETE_local: false peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } attrs { replace: true } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } } }
I20240502 21:26:07.513702 20169 raft_consensus.cc:2932] T ebf7371ff0ef4de99ced8fec0468af28 P 90e329b457c24caf93a9f880345eb288 [term 1 FOLLOWER]: Committing config change with OpId 1.114: config changed from index 96 to 114, 90e329b457c24caf93a9f880345eb288 (127.15.255.135) changed from NON_VOTER to VOTER. New config: { opid_index: 114 OBSOLETE_local: false peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } attrs { replace: true } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } } }
I20240502 21:26:07.516531 18472 raft_consensus.cc:2932] T ebf7371ff0ef4de99ced8fec0468af28 P faadabfb099b4abf90a466395be585b5 [term 1 FOLLOWER]: Committing config change with OpId 1.114: config changed from index 96 to 114, 90e329b457c24caf93a9f880345eb288 (127.15.255.135) changed from NON_VOTER to VOTER. New config: { opid_index: 114 OBSOLETE_local: false peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } attrs { replace: true } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } } }
W20240502 21:26:07.524570 18407 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.130:45139, user_credentials={real_user=slave}} blocked reactor thread for 49526.7us
W20240502 21:26:07.529515 18405 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.133:34217, user_credentials={real_user=slave}} blocked reactor thread for 57769.1us
W20240502 21:26:07.556859 18545 consensus_peers.cc:485] T a7a9256f2b784abcb780b190e2ff3983 P 82f50f4b66f3450aa9f58a0b317f5cea -> Peer 90e329b457c24caf93a9f880345eb288 (127.15.255.135:43395): Couldn't send request to peer 90e329b457c24caf93a9f880345eb288. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a7a9256f2b784abcb780b190e2ff3983. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:26:07.560561 18878 raft_consensus.cc:1261] T a7a9256f2b784abcb780b190e2ff3983 P c75831edb3b44dd59c00ef151adca49d [term 1 FOLLOWER]: Refusing update from remote peer 82f50f4b66f3450aa9f58a0b317f5cea: Log matching property violated. Preceding OpId in replica: term: 1 index: 117. Preceding OpId from leader: term: 1 index: 121. (index mismatch)
I20240502 21:26:07.608899 20817 ts_tablet_manager.cc:902] T a7a9256f2b784abcb780b190e2ff3983 P 90e329b457c24caf93a9f880345eb288: Initiating tablet copy from peer 82f50f4b66f3450aa9f58a0b317f5cea (127.15.255.130:45139)
I20240502 21:26:07.790246 20605 consensus_queue.cc:1035] T 37e9dcc6191c42d3a5c1503dedc5511e P c75831edb3b44dd59c00ef151adca49d [LEADER]: Connected to new peer: Peer: permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 117, Last known committed idx: 113, Time since last communication: 0.000s
I20240502 21:26:07.794607 19007 consensus_queue.cc:237] T e0e69b88ad014363b97d1f650df693a8 P 9be399440ded4e2ebe6b6e0bf486cbb1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 129, Committed index: 129, Last appended: 1.129, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 130 OBSOLETE_local: false peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } }
I20240502 21:26:07.797597 20169 raft_consensus.cc:1261] T e0e69b88ad014363b97d1f650df693a8 P 90e329b457c24caf93a9f880345eb288 [term 1 FOLLOWER]: Refusing update from remote peer 9be399440ded4e2ebe6b6e0bf486cbb1: Log matching property violated. Preceding OpId in replica: term: 1 index: 129. Preceding OpId from leader: term: 1 index: 130. (index mismatch)
I20240502 21:26:07.798913 20662 consensus_queue.cc:1035] T e0e69b88ad014363b97d1f650df693a8 P 9be399440ded4e2ebe6b6e0bf486cbb1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 130, Last known committed idx: 129, Time since last communication: 0.000s
I20240502 21:26:07.815335 20817 tablet_copy_client.cc:308] T a7a9256f2b784abcb780b190e2ff3983 P 90e329b457c24caf93a9f880345eb288: tablet copy: Beginning tablet copy session from remote peer at address 127.15.255.130:45139
W20240502 21:26:07.827805 18404 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.131:42501, user_credentials={real_user=slave}} blocked reactor thread for 51674.2us
W20240502 21:26:07.829979 18407 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.134:43635, user_credentials={real_user=slave}} blocked reactor thread for 49629.7us
I20240502 21:26:07.844484 18632 tablet_copy_service.cc:140] P 82f50f4b66f3450aa9f58a0b317f5cea: Received BeginTabletCopySession request for tablet a7a9256f2b784abcb780b190e2ff3983 from peer 90e329b457c24caf93a9f880345eb288 ({username='slave'} at 127.15.255.135:57495)
I20240502 21:26:07.844775 18632 tablet_copy_service.cc:161] P 82f50f4b66f3450aa9f58a0b317f5cea: Beginning new tablet copy session on tablet a7a9256f2b784abcb780b190e2ff3983 from peer 90e329b457c24caf93a9f880345eb288 at {username='slave'} at 127.15.255.135:57495: session id = 90e329b457c24caf93a9f880345eb288-a7a9256f2b784abcb780b190e2ff3983
I20240502 21:26:07.850874 18632 tablet_copy_source_session.cc:205] T a7a9256f2b784abcb780b190e2ff3983 P 82f50f4b66f3450aa9f58a0b317f5cea: Tablet Copy: opened 0 blocks and 1 log segments
W20240502 21:26:07.852061 18405 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.133:34217, user_credentials={real_user=slave}} blocked reactor thread for 79161.3us
I20240502 21:26:07.852782 20817 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7a9256f2b784abcb780b190e2ff3983. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:07.860488 20817 tablet_copy_client.cc:783] T a7a9256f2b784abcb780b190e2ff3983 P 90e329b457c24caf93a9f880345eb288: tablet copy: Starting download of 0 data blocks...
I20240502 21:26:07.870096 20817 tablet_copy_client.cc:647] T a7a9256f2b784abcb780b190e2ff3983 P 90e329b457c24caf93a9f880345eb288: tablet copy: Starting download of 1 WAL segments...
I20240502 21:26:07.875931 20737 tablet_bootstrap.cc:492] T 6f0107b0690c4c64b0a4f2472a1c8322 P 90e329b457c24caf93a9f880345eb288: Bootstrap replayed 1/1 log segments. Stats: ops{read=105 overwritten=0 applied=104 ignored=0} inserts{seen=776 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240502 21:26:07.878123 20817 tablet_copy_client.cc:523] T a7a9256f2b784abcb780b190e2ff3983 P 90e329b457c24caf93a9f880345eb288: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:26:07.884577 20817 tablet_bootstrap.cc:492] T a7a9256f2b784abcb780b190e2ff3983 P 90e329b457c24caf93a9f880345eb288: Bootstrap starting.
I20240502 21:26:07.896780 20737 tablet_bootstrap.cc:492] T 6f0107b0690c4c64b0a4f2472a1c8322 P 90e329b457c24caf93a9f880345eb288: Bootstrap complete.
I20240502 21:26:07.897107 20737 ts_tablet_manager.cc:1367] T 6f0107b0690c4c64b0a4f2472a1c8322 P 90e329b457c24caf93a9f880345eb288: Time spent bootstrapping tablet: real 1.160s user 0.168s sys 0.007s
I20240502 21:26:07.897956 20737 raft_consensus.cc:348] T 6f0107b0690c4c64b0a4f2472a1c8322 P 90e329b457c24caf93a9f880345eb288 [term 1 LEARNER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 98 OBSOLETE_local: false peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } attrs { replace: true } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: NON_VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: true } }
I20240502 21:26:07.899051 20737 raft_consensus.cc:729] T 6f0107b0690c4c64b0a4f2472a1c8322 P 90e329b457c24caf93a9f880345eb288 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 90e329b457c24caf93a9f880345eb288, State: Initialized, Role: LEARNER
I20240502 21:26:07.899374 20737 consensus_queue.cc:260] T 6f0107b0690c4c64b0a4f2472a1c8322 P 90e329b457c24caf93a9f880345eb288 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 104, Last appended: 1.105, Last appended by leader: 105, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 98 OBSOLETE_local: false peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } attrs { replace: true } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: NON_VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: true } }
I20240502 21:26:07.902997 20737 ts_tablet_manager.cc:1398] T 6f0107b0690c4c64b0a4f2472a1c8322 P 90e329b457c24caf93a9f880345eb288: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240502 21:26:07.907060 18496 tablet_copy_service.cc:342] P faadabfb099b4abf90a466395be585b5: Request end of tablet copy session 90e329b457c24caf93a9f880345eb288-6f0107b0690c4c64b0a4f2472a1c8322 received from {username='slave'} at 127.15.255.135:49783
I20240502 21:26:07.907248 18496 tablet_copy_service.cc:433] P faadabfb099b4abf90a466395be585b5: ending tablet copy session 90e329b457c24caf93a9f880345eb288-6f0107b0690c4c64b0a4f2472a1c8322 on tablet 6f0107b0690c4c64b0a4f2472a1c8322 with peer 90e329b457c24caf93a9f880345eb288
I20240502 21:26:07.912595 18898 tablet_copy_service.cc:140] P c75831edb3b44dd59c00ef151adca49d: Received BeginTabletCopySession request for tablet 37e9dcc6191c42d3a5c1503dedc5511e from peer faadabfb099b4abf90a466395be585b5 ({username='slave'} at 127.15.255.129:50861)
I20240502 21:26:07.912827 18898 tablet_copy_service.cc:161] P c75831edb3b44dd59c00ef151adca49d: Beginning new tablet copy session on tablet 37e9dcc6191c42d3a5c1503dedc5511e from peer faadabfb099b4abf90a466395be585b5 at {username='slave'} at 127.15.255.129:50861: session id = faadabfb099b4abf90a466395be585b5-37e9dcc6191c42d3a5c1503dedc5511e
I20240502 21:26:07.916054 18898 tablet_copy_source_session.cc:205] T 37e9dcc6191c42d3a5c1503dedc5511e P c75831edb3b44dd59c00ef151adca49d: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:26:07.930657 19143 raft_consensus.cc:1261] T ebf7371ff0ef4de99ced8fec0468af28 P 2877108756304e05bd60ba1186b870da [term 1 FOLLOWER]: Refusing update from remote peer c75831edb3b44dd59c00ef151adca49d: Log matching property violated. Preceding OpId in replica: term: 1 index: 113. Preceding OpId from leader: term: 1 index: 114. (index mismatch)
I20240502 21:26:07.930537 20641 consensus_queue.cc:1035] T a7a9256f2b784abcb780b190e2ff3983 P 82f50f4b66f3450aa9f58a0b317f5cea [LEADER]: Connected to new peer: Peer: permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 121, Last known committed idx: 117, Time since last communication: 0.000s
I20240502 21:26:07.965626 18611 raft_consensus.cc:2932] T 37e9dcc6191c42d3a5c1503dedc5511e P 82f50f4b66f3450aa9f58a0b317f5cea [term 1 FOLLOWER]: Committing config change with OpId 1.117: config changed from index -1 to 117, NON_VOTER faadabfb099b4abf90a466395be585b5 (127.15.255.129) added. New config: { opid_index: 117 OBSOLETE_local: false peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } attrs { replace: true } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: NON_VOTER last_known_addr { host: "127.15.255.129" port: 32983 } attrs { promote: true } } }
I20240502 21:26:07.974506 20689 tablet_bootstrap.cc:492] T 84b235fd89634168938c0dddb534462a P 90e329b457c24caf93a9f880345eb288: Bootstrap replayed 1/1 log segments. Stats: ops{read=122 overwritten=0 applied=122 ignored=0} inserts{seen=753 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:26:07.974984 20689 tablet_bootstrap.cc:492] T 84b235fd89634168938c0dddb534462a P 90e329b457c24caf93a9f880345eb288: Bootstrap complete.
I20240502 21:26:07.975289 20689 ts_tablet_manager.cc:1367] T 84b235fd89634168938c0dddb534462a P 90e329b457c24caf93a9f880345eb288: Time spent bootstrapping tablet: real 0.552s user 0.183s sys 0.000s
I20240502 21:26:07.976091 20689 raft_consensus.cc:348] T 84b235fd89634168938c0dddb534462a P 90e329b457c24caf93a9f880345eb288 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 118 OBSOLETE_local: false peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } attrs { replace: true } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: NON_VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: true } }
I20240502 21:26:07.976387 20689 raft_consensus.cc:729] T 84b235fd89634168938c0dddb534462a P 90e329b457c24caf93a9f880345eb288 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 90e329b457c24caf93a9f880345eb288, State: Initialized, Role: LEARNER
I20240502 21:26:07.976665 20689 consensus_queue.cc:260] T 84b235fd89634168938c0dddb534462a P 90e329b457c24caf93a9f880345eb288 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 122, Last appended: 1.122, Last appended by leader: 122, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 118 OBSOLETE_local: false peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } attrs { replace: true } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: NON_VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: true } }
I20240502 21:26:07.977664 20689 ts_tablet_manager.cc:1398] T 84b235fd89634168938c0dddb534462a P 90e329b457c24caf93a9f880345eb288: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240502 21:26:07.978610 18496 tablet_copy_service.cc:342] P faadabfb099b4abf90a466395be585b5: Request end of tablet copy session 90e329b457c24caf93a9f880345eb288-84b235fd89634168938c0dddb534462a received from {username='slave'} at 127.15.255.135:49783
I20240502 21:26:07.978744 18496 tablet_copy_service.cc:433] P faadabfb099b4abf90a466395be585b5: ending tablet copy session 90e329b457c24caf93a9f880345eb288-84b235fd89634168938c0dddb534462a on tablet 84b235fd89634168938c0dddb534462a with peer 90e329b457c24caf93a9f880345eb288
W20240502 21:26:08.006855 19077 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.133:34217, user_credentials={real_user=slave}} blocked reactor thread for 143095us
I20240502 21:26:08.010593 20806 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37e9dcc6191c42d3a5c1503dedc5511e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:08.018137 20806 tablet_copy_client.cc:783] T 37e9dcc6191c42d3a5c1503dedc5511e P faadabfb099b4abf90a466395be585b5: tablet copy: Starting download of 0 data blocks...
I20240502 21:26:08.018464 20806 tablet_copy_client.cc:647] T 37e9dcc6191c42d3a5c1503dedc5511e P faadabfb099b4abf90a466395be585b5: tablet copy: Starting download of 1 WAL segments...
I20240502 21:26:08.021514 19143 raft_consensus.cc:1261] T e0e69b88ad014363b97d1f650df693a8 P 2877108756304e05bd60ba1186b870da [term 1 FOLLOWER]: Refusing update from remote peer 9be399440ded4e2ebe6b6e0bf486cbb1: Log matching property violated. Preceding OpId in replica: term: 1 index: 129. Preceding OpId from leader: term: 1 index: 130. (index mismatch)
I20240502 21:26:08.022135 20844 consensus_queue.cc:1035] T e0e69b88ad014363b97d1f650df693a8 P 9be399440ded4e2ebe6b6e0bf486cbb1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 130, Last known committed idx: 129, Time since last communication: 0.000s
I20240502 21:26:08.025099 20605 consensus_queue.cc:1035] T ebf7371ff0ef4de99ced8fec0468af28 P c75831edb3b44dd59c00ef151adca49d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 114, Last known committed idx: 113, Time since last communication: 0.000s
I20240502 21:26:08.025034 20844 raft_consensus.cc:2932] T e0e69b88ad014363b97d1f650df693a8 P 9be399440ded4e2ebe6b6e0bf486cbb1 [term 1 LEADER]: Committing config change with OpId 1.130: config changed from index 129 to 130, VOTER 82f50f4b66f3450aa9f58a0b317f5cea (127.15.255.130) evicted. New config: { opid_index: 130 OBSOLETE_local: false peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } } }
I20240502 21:26:08.030814 19140 raft_consensus.cc:2932] T e0e69b88ad014363b97d1f650df693a8 P 2877108756304e05bd60ba1186b870da [term 1 FOLLOWER]: Committing config change with OpId 1.130: config changed from index 129 to 130, VOTER 82f50f4b66f3450aa9f58a0b317f5cea (127.15.255.130) evicted. New config: { opid_index: 130 OBSOLETE_local: false peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } } }
I20240502 21:26:08.031620 19139 raft_consensus.cc:2932] T ebf7371ff0ef4de99ced8fec0468af28 P 2877108756304e05bd60ba1186b870da [term 1 FOLLOWER]: Committing config change with OpId 1.114: config changed from index 96 to 114, 90e329b457c24caf93a9f880345eb288 (127.15.255.135) changed from NON_VOTER to VOTER. New config: { opid_index: 114 OBSOLETE_local: false peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } attrs { replace: true } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } } }
I20240502 21:26:08.038663 18321 catalog_manager.cc:5484] T e0e69b88ad014363b97d1f650df693a8 P 2877108756304e05bd60ba1186b870da reported cstate change: config changed from index 129 to 130, VOTER 82f50f4b66f3450aa9f58a0b317f5cea (127.15.255.130) evicted. New cstate: current_term: 1 leader_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" committed_config { opid_index: 130 OBSOLETE_local: false peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } } }
I20240502 21:26:08.063040 18313 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet e0e69b88ad014363b97d1f650df693a8 with cas_config_opid_index 129: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:26:08.065618 20169 raft_consensus.cc:1203] T 6f0107b0690c4c64b0a4f2472a1c8322 P 90e329b457c24caf93a9f880345eb288 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.97->[1.98-1.116] Dedup: 1.105->[1.106-1.116]
I20240502 21:26:08.114905 20806 tablet_copy_client.cc:523] T 37e9dcc6191c42d3a5c1503dedc5511e P faadabfb099b4abf90a466395be585b5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:26:08.119462 20806 tablet_bootstrap.cc:492] T 37e9dcc6191c42d3a5c1503dedc5511e P faadabfb099b4abf90a466395be585b5: Bootstrap starting.
W20240502 21:26:08.122365 18680 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.134:43635, user_credentials={real_user=slave}} blocked reactor thread for 100637us
I20240502 21:26:08.146778 18738 raft_consensus.cc:1261] T e0e69b88ad014363b97d1f650df693a8 P 501e3c0d44fe4d70af9ff926d9622417 [term 1 FOLLOWER]: Refusing update from remote peer 9be399440ded4e2ebe6b6e0bf486cbb1: Log matching property violated. Preceding OpId in replica: term: 1 index: 129. Preceding OpId from leader: term: 1 index: 130. (index mismatch)
I20240502 21:26:08.147291 20372 consensus_queue.cc:1035] T e0e69b88ad014363b97d1f650df693a8 P 9be399440ded4e2ebe6b6e0bf486cbb1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 130, Last known committed idx: 129, Time since last communication: 0.000s
I20240502 21:26:08.151510 18738 raft_consensus.cc:2932] T e0e69b88ad014363b97d1f650df693a8 P 501e3c0d44fe4d70af9ff926d9622417 [term 1 FOLLOWER]: Committing config change with OpId 1.130: config changed from index 129 to 130, VOTER 82f50f4b66f3450aa9f58a0b317f5cea (127.15.255.130) evicted. New config: { opid_index: 130 OBSOLETE_local: false peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } } }
I20240502 21:26:08.164033 20167 raft_consensus.cc:2932] T e0e69b88ad014363b97d1f650df693a8 P 90e329b457c24caf93a9f880345eb288 [term 1 FOLLOWER]: Committing config change with OpId 1.130: config changed from index 129 to 130, VOTER 82f50f4b66f3450aa9f58a0b317f5cea (127.15.255.130) evicted. New config: { opid_index: 130 OBSOLETE_local: false peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } } }
I20240502 21:26:08.238407 20712 raft_consensus.cc:1050] T 6f0107b0690c4c64b0a4f2472a1c8322 P faadabfb099b4abf90a466395be585b5: attempting to promote NON_VOTER 90e329b457c24caf93a9f880345eb288 to VOTER
I20240502 21:26:08.246163 20712 consensus_queue.cc:237] T 6f0107b0690c4c64b0a4f2472a1c8322 P faadabfb099b4abf90a466395be585b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 119, Committed index: 119, Last appended: 1.120, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 121 OBSOLETE_local: false peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } attrs { replace: true } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } }
I20240502 21:26:08.249598 20767 raft_consensus.cc:1013] T 6f0107b0690c4c64b0a4f2472a1c8322 P faadabfb099b4abf90a466395be585b5 [term 1 LEADER]: attempt to promote peer 90e329b457c24caf93a9f880345eb288: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240502 21:26:08.250824 20167 raft_consensus.cc:1261] T 6f0107b0690c4c64b0a4f2472a1c8322 P 90e329b457c24caf93a9f880345eb288 [term 1 LEARNER]: Refusing update from remote peer faadabfb099b4abf90a466395be585b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 120. Preceding OpId from leader: term: 1 index: 121. (index mismatch)
I20240502 21:26:08.252271 20767 consensus_queue.cc:1035] T 6f0107b0690c4c64b0a4f2472a1c8322 P faadabfb099b4abf90a466395be585b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 121, Last known committed idx: 119, Time since last communication: 0.000s
I20240502 21:26:08.255055 19010 raft_consensus.cc:1261] T 6f0107b0690c4c64b0a4f2472a1c8322 P 9be399440ded4e2ebe6b6e0bf486cbb1 [term 1 FOLLOWER]: Refusing update from remote peer faadabfb099b4abf90a466395be585b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 120. Preceding OpId from leader: term: 1 index: 121. (index mismatch)
I20240502 21:26:08.256878 20713 consensus_queue.cc:1035] T 6f0107b0690c4c64b0a4f2472a1c8322 P faadabfb099b4abf90a466395be585b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 121, Last known committed idx: 119, Time since last communication: 0.000s
I20240502 21:26:08.264343 19140 raft_consensus.cc:1261] T 6f0107b0690c4c64b0a4f2472a1c8322 P 2877108756304e05bd60ba1186b870da [term 1 FOLLOWER]: Refusing update from remote peer faadabfb099b4abf90a466395be585b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 120. Preceding OpId from leader: term: 1 index: 121. (index mismatch)
I20240502 21:26:08.265946 20446 consensus_queue.cc:1035] T 6f0107b0690c4c64b0a4f2472a1c8322 P faadabfb099b4abf90a466395be585b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 121, Last known committed idx: 119, Time since last communication: 0.000s
I20240502 21:26:08.271512 20817 tablet_bootstrap.cc:492] T a7a9256f2b784abcb780b190e2ff3983 P 90e329b457c24caf93a9f880345eb288: Bootstrap replayed 1/1 log segments. Stats: ops{read=126 overwritten=0 applied=126 ignored=0} inserts{seen=798 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:26:08.271978 20817 tablet_bootstrap.cc:492] T a7a9256f2b784abcb780b190e2ff3983 P 90e329b457c24caf93a9f880345eb288: Bootstrap complete.
I20240502 21:26:08.272300 20817 ts_tablet_manager.cc:1367] T a7a9256f2b784abcb780b190e2ff3983 P 90e329b457c24caf93a9f880345eb288: Time spent bootstrapping tablet: real 0.388s user 0.177s sys 0.010s
I20240502 21:26:08.273116 20817 raft_consensus.cc:348] T a7a9256f2b784abcb780b190e2ff3983 P 90e329b457c24caf93a9f880345eb288 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 121 OBSOLETE_local: false peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } attrs { replace: true } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: NON_VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: true } }
I20240502 21:26:08.273470 20817 raft_consensus.cc:729] T a7a9256f2b784abcb780b190e2ff3983 P 90e329b457c24caf93a9f880345eb288 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 90e329b457c24caf93a9f880345eb288, State: Initialized, Role: LEARNER
I20240502 21:26:08.274256 20871 raft_consensus.cc:2932] T 6f0107b0690c4c64b0a4f2472a1c8322 P faadabfb099b4abf90a466395be585b5 [term 1 LEADER]: Committing config change with OpId 1.121: config changed from index 98 to 121, 90e329b457c24caf93a9f880345eb288 (127.15.255.135) changed from NON_VOTER to VOTER. New config: { opid_index: 121 OBSOLETE_local: false peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } attrs { replace: true } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } } }
I20240502 21:26:08.276856 19143 raft_consensus.cc:2932] T 6f0107b0690c4c64b0a4f2472a1c8322 P 2877108756304e05bd60ba1186b870da [term 1 FOLLOWER]: Committing config change with OpId 1.121: config changed from index 98 to 121, 90e329b457c24caf93a9f880345eb288 (127.15.255.135) changed from NON_VOTER to VOTER. New config: { opid_index: 121 OBSOLETE_local: false peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } attrs { replace: true } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } } }
I20240502 21:26:08.279716 20169 raft_consensus.cc:2932] T 6f0107b0690c4c64b0a4f2472a1c8322 P 90e329b457c24caf93a9f880345eb288 [term 1 FOLLOWER]: Committing config change with OpId 1.121: config changed from index 98 to 121, 90e329b457c24caf93a9f880345eb288 (127.15.255.135) changed from NON_VOTER to VOTER. New config: { opid_index: 121 OBSOLETE_local: false peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } attrs { replace: true } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } } }
I20240502 21:26:08.286263 20817 consensus_queue.cc:260] T a7a9256f2b784abcb780b190e2ff3983 P 90e329b457c24caf93a9f880345eb288 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 126, Last appended: 1.126, Last appended by leader: 126, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 121 OBSOLETE_local: false peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } attrs { replace: true } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: NON_VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: true } }
I20240502 21:26:08.295353 18875 raft_consensus.cc:1261] T e0e69b88ad014363b97d1f650df693a8 P c75831edb3b44dd59c00ef151adca49d [term 1 FOLLOWER]: Refusing update from remote peer 9be399440ded4e2ebe6b6e0bf486cbb1: Log matching property violated. Preceding OpId in replica: term: 1 index: 129. Preceding OpId from leader: term: 1 index: 130. (index mismatch)
W20240502 21:26:08.295801 18812 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.130:45139, user_credentials={real_user=slave}} blocked reactor thread for 68742.4us
I20240502 21:26:08.301666 18877 raft_consensus.cc:2932] T a7a9256f2b784abcb780b190e2ff3983 P c75831edb3b44dd59c00ef151adca49d [term 1 FOLLOWER]: Committing config change with OpId 1.121: config changed from index -1 to 121, NON_VOTER 90e329b457c24caf93a9f880345eb288 (127.15.255.135) added. New config: { opid_index: 121 OBSOLETE_local: false peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } attrs { replace: true } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: NON_VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: true } } }
I20240502 21:26:08.304863 18323 catalog_manager.cc:5484] T 6f0107b0690c4c64b0a4f2472a1c8322 P faadabfb099b4abf90a466395be585b5 reported cstate change: config changed from index 98 to 121, 90e329b457c24caf93a9f880345eb288 (127.15.255.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "faadabfb099b4abf90a466395be585b5" committed_config { opid_index: 121 OBSOLETE_local: false peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:26:08.309021 20844 consensus_queue.cc:1035] T e0e69b88ad014363b97d1f650df693a8 P 9be399440ded4e2ebe6b6e0bf486cbb1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 130, Last known committed idx: 129, Time since last communication: 0.000s
I20240502 21:26:08.313388 19010 raft_consensus.cc:2932] T 6f0107b0690c4c64b0a4f2472a1c8322 P 9be399440ded4e2ebe6b6e0bf486cbb1 [term 1 FOLLOWER]: Committing config change with OpId 1.121: config changed from index 98 to 121, 90e329b457c24caf93a9f880345eb288 (127.15.255.135) changed from NON_VOTER to VOTER. New config: { opid_index: 121 OBSOLETE_local: false peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } attrs { replace: true } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } } }
I20240502 21:26:08.335129 18875 raft_consensus.cc:2932] T e0e69b88ad014363b97d1f650df693a8 P c75831edb3b44dd59c00ef151adca49d [term 1 FOLLOWER]: Committing config change with OpId 1.130: config changed from index 129 to 130, VOTER 82f50f4b66f3450aa9f58a0b317f5cea (127.15.255.130) evicted. New config: { opid_index: 130 OBSOLETE_local: false peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } } }
I20240502 21:26:08.353214 18878 raft_consensus.cc:1261] T 6f0107b0690c4c64b0a4f2472a1c8322 P c75831edb3b44dd59c00ef151adca49d [term 1 FOLLOWER]: Refusing update from remote peer faadabfb099b4abf90a466395be585b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 117. Preceding OpId from leader: term: 1 index: 121. (index mismatch)
I20240502 21:26:08.357120 20817 ts_tablet_manager.cc:1398] T a7a9256f2b784abcb780b190e2ff3983 P 90e329b457c24caf93a9f880345eb288: Time spent starting tablet: real 0.085s user 0.002s sys 0.000s
I20240502 21:26:08.358588 18632 tablet_copy_service.cc:342] P 82f50f4b66f3450aa9f58a0b317f5cea: Request end of tablet copy session 90e329b457c24caf93a9f880345eb288-a7a9256f2b784abcb780b190e2ff3983 received from {username='slave'} at 127.15.255.135:57495
I20240502 21:26:08.358781 18632 tablet_copy_service.cc:433] P 82f50f4b66f3450aa9f58a0b317f5cea: ending tablet copy session 90e329b457c24caf93a9f880345eb288-a7a9256f2b784abcb780b190e2ff3983 on tablet a7a9256f2b784abcb780b190e2ff3983 with peer 90e329b457c24caf93a9f880345eb288
I20240502 21:26:08.359795 20871 consensus_queue.cc:1035] T 6f0107b0690c4c64b0a4f2472a1c8322 P faadabfb099b4abf90a466395be585b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 121, Last known committed idx: 116, Time since last communication: 0.000s
I20240502 21:26:08.366245 18590 tablet_service.cc:1514] Processing DeleteTablet for tablet e0e69b88ad014363b97d1f650df693a8 with delete_type TABLET_DATA_TOMBSTONED (TS 82f50f4b66f3450aa9f58a0b317f5cea not found in new config with opid_index 130) from {username='slave'} at 127.0.0.1:33750
I20240502 21:26:08.369843 18606 raft_consensus.cc:1261] T 6f0107b0690c4c64b0a4f2472a1c8322 P 82f50f4b66f3450aa9f58a0b317f5cea [term 1 FOLLOWER]: Refusing update from remote peer faadabfb099b4abf90a466395be585b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 119. Preceding OpId from leader: term: 1 index: 121. (index mismatch)
I20240502 21:26:08.371677 20875 tablet_replica.cc:330] T e0e69b88ad014363b97d1f650df693a8 P 82f50f4b66f3450aa9f58a0b317f5cea: stopping tablet replica
I20240502 21:26:08.375470 20875 raft_consensus.cc:2229] T e0e69b88ad014363b97d1f650df693a8 P 82f50f4b66f3450aa9f58a0b317f5cea [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:26:08.384904 20875 raft_consensus.cc:2258] T e0e69b88ad014363b97d1f650df693a8 P 82f50f4b66f3450aa9f58a0b317f5cea [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:08.386628 20856 consensus_queue.cc:1035] T 6f0107b0690c4c64b0a4f2472a1c8322 P faadabfb099b4abf90a466395be585b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 121, Last known committed idx: 119, Time since last communication: 0.000s
I20240502 21:26:08.415275 20875 ts_tablet_manager.cc:1875] T e0e69b88ad014363b97d1f650df693a8 P 82f50f4b66f3450aa9f58a0b317f5cea: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:26:08.416370 18610 raft_consensus.cc:2932] T 6f0107b0690c4c64b0a4f2472a1c8322 P 82f50f4b66f3450aa9f58a0b317f5cea [term 1 FOLLOWER]: Committing config change with OpId 1.121: config changed from index 98 to 121, 90e329b457c24caf93a9f880345eb288 (127.15.255.135) changed from NON_VOTER to VOTER. New config: { opid_index: 121 OBSOLETE_local: false peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } attrs { replace: true } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } } }
I20240502 21:26:08.444877 20169 raft_consensus.cc:1203] T 84b235fd89634168938c0dddb534462a P 90e329b457c24caf93a9f880345eb288 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.117->[1.118-1.129] Dedup: 1.122->[1.123-1.129]
I20240502 21:26:08.459818 20875 ts_tablet_manager.cc:1888] T e0e69b88ad014363b97d1f650df693a8 P 82f50f4b66f3450aa9f58a0b317f5cea: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.129
I20240502 21:26:08.460466 20875 log.cc:1196] T e0e69b88ad014363b97d1f650df693a8 P 82f50f4b66f3450aa9f58a0b317f5cea: Deleting WAL directory at /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-1/wal/wals/e0e69b88ad014363b97d1f650df693a8
I20240502 21:26:08.506702 18315 catalog_manager.cc:4830] TS 82f50f4b66f3450aa9f58a0b317f5cea (127.15.255.130:45139): tablet e0e69b88ad014363b97d1f650df693a8 (table rebalance_test_table_1 [id=76deee1e4597471ebedfc30f6d635eaa]) successfully deleted
I20240502 21:26:08.518201 18875 consensus_queue.cc:237] T ebf7371ff0ef4de99ced8fec0468af28 P c75831edb3b44dd59c00ef151adca49d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 116, Committed index: 116, Last appended: 1.116, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 117 OBSOLETE_local: false peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } }
I20240502 21:26:08.523105 18738 raft_consensus.cc:1261] T ebf7371ff0ef4de99ced8fec0468af28 P 501e3c0d44fe4d70af9ff926d9622417 [term 1 FOLLOWER]: Refusing update from remote peer c75831edb3b44dd59c00ef151adca49d: Log matching property violated. Preceding OpId in replica: term: 1 index: 116. Preceding OpId from leader: term: 1 index: 117. (index mismatch)
I20240502 21:26:08.523620 19005 raft_consensus.cc:1261] T ebf7371ff0ef4de99ced8fec0468af28 P 9be399440ded4e2ebe6b6e0bf486cbb1 [term 1 FOLLOWER]: Refusing update from remote peer c75831edb3b44dd59c00ef151adca49d: Log matching property violated. Preceding OpId in replica: term: 1 index: 116. Preceding OpId from leader: term: 1 index: 117. (index mismatch)
I20240502 21:26:08.524487 20606 consensus_queue.cc:1035] T ebf7371ff0ef4de99ced8fec0468af28 P c75831edb3b44dd59c00ef151adca49d [LEADER]: Connected to new peer: Peer: permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 117, Last known committed idx: 116, Time since last communication: 0.000s
I20240502 21:26:08.528267 20800 consensus_queue.cc:1035] T ebf7371ff0ef4de99ced8fec0468af28 P c75831edb3b44dd59c00ef151adca49d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 117, Last known committed idx: 116, Time since last communication: 0.002s
I20240502 21:26:08.590621 20167 raft_consensus.cc:1203] T a7a9256f2b784abcb780b190e2ff3983 P 90e329b457c24caf93a9f880345eb288 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.120->[1.121-1.129] Dedup: 1.126->[1.127-1.129]
I20240502 21:26:08.597695 20636 raft_consensus.cc:1050] T 84b235fd89634168938c0dddb534462a P faadabfb099b4abf90a466395be585b5: attempting to promote NON_VOTER 90e329b457c24caf93a9f880345eb288 to VOTER
I20240502 21:26:08.598629 20636 consensus_queue.cc:237] T 84b235fd89634168938c0dddb534462a P faadabfb099b4abf90a466395be585b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 130, Committed index: 130, Last appended: 1.130, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 131 OBSOLETE_local: false peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } attrs { replace: true } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } }
I20240502 21:26:08.601742 20168 raft_consensus.cc:1261] T 84b235fd89634168938c0dddb534462a P 90e329b457c24caf93a9f880345eb288 [term 1 LEARNER]: Refusing update from remote peer faadabfb099b4abf90a466395be585b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 130. Preceding OpId from leader: term: 1 index: 131. (index mismatch)
I20240502 21:26:08.602406 20767 consensus_queue.cc:1035] T 84b235fd89634168938c0dddb534462a P faadabfb099b4abf90a466395be585b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 131, Last known committed idx: 129, Time since last communication: 0.000s
I20240502 21:26:08.610337 19005 raft_consensus.cc:1261] T 84b235fd89634168938c0dddb534462a P 9be399440ded4e2ebe6b6e0bf486cbb1 [term 1 FOLLOWER]: Refusing update from remote peer faadabfb099b4abf90a466395be585b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 130. Preceding OpId from leader: term: 1 index: 131. (index mismatch)
I20240502 21:26:08.610839 20767 consensus_queue.cc:1035] T 84b235fd89634168938c0dddb534462a P faadabfb099b4abf90a466395be585b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 131, Last known committed idx: 130, Time since last communication: 0.000s
I20240502 21:26:08.611505 19141 raft_consensus.cc:1261] T 84b235fd89634168938c0dddb534462a P 2877108756304e05bd60ba1186b870da [term 1 FOLLOWER]: Refusing update from remote peer faadabfb099b4abf90a466395be585b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 130. Preceding OpId from leader: term: 1 index: 131. (index mismatch)
I20240502 21:26:08.612023 20856 consensus_queue.cc:1035] T 84b235fd89634168938c0dddb534462a P faadabfb099b4abf90a466395be585b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 131, Last known committed idx: 130, Time since last communication: 0.000s
I20240502 21:26:08.617823 18610 raft_consensus.cc:1261] T 84b235fd89634168938c0dddb534462a P 82f50f4b66f3450aa9f58a0b317f5cea [term 1 FOLLOWER]: Refusing update from remote peer faadabfb099b4abf90a466395be585b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 130. Preceding OpId from leader: term: 1 index: 131. (index mismatch)
I20240502 21:26:08.621351 20763 raft_consensus.cc:1050] T a7a9256f2b784abcb780b190e2ff3983 P 82f50f4b66f3450aa9f58a0b317f5cea: attempting to promote NON_VOTER 90e329b457c24caf93a9f880345eb288 to VOTER
I20240502 21:26:08.622401 20763 consensus_queue.cc:237] T a7a9256f2b784abcb780b190e2ff3983 P 82f50f4b66f3450aa9f58a0b317f5cea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 129, Committed index: 129, Last appended: 1.129, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 130 OBSOLETE_local: false peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } attrs { replace: true } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } }
I20240502 21:26:08.624253 20636 raft_consensus.cc:2932] T 84b235fd89634168938c0dddb534462a P faadabfb099b4abf90a466395be585b5 [term 1 LEADER]: Committing config change with OpId 1.131: config changed from index 118 to 131, 90e329b457c24caf93a9f880345eb288 (127.15.255.135) changed from NON_VOTER to VOTER. New config: { opid_index: 131 OBSOLETE_local: false peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } attrs { replace: true } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } } }
I20240502 21:26:08.625540 20168 raft_consensus.cc:1261] T a7a9256f2b784abcb780b190e2ff3983 P 90e329b457c24caf93a9f880345eb288 [term 1 LEARNER]: Refusing update from remote peer 82f50f4b66f3450aa9f58a0b317f5cea: Log matching property violated. Preceding OpId in replica: term: 1 index: 129. Preceding OpId from leader: term: 1 index: 130. (index mismatch)
I20240502 21:26:08.626096 20640 consensus_queue.cc:1035] T a7a9256f2b784abcb780b190e2ff3983 P 82f50f4b66f3450aa9f58a0b317f5cea [LEADER]: Connected to new peer: Peer: permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 130, Last known committed idx: 129, Time since last communication: 0.000s
I20240502 21:26:08.627178 20856 consensus_queue.cc:1035] T 84b235fd89634168938c0dddb534462a P faadabfb099b4abf90a466395be585b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 131, Last known committed idx: 129, Time since last communication: 0.000s
I20240502 21:26:08.629839 18738 raft_consensus.cc:1261] T a7a9256f2b784abcb780b190e2ff3983 P 501e3c0d44fe4d70af9ff926d9622417 [term 1 FOLLOWER]: Refusing update from remote peer 82f50f4b66f3450aa9f58a0b317f5cea: Log matching property violated. Preceding OpId in replica: term: 1 index: 129. Preceding OpId from leader: term: 1 index: 130. (index mismatch)
I20240502 21:26:08.629433 19141 raft_consensus.cc:2932] T 84b235fd89634168938c0dddb534462a P 2877108756304e05bd60ba1186b870da [term 1 FOLLOWER]: Committing config change with OpId 1.131: config changed from index 118 to 131, 90e329b457c24caf93a9f880345eb288 (127.15.255.135) changed from NON_VOTER to VOTER. New config: { opid_index: 131 OBSOLETE_local: false peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } attrs { replace: true } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } } }
I20240502 21:26:08.630354 20763 consensus_queue.cc:1035] T a7a9256f2b784abcb780b190e2ff3983 P 82f50f4b66f3450aa9f58a0b317f5cea [LEADER]: Connected to new peer: Peer: permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 130, Last known committed idx: 129, Time since last communication: 0.000s
I20240502 21:26:08.634594 18474 raft_consensus.cc:1261] T a7a9256f2b784abcb780b190e2ff3983 P faadabfb099b4abf90a466395be585b5 [term 1 FOLLOWER]: Refusing update from remote peer 82f50f4b66f3450aa9f58a0b317f5cea: Log matching property violated. Preceding OpId in replica: term: 1 index: 129. Preceding OpId from leader: term: 1 index: 130. (index mismatch)
I20240502 21:26:08.634820 18606 raft_consensus.cc:2932] T 84b235fd89634168938c0dddb534462a P 82f50f4b66f3450aa9f58a0b317f5cea [term 1 FOLLOWER]: Committing config change with OpId 1.131: config changed from index 118 to 131, 90e329b457c24caf93a9f880345eb288 (127.15.255.135) changed from NON_VOTER to VOTER. New config: { opid_index: 131 OBSOLETE_local: false peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } attrs { replace: true } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } } }
I20240502 21:26:08.638146 19006 raft_consensus.cc:2932] T 84b235fd89634168938c0dddb534462a P 9be399440ded4e2ebe6b6e0bf486cbb1 [term 1 FOLLOWER]: Committing config change with OpId 1.131: config changed from index 118 to 131, 90e329b457c24caf93a9f880345eb288 (127.15.255.135) changed from NON_VOTER to VOTER. New config: { opid_index: 131 OBSOLETE_local: false peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } attrs { replace: true } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } } }
I20240502 21:26:08.639223 19005 raft_consensus.cc:1261] T a7a9256f2b784abcb780b190e2ff3983 P 9be399440ded4e2ebe6b6e0bf486cbb1 [term 1 FOLLOWER]: Refusing update from remote peer 82f50f4b66f3450aa9f58a0b317f5cea: Log matching property violated. Preceding OpId in replica: term: 1 index: 129. Preceding OpId from leader: term: 1 index: 130. (index mismatch)
I20240502 21:26:08.640635 18324 catalog_manager.cc:5484] T 84b235fd89634168938c0dddb534462a P faadabfb099b4abf90a466395be585b5 reported cstate change: config changed from index 118 to 131, 90e329b457c24caf93a9f880345eb288 (127.15.255.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "faadabfb099b4abf90a466395be585b5" committed_config { opid_index: 131 OBSOLETE_local: false peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:26:08.642040 20168 raft_consensus.cc:2932] T 84b235fd89634168938c0dddb534462a P 90e329b457c24caf93a9f880345eb288 [term 1 FOLLOWER]: Committing config change with OpId 1.131: config changed from index 118 to 131, 90e329b457c24caf93a9f880345eb288 (127.15.255.135) changed from NON_VOTER to VOTER. New config: { opid_index: 131 OBSOLETE_local: false peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } attrs { replace: true } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } } }
I20240502 21:26:08.645459 20763 consensus_queue.cc:1035] T a7a9256f2b784abcb780b190e2ff3983 P 82f50f4b66f3450aa9f58a0b317f5cea [LEADER]: Connected to new peer: Peer: permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 130, Last known committed idx: 129, Time since last communication: 0.000s
I20240502 21:26:08.645921 20763 consensus_queue.cc:1035] T a7a9256f2b784abcb780b190e2ff3983 P 82f50f4b66f3450aa9f58a0b317f5cea [LEADER]: Connected to new peer: Peer: permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 130, Last known committed idx: 129, Time since last communication: 0.000s
I20240502 21:26:08.671061 18471 consensus_queue.cc:237] T 84b235fd89634168938c0dddb534462a P faadabfb099b4abf90a466395be585b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 132, Committed index: 132, Last appended: 1.133, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 134 OBSOLETE_local: false peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } }
I20240502 21:26:08.672576 20169 raft_consensus.cc:1261] T ebf7371ff0ef4de99ced8fec0468af28 P 90e329b457c24caf93a9f880345eb288 [term 1 FOLLOWER]: Refusing update from remote peer c75831edb3b44dd59c00ef151adca49d: Log matching property violated. Preceding OpId in replica: term: 1 index: 116. Preceding OpId from leader: term: 1 index: 117. (index mismatch)
I20240502 21:26:08.674193 20169 raft_consensus.cc:1261] T 84b235fd89634168938c0dddb534462a P 90e329b457c24caf93a9f880345eb288 [term 1 FOLLOWER]: Refusing update from remote peer faadabfb099b4abf90a466395be585b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 133. Preceding OpId from leader: term: 1 index: 134. (index mismatch)
I20240502 21:26:08.674770 20713 consensus_queue.cc:1035] T 84b235fd89634168938c0dddb534462a P faadabfb099b4abf90a466395be585b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 134, Last known committed idx: 132, Time since last communication: 0.000s
I20240502 21:26:08.677727 19005 raft_consensus.cc:1261] T 84b235fd89634168938c0dddb534462a P 9be399440ded4e2ebe6b6e0bf486cbb1 [term 1 FOLLOWER]: Refusing update from remote peer faadabfb099b4abf90a466395be585b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 133. Preceding OpId from leader: term: 1 index: 134. (index mismatch)
I20240502 21:26:08.678447 20713 consensus_queue.cc:1035] T 84b235fd89634168938c0dddb534462a P faadabfb099b4abf90a466395be585b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 134, Last known committed idx: 132, Time since last communication: 0.000s
I20240502 21:26:08.682045 20713 raft_consensus.cc:2932] T 84b235fd89634168938c0dddb534462a P faadabfb099b4abf90a466395be585b5 [term 1 LEADER]: Committing config change with OpId 1.134: config changed from index 131 to 134, VOTER c75831edb3b44dd59c00ef151adca49d (127.15.255.132) evicted. New config: { opid_index: 134 OBSOLETE_local: false peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } } }
I20240502 21:26:08.684113 20800 consensus_queue.cc:1035] T ebf7371ff0ef4de99ced8fec0468af28 P c75831edb3b44dd59c00ef151adca49d [LEADER]: Connected to new peer: Peer: permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 117, Last known committed idx: 116, Time since last communication: 0.007s
I20240502 21:26:08.686774 19005 raft_consensus.cc:2932] T 84b235fd89634168938c0dddb534462a P 9be399440ded4e2ebe6b6e0bf486cbb1 [term 1 FOLLOWER]: Committing config change with OpId 1.134: config changed from index 131 to 134, VOTER c75831edb3b44dd59c00ef151adca49d (127.15.255.132) evicted. New config: { opid_index: 134 OBSOLETE_local: false peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } } }
I20240502 21:26:08.697336 18323 catalog_manager.cc:5484] T 84b235fd89634168938c0dddb534462a P 9be399440ded4e2ebe6b6e0bf486cbb1 reported cstate change: config changed from index 131 to 134, VOTER c75831edb3b44dd59c00ef151adca49d (127.15.255.132) evicted. New cstate: current_term: 1 leader_uuid: "faadabfb099b4abf90a466395be585b5" committed_config { opid_index: 134 OBSOLETE_local: false peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } } }
I20240502 21:26:08.703989 18313 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 84b235fd89634168938c0dddb534462a with cas_config_opid_index 131: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:26:08.705125 20169 raft_consensus.cc:2932] T 84b235fd89634168938c0dddb534462a P 90e329b457c24caf93a9f880345eb288 [term 1 FOLLOWER]: Committing config change with OpId 1.134: config changed from index 131 to 134, VOTER c75831edb3b44dd59c00ef151adca49d (127.15.255.132) evicted. New config: { opid_index: 134 OBSOLETE_local: false peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } } }
I20240502 21:26:08.708081 20304 raft_consensus.cc:2932] T ebf7371ff0ef4de99ced8fec0468af28 P c75831edb3b44dd59c00ef151adca49d [term 1 LEADER]: Committing config change with OpId 1.117: config changed from index 114 to 117, VOTER faadabfb099b4abf90a466395be585b5 (127.15.255.129) evicted. New config: { opid_index: 117 OBSOLETE_local: false peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } } }
I20240502 21:26:08.714287 18743 raft_consensus.cc:2932] T ebf7371ff0ef4de99ced8fec0468af28 P 501e3c0d44fe4d70af9ff926d9622417 [term 1 FOLLOWER]: Committing config change with OpId 1.117: config changed from index 114 to 117, VOTER faadabfb099b4abf90a466395be585b5 (127.15.255.129) evicted. New config: { opid_index: 117 OBSOLETE_local: false peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } } }
I20240502 21:26:08.722414 18606 raft_consensus.cc:1261] T 84b235fd89634168938c0dddb534462a P 82f50f4b66f3450aa9f58a0b317f5cea [term 1 FOLLOWER]: Refusing update from remote peer faadabfb099b4abf90a466395be585b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 133. Preceding OpId from leader: term: 1 index: 134. (index mismatch)
I20240502 21:26:08.724781 20763 raft_consensus.cc:2932] T a7a9256f2b784abcb780b190e2ff3983 P 82f50f4b66f3450aa9f58a0b317f5cea [term 1 LEADER]: Committing config change with OpId 1.130: config changed from index 121 to 130, 90e329b457c24caf93a9f880345eb288 (127.15.255.135) changed from NON_VOTER to VOTER. New config: { opid_index: 130 OBSOLETE_local: false peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } attrs { replace: true } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } } }
I20240502 21:26:08.729398 20169 raft_consensus.cc:2932] T a7a9256f2b784abcb780b190e2ff3983 P 90e329b457c24caf93a9f880345eb288 [term 1 FOLLOWER]: Committing config change with OpId 1.130: config changed from index 121 to 130, 90e329b457c24caf93a9f880345eb288 (127.15.255.135) changed from NON_VOTER to VOTER. New config: { opid_index: 130 OBSOLETE_local: false peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } attrs { replace: true } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } } }
I20240502 21:26:08.731001 18743 raft_consensus.cc:2932] T a7a9256f2b784abcb780b190e2ff3983 P 501e3c0d44fe4d70af9ff926d9622417 [term 1 FOLLOWER]: Committing config change with OpId 1.130: config changed from index 121 to 130, 90e329b457c24caf93a9f880345eb288 (127.15.255.135) changed from NON_VOTER to VOTER. New config: { opid_index: 130 OBSOLETE_local: false peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } attrs { replace: true } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } } }
I20240502 21:26:08.731771 18323 catalog_manager.cc:5484] T ebf7371ff0ef4de99ced8fec0468af28 P 501e3c0d44fe4d70af9ff926d9622417 reported cstate change: config changed from index 114 to 117, VOTER faadabfb099b4abf90a466395be585b5 (127.15.255.129) evicted. New cstate: current_term: 1 leader_uuid: "c75831edb3b44dd59c00ef151adca49d" committed_config { opid_index: 117 OBSOLETE_local: false peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } } }
I20240502 21:26:08.734216 18319 catalog_manager.cc:5484] T a7a9256f2b784abcb780b190e2ff3983 P 82f50f4b66f3450aa9f58a0b317f5cea reported cstate change: config changed from index 121 to 130, 90e329b457c24caf93a9f880345eb288 (127.15.255.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" committed_config { opid_index: 130 OBSOLETE_local: false peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:26:08.736783 18873 raft_consensus.cc:1261] T 84b235fd89634168938c0dddb534462a P c75831edb3b44dd59c00ef151adca49d [term 1 FOLLOWER]: Refusing update from remote peer faadabfb099b4abf90a466395be585b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 123. Preceding OpId from leader: term: 1 index: 131. (index mismatch)
I20240502 21:26:08.736882 19005 raft_consensus.cc:2932] T a7a9256f2b784abcb780b190e2ff3983 P 9be399440ded4e2ebe6b6e0bf486cbb1 [term 1 FOLLOWER]: Committing config change with OpId 1.130: config changed from index 121 to 130, 90e329b457c24caf93a9f880345eb288 (127.15.255.135) changed from NON_VOTER to VOTER. New config: { opid_index: 130 OBSOLETE_local: false peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } attrs { replace: true } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } } }
I20240502 21:26:08.740442 18314 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet ebf7371ff0ef4de99ced8fec0468af28 with cas_config_opid_index 114: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:26:08.745075 19140 raft_consensus.cc:1261] T ebf7371ff0ef4de99ced8fec0468af28 P 2877108756304e05bd60ba1186b870da [term 1 FOLLOWER]: Refusing update from remote peer c75831edb3b44dd59c00ef151adca49d: Log matching property violated. Preceding OpId in replica: term: 1 index: 116. Preceding OpId from leader: term: 1 index: 117. (index mismatch)
I20240502 21:26:08.757766 19139 raft_consensus.cc:1261] T 84b235fd89634168938c0dddb534462a P 2877108756304e05bd60ba1186b870da [term 1 FOLLOWER]: Refusing update from remote peer faadabfb099b4abf90a466395be585b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 132. Preceding OpId from leader: term: 1 index: 134. (index mismatch)
W20240502 21:26:08.758781 18811 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.135:43395, user_credentials={real_user=slave}} blocked reactor thread for 50903.3us
I20240502 21:26:08.789710 18876 raft_consensus.cc:2932] T 6f0107b0690c4c64b0a4f2472a1c8322 P c75831edb3b44dd59c00ef151adca49d [term 1 FOLLOWER]: Committing config change with OpId 1.121: config changed from index 98 to 121, 90e329b457c24caf93a9f880345eb288 (127.15.255.135) changed from NON_VOTER to VOTER. New config: { opid_index: 121 OBSOLETE_local: false peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } attrs { replace: true } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } } }
I20240502 21:26:08.792860 20871 consensus_queue.cc:1035] T 84b235fd89634168938c0dddb534462a P faadabfb099b4abf90a466395be585b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 134, Last known committed idx: 132, Time since last communication: 0.057s
W20240502 21:26:08.799242 18812 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.134:43635, user_credentials={real_user=slave}} blocked reactor thread for 60133us
I20240502 21:26:08.799377 20169 raft_consensus.cc:2932] T ebf7371ff0ef4de99ced8fec0468af28 P 90e329b457c24caf93a9f880345eb288 [term 1 FOLLOWER]: Committing config change with OpId 1.117: config changed from index 114 to 117, VOTER faadabfb099b4abf90a466395be585b5 (127.15.255.129) evicted. New config: { opid_index: 117 OBSOLETE_local: false peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } } }
I20240502 21:26:08.800206 20800 consensus_queue.cc:1035] T ebf7371ff0ef4de99ced8fec0468af28 P c75831edb3b44dd59c00ef151adca49d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 117, Last known committed idx: 116, Time since last communication: 0.000s
I20240502 21:26:08.801199 18857 tablet_service.cc:1514] Processing DeleteTablet for tablet 84b235fd89634168938c0dddb534462a with delete_type TABLET_DATA_TOMBSTONED (TS c75831edb3b44dd59c00ef151adca49d not found in new config with opid_index 134) from {username='slave'} at 127.0.0.1:38486
I20240502 21:26:08.798935 19010 raft_consensus.cc:2932] T ebf7371ff0ef4de99ced8fec0468af28 P 9be399440ded4e2ebe6b6e0bf486cbb1 [term 1 FOLLOWER]: Committing config change with OpId 1.117: config changed from index 114 to 117, VOTER faadabfb099b4abf90a466395be585b5 (127.15.255.129) evicted. New config: { opid_index: 117 OBSOLETE_local: false peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } } }
I20240502 21:26:08.804704 20712 consensus_queue.cc:1035] T 84b235fd89634168938c0dddb534462a P faadabfb099b4abf90a466395be585b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 134, Last known committed idx: 132, Time since last communication: 0.012s
I20240502 21:26:08.807737 18875 raft_consensus.cc:1261] T a7a9256f2b784abcb780b190e2ff3983 P c75831edb3b44dd59c00ef151adca49d [term 1 FOLLOWER]: Refusing update from remote peer 82f50f4b66f3450aa9f58a0b317f5cea: Log matching property violated. Preceding OpId in replica: term: 1 index: 126. Preceding OpId from leader: term: 1 index: 130. (index mismatch)
I20240502 21:26:08.814296 19141 raft_consensus.cc:2932] T 84b235fd89634168938c0dddb534462a P 2877108756304e05bd60ba1186b870da [term 1 FOLLOWER]: Committing config change with OpId 1.134: config changed from index 131 to 134, VOTER c75831edb3b44dd59c00ef151adca49d (127.15.255.132) evicted. New config: { opid_index: 134 OBSOLETE_local: false peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } } }
I20240502 21:26:08.833966 19143 raft_consensus.cc:2932] T ebf7371ff0ef4de99ced8fec0468af28 P 2877108756304e05bd60ba1186b870da [term 1 FOLLOWER]: Committing config change with OpId 1.117: config changed from index 114 to 117, VOTER faadabfb099b4abf90a466395be585b5 (127.15.255.129) evicted. New config: { opid_index: 117 OBSOLETE_local: false peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } } }
I20240502 21:26:08.839748 20763 consensus_queue.cc:1035] T a7a9256f2b784abcb780b190e2ff3983 P 82f50f4b66f3450aa9f58a0b317f5cea [LEADER]: Connected to new peer: Peer: permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 130, Last known committed idx: 126, Time since last communication: 0.000s
I20240502 21:26:08.841213 18608 raft_consensus.cc:2932] T 84b235fd89634168938c0dddb534462a P 82f50f4b66f3450aa9f58a0b317f5cea [term 1 FOLLOWER]: Committing config change with OpId 1.134: config changed from index 131 to 134, VOTER c75831edb3b44dd59c00ef151adca49d (127.15.255.132) evicted. New config: { opid_index: 134 OBSOLETE_local: false peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } } }
I20240502 21:26:08.796710 18471 raft_consensus.cc:2932] T a7a9256f2b784abcb780b190e2ff3983 P faadabfb099b4abf90a466395be585b5 [term 1 FOLLOWER]: Committing config change with OpId 1.130: config changed from index 121 to 130, 90e329b457c24caf93a9f880345eb288 (127.15.255.135) changed from NON_VOTER to VOTER. New config: { opid_index: 130 OBSOLETE_local: false peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } attrs { replace: true } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } } }
I20240502 21:26:08.884955 18451 tablet_service.cc:1514] Processing DeleteTablet for tablet ebf7371ff0ef4de99ced8fec0468af28 with delete_type TABLET_DATA_TOMBSTONED (TS faadabfb099b4abf90a466395be585b5 not found in new config with opid_index 117) from {username='slave'} at 127.0.0.1:39378
I20240502 21:26:08.921265 20897 tablet_replica.cc:330] T ebf7371ff0ef4de99ced8fec0468af28 P faadabfb099b4abf90a466395be585b5: stopping tablet replica
I20240502 21:26:08.923242 20897 raft_consensus.cc:2229] T ebf7371ff0ef4de99ced8fec0468af28 P faadabfb099b4abf90a466395be585b5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:26:08.929997 20897 raft_consensus.cc:2258] T ebf7371ff0ef4de99ced8fec0468af28 P faadabfb099b4abf90a466395be585b5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:08.936038 20897 ts_tablet_manager.cc:1875] T ebf7371ff0ef4de99ced8fec0468af28 P faadabfb099b4abf90a466395be585b5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:26:08.961361 20897 ts_tablet_manager.cc:1888] T ebf7371ff0ef4de99ced8fec0468af28 P faadabfb099b4abf90a466395be585b5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.116
I20240502 21:26:08.962193 20897 log.cc:1196] T ebf7371ff0ef4de99ced8fec0468af28 P faadabfb099b4abf90a466395be585b5: Deleting WAL directory at /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-0/wal/wals/ebf7371ff0ef4de99ced8fec0468af28
I20240502 21:26:08.963346 18313 catalog_manager.cc:4830] TS faadabfb099b4abf90a466395be585b5 (127.15.255.129:32983): tablet ebf7371ff0ef4de99ced8fec0468af28 (table rebalance_test_table_3 [id=5af23c390f054d07b5a7985c2611ca30]) successfully deleted
I20240502 21:26:08.994828 18876 raft_consensus.cc:2932] T a7a9256f2b784abcb780b190e2ff3983 P c75831edb3b44dd59c00ef151adca49d [term 1 FOLLOWER]: Committing config change with OpId 1.130: config changed from index 121 to 130, 90e329b457c24caf93a9f880345eb288 (127.15.255.135) changed from NON_VOTER to VOTER. New config: { opid_index: 130 OBSOLETE_local: false peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } attrs { replace: true } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } } }
W20240502 21:26:09.077351 18472 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.695s user 0.005s sys 0.003s
W20240502 21:26:09.077482 18472 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.695s user 0.005s sys 0.003s
I20240502 21:26:09.156375 18470 consensus_queue.cc:237] T 6f0107b0690c4c64b0a4f2472a1c8322 P faadabfb099b4abf90a466395be585b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 132, Committed index: 132, Last appended: 1.133, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 134 OBSOLETE_local: false peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } }
I20240502 21:26:09.159482 20169 raft_consensus.cc:1261] T 6f0107b0690c4c64b0a4f2472a1c8322 P 90e329b457c24caf93a9f880345eb288 [term 1 FOLLOWER]: Refusing update from remote peer faadabfb099b4abf90a466395be585b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 133. Preceding OpId from leader: term: 1 index: 134. (index mismatch)
I20240502 21:26:09.160389 20871 consensus_queue.cc:1035] T 6f0107b0690c4c64b0a4f2472a1c8322 P faadabfb099b4abf90a466395be585b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 134, Last known committed idx: 132, Time since last communication: 0.000s
I20240502 21:26:09.161600 19008 raft_consensus.cc:1261] T 6f0107b0690c4c64b0a4f2472a1c8322 P 9be399440ded4e2ebe6b6e0bf486cbb1 [term 1 FOLLOWER]: Refusing update from remote peer faadabfb099b4abf90a466395be585b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 133. Preceding OpId from leader: term: 1 index: 134. (index mismatch)
I20240502 21:26:09.163134 20871 consensus_queue.cc:1035] T 6f0107b0690c4c64b0a4f2472a1c8322 P faadabfb099b4abf90a466395be585b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 134, Last known committed idx: 132, Time since last communication: 0.000s
I20240502 21:26:09.165763 20712 raft_consensus.cc:2932] T 6f0107b0690c4c64b0a4f2472a1c8322 P faadabfb099b4abf90a466395be585b5 [term 1 LEADER]: Committing config change with OpId 1.134: config changed from index 121 to 134, VOTER 2877108756304e05bd60ba1186b870da (127.15.255.134) evicted. New config: { opid_index: 134 OBSOLETE_local: false peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } } }
I20240502 21:26:09.170534 20169 raft_consensus.cc:2932] T 6f0107b0690c4c64b0a4f2472a1c8322 P 90e329b457c24caf93a9f880345eb288 [term 1 FOLLOWER]: Committing config change with OpId 1.134: config changed from index 121 to 134, VOTER 2877108756304e05bd60ba1186b870da (127.15.255.134) evicted. New config: { opid_index: 134 OBSOLETE_local: false peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } } }
I20240502 21:26:09.177997 18321 catalog_manager.cc:5484] T 6f0107b0690c4c64b0a4f2472a1c8322 P 90e329b457c24caf93a9f880345eb288 reported cstate change: config changed from index 121 to 134, VOTER 2877108756304e05bd60ba1186b870da (127.15.255.134) evicted. New cstate: current_term: 1 leader_uuid: "faadabfb099b4abf90a466395be585b5" committed_config { opid_index: 134 OBSOLETE_local: false peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } } }
I20240502 21:26:09.181290 18608 raft_consensus.cc:1261] T 6f0107b0690c4c64b0a4f2472a1c8322 P 82f50f4b66f3450aa9f58a0b317f5cea [term 1 FOLLOWER]: Refusing update from remote peer faadabfb099b4abf90a466395be585b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 133. Preceding OpId from leader: term: 1 index: 134. (index mismatch)
I20240502 21:26:09.190332 18313 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 6f0107b0690c4c64b0a4f2472a1c8322 with cas_config_opid_index 121: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:26:09.191901 19124 tablet_service.cc:1514] Processing DeleteTablet for tablet 6f0107b0690c4c64b0a4f2472a1c8322 with delete_type TABLET_DATA_TOMBSTONED (TS 2877108756304e05bd60ba1186b870da not found in new config with opid_index 134) from {username='slave'} at 127.0.0.1:57946
I20240502 21:26:09.196763 18877 raft_consensus.cc:1261] T 6f0107b0690c4c64b0a4f2472a1c8322 P c75831edb3b44dd59c00ef151adca49d [term 1 FOLLOWER]: Refusing update from remote peer faadabfb099b4abf90a466395be585b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 132. Preceding OpId from leader: term: 1 index: 134. (index mismatch)
I20240502 21:26:09.198351 19010 raft_consensus.cc:2932] T 6f0107b0690c4c64b0a4f2472a1c8322 P 9be399440ded4e2ebe6b6e0bf486cbb1 [term 1 FOLLOWER]: Committing config change with OpId 1.134: config changed from index 121 to 134, VOTER 2877108756304e05bd60ba1186b870da (127.15.255.134) evicted. New config: { opid_index: 134 OBSOLETE_local: false peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } } }
I20240502 21:26:09.214502 20893 tablet_replica.cc:330] T 84b235fd89634168938c0dddb534462a P c75831edb3b44dd59c00ef151adca49d: stopping tablet replica
I20240502 21:26:09.215049 20893 raft_consensus.cc:2229] T 84b235fd89634168938c0dddb534462a P c75831edb3b44dd59c00ef151adca49d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:26:09.215755 20893 raft_consensus.cc:2258] T 84b235fd89634168938c0dddb534462a P c75831edb3b44dd59c00ef151adca49d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:09.217898 20712 consensus_queue.cc:1035] T 6f0107b0690c4c64b0a4f2472a1c8322 P faadabfb099b4abf90a466395be585b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 134, Last known committed idx: 132, Time since last communication: 0.000s
I20240502 21:26:09.227453 20893 ts_tablet_manager.cc:1875] T 84b235fd89634168938c0dddb534462a P c75831edb3b44dd59c00ef151adca49d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:26:09.241570 18610 raft_consensus.cc:2932] T 6f0107b0690c4c64b0a4f2472a1c8322 P 82f50f4b66f3450aa9f58a0b317f5cea [term 1 FOLLOWER]: Committing config change with OpId 1.134: config changed from index 121 to 134, VOTER 2877108756304e05bd60ba1186b870da (127.15.255.134) evicted. New config: { opid_index: 134 OBSOLETE_local: false peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } } }
I20240502 21:26:09.303316 20893 ts_tablet_manager.cc:1888] T 84b235fd89634168938c0dddb534462a P c75831edb3b44dd59c00ef151adca49d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.128
I20240502 21:26:09.303579 20893 log.cc:1196] T 84b235fd89634168938c0dddb534462a P c75831edb3b44dd59c00ef151adca49d: Deleting WAL directory at /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-3/wal/wals/84b235fd89634168938c0dddb534462a
I20240502 21:26:09.304101 20856 consensus_queue.cc:1035] T 6f0107b0690c4c64b0a4f2472a1c8322 P faadabfb099b4abf90a466395be585b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 134, Last known committed idx: 131, Time since last communication: 0.000s
I20240502 21:26:09.321960 18470 consensus_queue.cc:237] T 693a8666bbe54ae1b774f7f4da5ba67e P faadabfb099b4abf90a466395be585b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 134, Committed index: 134, Last appended: 1.134, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 135 OBSOLETE_local: false peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } attrs { replace: true } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: NON_VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: true } }
W20240502 21:26:09.381013 18406 consensus_peers.cc:485] T 693a8666bbe54ae1b774f7f4da5ba67e P faadabfb099b4abf90a466395be585b5 -> Peer 90e329b457c24caf93a9f880345eb288 (127.15.255.135:43395): Couldn't send request to peer 90e329b457c24caf93a9f880345eb288. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 693a8666bbe54ae1b774f7f4da5ba67e. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:26:09.384544 19005 raft_consensus.cc:1261] T 693a8666bbe54ae1b774f7f4da5ba67e P 9be399440ded4e2ebe6b6e0bf486cbb1 [term 1 FOLLOWER]: Refusing update from remote peer faadabfb099b4abf90a466395be585b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 134. Preceding OpId from leader: term: 1 index: 135. (index mismatch)
I20240502 21:26:09.387995 18737 raft_consensus.cc:1261] T 693a8666bbe54ae1b774f7f4da5ba67e P 501e3c0d44fe4d70af9ff926d9622417 [term 1 FOLLOWER]: Refusing update from remote peer faadabfb099b4abf90a466395be585b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 134. Preceding OpId from leader: term: 1 index: 135. (index mismatch)
I20240502 21:26:09.390393 20712 consensus_queue.cc:1035] T 693a8666bbe54ae1b774f7f4da5ba67e P faadabfb099b4abf90a466395be585b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 135, Last known committed idx: 134, Time since last communication: 0.000s
I20240502 21:26:09.397204 20712 consensus_queue.cc:1035] T 693a8666bbe54ae1b774f7f4da5ba67e P faadabfb099b4abf90a466395be585b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 135, Last known committed idx: 134, Time since last communication: 0.000s
I20240502 21:26:09.400031 20636 raft_consensus.cc:2932] T 693a8666bbe54ae1b774f7f4da5ba67e P faadabfb099b4abf90a466395be585b5 [term 1 LEADER]: Committing config change with OpId 1.135: config changed from index -1 to 135, NON_VOTER 90e329b457c24caf93a9f880345eb288 (127.15.255.135) added. New config: { opid_index: 135 OBSOLETE_local: false peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } attrs { replace: true } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: NON_VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: true } } }
I20240502 21:26:09.407529 19005 raft_consensus.cc:2932] T 693a8666bbe54ae1b774f7f4da5ba67e P 9be399440ded4e2ebe6b6e0bf486cbb1 [term 1 FOLLOWER]: Committing config change with OpId 1.135: config changed from index -1 to 135, NON_VOTER 90e329b457c24caf93a9f880345eb288 (127.15.255.135) added. New config: { opid_index: 135 OBSOLETE_local: false peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } attrs { replace: true } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: NON_VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: true } } }
I20240502 21:26:09.424479 18737 raft_consensus.cc:2932] T 693a8666bbe54ae1b774f7f4da5ba67e P 501e3c0d44fe4d70af9ff926d9622417 [term 1 FOLLOWER]: Committing config change with OpId 1.135: config changed from index -1 to 135, NON_VOTER 90e329b457c24caf93a9f880345eb288 (127.15.255.135) added. New config: { opid_index: 135 OBSOLETE_local: false peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } attrs { replace: true } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: NON_VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: true } } }
I20240502 21:26:09.431699 18320 catalog_manager.cc:5484] T 693a8666bbe54ae1b774f7f4da5ba67e P 501e3c0d44fe4d70af9ff926d9622417 reported cstate change: config changed from index -1 to 135, NON_VOTER 90e329b457c24caf93a9f880345eb288 (127.15.255.135) added. New cstate: current_term: 1 leader_uuid: "faadabfb099b4abf90a466395be585b5" committed_config { opid_index: 135 OBSOLETE_local: false peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } attrs { replace: true } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: NON_VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: true } } }
W20240502 21:26:09.464152 18407 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.134:43635, user_credentials={real_user=slave}} blocked reactor thread for 92979.6us
W20240502 21:26:09.526566 18407 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.130:45139, user_credentials={real_user=slave}} blocked reactor thread for 46550.1us
I20240502 21:26:09.526489 18610 consensus_queue.cc:237] T a7a9256f2b784abcb780b190e2ff3983 P 82f50f4b66f3450aa9f58a0b317f5cea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 131, Committed index: 131, Last appended: 1.131, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 132 OBSOLETE_local: false peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } }
I20240502 21:26:09.530386 20168 raft_consensus.cc:1261] T a7a9256f2b784abcb780b190e2ff3983 P 90e329b457c24caf93a9f880345eb288 [term 1 FOLLOWER]: Refusing update from remote peer 82f50f4b66f3450aa9f58a0b317f5cea: Log matching property violated. Preceding OpId in replica: term: 1 index: 131. Preceding OpId from leader: term: 1 index: 132. (index mismatch)
I20240502 21:26:09.531073 20641 consensus_queue.cc:1035] T a7a9256f2b784abcb780b190e2ff3983 P 82f50f4b66f3450aa9f58a0b317f5cea [LEADER]: Connected to new peer: Peer: permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 132, Last known committed idx: 131, Time since last communication: 0.000s
I20240502 21:26:09.531330 18474 raft_consensus.cc:1261] T a7a9256f2b784abcb780b190e2ff3983 P faadabfb099b4abf90a466395be585b5 [term 1 FOLLOWER]: Refusing update from remote peer 82f50f4b66f3450aa9f58a0b317f5cea: Log matching property violated. Preceding OpId in replica: term: 1 index: 131. Preceding OpId from leader: term: 1 index: 132. (index mismatch)
I20240502 21:26:09.532025 20641 consensus_queue.cc:1035] T a7a9256f2b784abcb780b190e2ff3983 P 82f50f4b66f3450aa9f58a0b317f5cea [LEADER]: Connected to new peer: Peer: permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 132, Last known committed idx: 131, Time since last communication: 0.000s
I20240502 21:26:09.532709 18737 raft_consensus.cc:1261] T a7a9256f2b784abcb780b190e2ff3983 P 501e3c0d44fe4d70af9ff926d9622417 [term 1 FOLLOWER]: Refusing update from remote peer 82f50f4b66f3450aa9f58a0b317f5cea: Log matching property violated. Preceding OpId in replica: term: 1 index: 131. Preceding OpId from leader: term: 1 index: 132. (index mismatch)
I20240502 21:26:09.538117 20641 raft_consensus.cc:2932] T a7a9256f2b784abcb780b190e2ff3983 P 82f50f4b66f3450aa9f58a0b317f5cea [term 1 LEADER]: Committing config change with OpId 1.132: config changed from index 130 to 132, VOTER 9be399440ded4e2ebe6b6e0bf486cbb1 (127.15.255.133) evicted. New config: { opid_index: 132 OBSOLETE_local: false peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } } }
I20240502 21:26:09.543649 20640 consensus_queue.cc:1035] T a7a9256f2b784abcb780b190e2ff3983 P 82f50f4b66f3450aa9f58a0b317f5cea [LEADER]: Connected to new peer: Peer: permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 132, Last known committed idx: 131, Time since last communication: 0.000s
I20240502 21:26:09.545148 18474 raft_consensus.cc:2932] T a7a9256f2b784abcb780b190e2ff3983 P faadabfb099b4abf90a466395be585b5 [term 1 FOLLOWER]: Committing config change with OpId 1.132: config changed from index 130 to 132, VOTER 9be399440ded4e2ebe6b6e0bf486cbb1 (127.15.255.133) evicted. New config: { opid_index: 132 OBSOLETE_local: false peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } } }
I20240502 21:26:09.556936 18315 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet a7a9256f2b784abcb780b190e2ff3983 with cas_config_opid_index 130: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:26:09.559195 18744 raft_consensus.cc:2932] T a7a9256f2b784abcb780b190e2ff3983 P 501e3c0d44fe4d70af9ff926d9622417 [term 1 FOLLOWER]: Committing config change with OpId 1.132: config changed from index 130 to 132, VOTER 9be399440ded4e2ebe6b6e0bf486cbb1 (127.15.255.133) evicted. New config: { opid_index: 132 OBSOLETE_local: false peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } } }
I20240502 21:26:09.561125 20806 tablet_bootstrap.cc:492] T 37e9dcc6191c42d3a5c1503dedc5511e P faadabfb099b4abf90a466395be585b5: Bootstrap replayed 1/1 log segments. Stats: ops{read=125 overwritten=0 applied=125 ignored=0} inserts{seen=810 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:26:09.561563 20806 tablet_bootstrap.cc:492] T 37e9dcc6191c42d3a5c1503dedc5511e P faadabfb099b4abf90a466395be585b5: Bootstrap complete.
I20240502 21:26:09.561864 20806 ts_tablet_manager.cc:1367] T 37e9dcc6191c42d3a5c1503dedc5511e P faadabfb099b4abf90a466395be585b5: Time spent bootstrapping tablet: real 1.442s user 0.169s sys 0.015s
I20240502 21:26:09.562613 20806 raft_consensus.cc:348] T 37e9dcc6191c42d3a5c1503dedc5511e P faadabfb099b4abf90a466395be585b5 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 117 OBSOLETE_local: false peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } attrs { replace: true } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: NON_VOTER last_known_addr { host: "127.15.255.129" port: 32983 } attrs { promote: true } }
I20240502 21:26:09.562898 20806 raft_consensus.cc:729] T 37e9dcc6191c42d3a5c1503dedc5511e P faadabfb099b4abf90a466395be585b5 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: faadabfb099b4abf90a466395be585b5, State: Initialized, Role: LEARNER
I20240502 21:26:09.563099 20806 consensus_queue.cc:260] T 37e9dcc6191c42d3a5c1503dedc5511e P faadabfb099b4abf90a466395be585b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 125, Last appended: 1.125, Last appended by leader: 125, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 117 OBSOLETE_local: false peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } attrs { replace: true } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: NON_VOTER last_known_addr { host: "127.15.255.129" port: 32983 } attrs { promote: true } }
I20240502 21:26:09.563823 20806 ts_tablet_manager.cc:1398] T 37e9dcc6191c42d3a5c1503dedc5511e P faadabfb099b4abf90a466395be585b5: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240502 21:26:09.564275 18606 raft_consensus.cc:1261] T 693a8666bbe54ae1b774f7f4da5ba67e P 82f50f4b66f3450aa9f58a0b317f5cea [term 1 FOLLOWER]: Refusing update from remote peer faadabfb099b4abf90a466395be585b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 134. Preceding OpId from leader: term: 1 index: 135. (index mismatch)
I20240502 21:26:09.566026 18329 catalog_manager.cc:5484] T a7a9256f2b784abcb780b190e2ff3983 P 501e3c0d44fe4d70af9ff926d9622417 reported cstate change: config changed from index 130 to 132, VOTER 9be399440ded4e2ebe6b6e0bf486cbb1 (127.15.255.133) evicted. New cstate: current_term: 1 leader_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" committed_config { opid_index: 132 OBSOLETE_local: false peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } } }
I20240502 21:26:09.573297 18989 tablet_service.cc:1514] Processing DeleteTablet for tablet a7a9256f2b784abcb780b190e2ff3983 with delete_type TABLET_DATA_TOMBSTONED (TS 9be399440ded4e2ebe6b6e0bf486cbb1 not found in new config with opid_index 132) from {username='slave'} at 127.0.0.1:40858
I20240502 21:26:09.576804 18877 raft_consensus.cc:1261] T a7a9256f2b784abcb780b190e2ff3983 P c75831edb3b44dd59c00ef151adca49d [term 1 FOLLOWER]: Refusing update from remote peer 82f50f4b66f3450aa9f58a0b317f5cea: Log matching property violated. Preceding OpId in replica: term: 1 index: 131. Preceding OpId from leader: term: 1 index: 132. (index mismatch)
I20240502 21:26:09.583213 18314 catalog_manager.cc:4830] TS c75831edb3b44dd59c00ef151adca49d (127.15.255.132:43607): tablet 84b235fd89634168938c0dddb534462a (table rebalance_test_table_0 [id=086ede98fc604333833bce759f342073]) successfully deleted
I20240502 21:26:09.598907 19140 raft_consensus.cc:1261] T 693a8666bbe54ae1b774f7f4da5ba67e P 2877108756304e05bd60ba1186b870da [term 1 FOLLOWER]: Refusing update from remote peer faadabfb099b4abf90a466395be585b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 134. Preceding OpId from leader: term: 1 index: 135. (index mismatch)
I20240502 21:26:09.625196 20871 consensus_queue.cc:1035] T 693a8666bbe54ae1b774f7f4da5ba67e P faadabfb099b4abf90a466395be585b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 135, Last known committed idx: 134, Time since last communication: 0.000s
I20240502 21:26:09.625982 20636 consensus_queue.cc:1035] T 693a8666bbe54ae1b774f7f4da5ba67e P faadabfb099b4abf90a466395be585b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 135, Last known committed idx: 134, Time since last communication: 0.000s
I20240502 21:26:09.630144 18898 tablet_copy_service.cc:342] P c75831edb3b44dd59c00ef151adca49d: Request end of tablet copy session faadabfb099b4abf90a466395be585b5-37e9dcc6191c42d3a5c1503dedc5511e received from {username='slave'} at 127.15.255.129:50861
I20240502 21:26:09.630299 18898 tablet_copy_service.cc:433] P c75831edb3b44dd59c00ef151adca49d: ending tablet copy session faadabfb099b4abf90a466395be585b5-37e9dcc6191c42d3a5c1503dedc5511e on tablet 37e9dcc6191c42d3a5c1503dedc5511e with peer faadabfb099b4abf90a466395be585b5
I20240502 21:26:09.634719 20921 tablet_replica.cc:330] T a7a9256f2b784abcb780b190e2ff3983 P 9be399440ded4e2ebe6b6e0bf486cbb1: stopping tablet replica
I20240502 21:26:09.635198 20921 raft_consensus.cc:2229] T a7a9256f2b784abcb780b190e2ff3983 P 9be399440ded4e2ebe6b6e0bf486cbb1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:26:09.635885 20921 raft_consensus.cc:2258] T a7a9256f2b784abcb780b190e2ff3983 P 9be399440ded4e2ebe6b6e0bf486cbb1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:09.639389 18611 raft_consensus.cc:2932] T 693a8666bbe54ae1b774f7f4da5ba67e P 82f50f4b66f3450aa9f58a0b317f5cea [term 1 FOLLOWER]: Committing config change with OpId 1.135: config changed from index -1 to 135, NON_VOTER 90e329b457c24caf93a9f880345eb288 (127.15.255.135) added. New config: { opid_index: 135 OBSOLETE_local: false peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } attrs { replace: true } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: NON_VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: true } } }
I20240502 21:26:09.646524 20640 consensus_queue.cc:1035] T a7a9256f2b784abcb780b190e2ff3983 P 82f50f4b66f3450aa9f58a0b317f5cea [LEADER]: Connected to new peer: Peer: permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 132, Last known committed idx: 131, Time since last communication: 0.000s
I20240502 21:26:09.655951 19139 raft_consensus.cc:2932] T 693a8666bbe54ae1b774f7f4da5ba67e P 2877108756304e05bd60ba1186b870da [term 1 FOLLOWER]: Committing config change with OpId 1.135: config changed from index -1 to 135, NON_VOTER 90e329b457c24caf93a9f880345eb288 (127.15.255.135) added. New config: { opid_index: 135 OBSOLETE_local: false peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } attrs { replace: true } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: NON_VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: true } } }
I20240502 21:26:09.669317 18877 raft_consensus.cc:2932] T 6f0107b0690c4c64b0a4f2472a1c8322 P c75831edb3b44dd59c00ef151adca49d [term 1 FOLLOWER]: Committing config change with OpId 1.134: config changed from index 121 to 134, VOTER 2877108756304e05bd60ba1186b870da (127.15.255.134) evicted. New config: { opid_index: 134 OBSOLETE_local: false peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } } }
I20240502 21:26:09.712100 20921 ts_tablet_manager.cc:1875] T a7a9256f2b784abcb780b190e2ff3983 P 9be399440ded4e2ebe6b6e0bf486cbb1: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:26:09.726961 20912 tablet_replica.cc:330] T 6f0107b0690c4c64b0a4f2472a1c8322 P 2877108756304e05bd60ba1186b870da: stopping tablet replica
I20240502 21:26:09.727588 20912 raft_consensus.cc:2229] T 6f0107b0690c4c64b0a4f2472a1c8322 P 2877108756304e05bd60ba1186b870da [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:26:09.728286 20912 pending_rounds.cc:62] T 6f0107b0690c4c64b0a4f2472a1c8322 P 2877108756304e05bd60ba1186b870da: Trying to abort 1 pending ops.
I20240502 21:26:09.730489 20168 raft_consensus.cc:2932] T a7a9256f2b784abcb780b190e2ff3983 P 90e329b457c24caf93a9f880345eb288 [term 1 FOLLOWER]: Committing config change with OpId 1.132: config changed from index 130 to 132, VOTER 9be399440ded4e2ebe6b6e0bf486cbb1 (127.15.255.133) evicted. New config: { opid_index: 132 OBSOLETE_local: false peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } } }
I20240502 21:26:09.736070 20912 pending_rounds.cc:68] T 6f0107b0690c4c64b0a4f2472a1c8322 P 2877108756304e05bd60ba1186b870da: Aborting op as it isn't in flight: id { term: 1 index: 133 } timestamp: 7023350452216406016 op_type: WRITE_OP write_request { tablet_id: "6f0107b0690c4c64b0a4f2472a1c8322" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000\345tnxQL\275}\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""175fa71691a58455" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7023350451795566592 authz_token { token_data: "\010\216\214\320\261\006\"3\n\005slave\022*\n c79c94b660074ffc94fcea415b4caae8\020\001\030\001 \001(\001" signature: "<redacted>""\234zO\371,X\245\257!o\364\026\304q\231\333\306\006\037\226\003\331\367vk\230\202\255/\244*\377\314>w\321;\\\352\030\255l\312\234\345\200\371Q\275Z\321%:\337\037\277\241\306\352\376K\243\327\213" signing_key_seq_num: 0 } } request_id { client_id: "df649a9d7e6f4877835490901b211482" seq_no: 376 first_incomplete_seq_no: 376 attempt_no: 0 }
I20240502 21:26:09.737155 20912 raft_consensus.cc:2258] T 6f0107b0690c4c64b0a4f2472a1c8322 P 2877108756304e05bd60ba1186b870da [term 1 FOLLOWER]: Raft consensus is shut down!
W20240502 21:26:09.737355 19124 thread.cc:636] tablet-delete [worker] (thread pool) Time spent creating pthread: real 0.545s user 0.013s sys 0.030s
W20240502 21:26:09.739847 19124 thread.cc:602] tablet-delete [worker] (thread pool) Time spent starting thread: real 0.548s user 0.013s sys 0.030s
I20240502 21:26:09.743441 20912 ts_tablet_manager.cc:1875] T 6f0107b0690c4c64b0a4f2472a1c8322 P 2877108756304e05bd60ba1186b870da: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:26:09.755825 20921 ts_tablet_manager.cc:1888] T a7a9256f2b784abcb780b190e2ff3983 P 9be399440ded4e2ebe6b6e0bf486cbb1: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.131
I20240502 21:26:09.757688 20921 log.cc:1196] T a7a9256f2b784abcb780b190e2ff3983 P 9be399440ded4e2ebe6b6e0bf486cbb1: Deleting WAL directory at /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-4/wal/wals/a7a9256f2b784abcb780b190e2ff3983
I20240502 21:26:09.781672 18313 catalog_manager.cc:4830] TS 9be399440ded4e2ebe6b6e0bf486cbb1 (127.15.255.133:34217): tablet a7a9256f2b784abcb780b190e2ff3983 (table rebalance_test_table_1 [id=76deee1e4597471ebedfc30f6d635eaa]) successfully deleted
I20240502 21:26:09.785025 20912 ts_tablet_manager.cc:1888] T 6f0107b0690c4c64b0a4f2472a1c8322 P 2877108756304e05bd60ba1186b870da: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.133
I20240502 21:26:09.786459 20912 log.cc:1196] T 6f0107b0690c4c64b0a4f2472a1c8322 P 2877108756304e05bd60ba1186b870da: Deleting WAL directory at /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-5/wal/wals/6f0107b0690c4c64b0a4f2472a1c8322
I20240502 21:26:09.788718 18315 catalog_manager.cc:4830] TS 2877108756304e05bd60ba1186b870da (127.15.255.134:43635): tablet 6f0107b0690c4c64b0a4f2472a1c8322 (table rebalance_test_table_4 [id=c79c94b660074ffc94fcea415b4caae8]) successfully deleted
W20240502 21:26:09.791564 18407 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.130:45139, user_credentials={real_user=slave}} blocked reactor thread for 80884.5us
I20240502 21:26:09.915244 18470 raft_consensus.cc:1203] T 37e9dcc6191c42d3a5c1503dedc5511e P faadabfb099b4abf90a466395be585b5 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.116->[1.117-1.133] Dedup: 1.125->[1.126-1.133]
I20240502 21:26:09.993721 20935 ts_tablet_manager.cc:902] T 693a8666bbe54ae1b774f7f4da5ba67e P 90e329b457c24caf93a9f880345eb288: Initiating tablet copy from peer faadabfb099b4abf90a466395be585b5 (127.15.255.129:32983)
I20240502 21:26:10.129995 18877 raft_consensus.cc:2932] T a7a9256f2b784abcb780b190e2ff3983 P c75831edb3b44dd59c00ef151adca49d [term 1 FOLLOWER]: Committing config change with OpId 1.132: config changed from index 130 to 132, VOTER 9be399440ded4e2ebe6b6e0bf486cbb1 (127.15.255.133) evicted. New config: { opid_index: 132 OBSOLETE_local: false peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } } }
I20240502 21:26:10.317793 20885 raft_consensus.cc:1050] T 37e9dcc6191c42d3a5c1503dedc5511e P c75831edb3b44dd59c00ef151adca49d: attempting to promote NON_VOTER faadabfb099b4abf90a466395be585b5 to VOTER
W20240502 21:26:10.352232 18407 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.130:45139, user_credentials={real_user=slave}} blocked reactor thread for 50784.6us
W20240502 21:26:10.477780 18406 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.135:43395, user_credentials={real_user=slave}} blocked reactor thread for 96798.9us
W20240502 21:26:10.514343 18407 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.130:45139, user_credentials={real_user=slave}} blocked reactor thread for 120553us
W20240502 21:26:10.605235 18812 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.134:43635, user_credentials={real_user=slave}} blocked reactor thread for 47165.3us
I20240502 21:26:10.613483 20885 consensus_queue.cc:237] T 37e9dcc6191c42d3a5c1503dedc5511e P c75831edb3b44dd59c00ef151adca49d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 133, Committed index: 133, Last appended: 1.134, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 135 OBSOLETE_local: false peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } attrs { replace: true } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } attrs { promote: false } }
I20240502 21:26:10.625449 18472 raft_consensus.cc:1261] T 37e9dcc6191c42d3a5c1503dedc5511e P faadabfb099b4abf90a466395be585b5 [term 1 LEARNER]: Refusing update from remote peer c75831edb3b44dd59c00ef151adca49d: Log matching property violated. Preceding OpId in replica: term: 1 index: 134. Preceding OpId from leader: term: 1 index: 135. (index mismatch)
I20240502 21:26:10.626649 20886 consensus_queue.cc:1035] T 37e9dcc6191c42d3a5c1503dedc5511e P c75831edb3b44dd59c00ef151adca49d [LEADER]: Connected to new peer: Peer: permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 135, Last known committed idx: 133, Time since last communication: 0.000s
I20240502 21:26:10.628005 19010 raft_consensus.cc:1261] T 37e9dcc6191c42d3a5c1503dedc5511e P 9be399440ded4e2ebe6b6e0bf486cbb1 [term 1 FOLLOWER]: Refusing update from remote peer c75831edb3b44dd59c00ef151adca49d: Log matching property violated. Preceding OpId in replica: term: 1 index: 134. Preceding OpId from leader: term: 1 index: 135. (index mismatch)
I20240502 21:26:10.629042 20886 consensus_queue.cc:1035] T 37e9dcc6191c42d3a5c1503dedc5511e P c75831edb3b44dd59c00ef151adca49d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 135, Last known committed idx: 134, Time since last communication: 0.000s
I20240502 21:26:10.639071 18739 raft_consensus.cc:1261] T 37e9dcc6191c42d3a5c1503dedc5511e P 501e3c0d44fe4d70af9ff926d9622417 [term 1 FOLLOWER]: Refusing update from remote peer c75831edb3b44dd59c00ef151adca49d: Log matching property violated. Preceding OpId in replica: term: 1 index: 134. Preceding OpId from leader: term: 1 index: 135. (index mismatch)
I20240502 21:26:10.643071 20886 consensus_queue.cc:1035] T 37e9dcc6191c42d3a5c1503dedc5511e P c75831edb3b44dd59c00ef151adca49d [LEADER]: Connected to new peer: Peer: permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 135, Last known committed idx: 133, Time since last communication: 0.000s
I20240502 21:26:10.647490 18739 raft_consensus.cc:2932] T 37e9dcc6191c42d3a5c1503dedc5511e P 501e3c0d44fe4d70af9ff926d9622417 [term 1 FOLLOWER]: Committing config change with OpId 1.135: config changed from index 117 to 135, faadabfb099b4abf90a466395be585b5 (127.15.255.129) changed from NON_VOTER to VOTER. New config: { opid_index: 135 OBSOLETE_local: false peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } attrs { replace: true } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } attrs { promote: false } } }
I20240502 21:26:10.654928 18329 catalog_manager.cc:5484] T 37e9dcc6191c42d3a5c1503dedc5511e P 501e3c0d44fe4d70af9ff926d9622417 reported cstate change: config changed from index 117 to 135, faadabfb099b4abf90a466395be585b5 (127.15.255.129) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c75831edb3b44dd59c00ef151adca49d" committed_config { opid_index: 135 OBSOLETE_local: false peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } attrs { replace: true } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } attrs { promote: false } } }
I20240502 21:26:10.711055 18472 raft_consensus.cc:2932] T 37e9dcc6191c42d3a5c1503dedc5511e P faadabfb099b4abf90a466395be585b5 [term 1 FOLLOWER]: Committing config change with OpId 1.135: config changed from index 117 to 135, faadabfb099b4abf90a466395be585b5 (127.15.255.129) changed from NON_VOTER to VOTER. New config: { opid_index: 135 OBSOLETE_local: false peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } attrs { replace: true } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } attrs { promote: false } } }
I20240502 21:26:10.714794 20304 raft_consensus.cc:2932] T 37e9dcc6191c42d3a5c1503dedc5511e P c75831edb3b44dd59c00ef151adca49d [term 1 LEADER]: Committing config change with OpId 1.135: config changed from index 117 to 135, faadabfb099b4abf90a466395be585b5 (127.15.255.129) changed from NON_VOTER to VOTER. New config: { opid_index: 135 OBSOLETE_local: false peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } attrs { replace: true } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } attrs { promote: false } } }
I20240502 21:26:10.729118 19010 raft_consensus.cc:2932] T 37e9dcc6191c42d3a5c1503dedc5511e P 9be399440ded4e2ebe6b6e0bf486cbb1 [term 1 FOLLOWER]: Committing config change with OpId 1.135: config changed from index 117 to 135, faadabfb099b4abf90a466395be585b5 (127.15.255.129) changed from NON_VOTER to VOTER. New config: { opid_index: 135 OBSOLETE_local: false peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } attrs { replace: true } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } attrs { promote: false } } }
W20240502 21:26:10.785481 18406 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.132:43607, user_credentials={real_user=slave}} blocked reactor thread for 87275us
I20240502 21:26:10.789697 19141 raft_consensus.cc:1261] T 37e9dcc6191c42d3a5c1503dedc5511e P 2877108756304e05bd60ba1186b870da [term 1 FOLLOWER]: Refusing update from remote peer c75831edb3b44dd59c00ef151adca49d: Log matching property violated. Preceding OpId in replica: term: 1 index: 134. Preceding OpId from leader: term: 1 index: 135. (index mismatch)
W20240502 21:26:10.851650 18878 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.503s user 0.096s sys 0.115s
W20240502 21:26:10.851805 18878 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.503s user 0.096s sys 0.115s
I20240502 21:26:10.873577 18609 raft_consensus.cc:1261] T 37e9dcc6191c42d3a5c1503dedc5511e P 82f50f4b66f3450aa9f58a0b317f5cea [term 1 FOLLOWER]: Refusing update from remote peer c75831edb3b44dd59c00ef151adca49d: Log matching property violated. Preceding OpId in replica: term: 1 index: 134. Preceding OpId from leader: term: 1 index: 135. (index mismatch)
I20240502 21:26:10.875816 20935 tablet_copy_client.cc:308] T 693a8666bbe54ae1b774f7f4da5ba67e P 90e329b457c24caf93a9f880345eb288: tablet copy: Beginning tablet copy session from remote peer at address 127.15.255.129:32983
I20240502 21:26:10.876732 18496 tablet_copy_service.cc:140] P faadabfb099b4abf90a466395be585b5: Received BeginTabletCopySession request for tablet 693a8666bbe54ae1b774f7f4da5ba67e from peer 90e329b457c24caf93a9f880345eb288 ({username='slave'} at 127.15.255.135:49783)
I20240502 21:26:10.876895 18496 tablet_copy_service.cc:161] P faadabfb099b4abf90a466395be585b5: Beginning new tablet copy session on tablet 693a8666bbe54ae1b774f7f4da5ba67e from peer 90e329b457c24caf93a9f880345eb288 at {username='slave'} at 127.15.255.135:49783: session id = 90e329b457c24caf93a9f880345eb288-693a8666bbe54ae1b774f7f4da5ba67e
I20240502 21:26:10.879356 18496 tablet_copy_source_session.cc:205] T 693a8666bbe54ae1b774f7f4da5ba67e P faadabfb099b4abf90a466395be585b5: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:26:10.880795 20935 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 693a8666bbe54ae1b774f7f4da5ba67e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:10.887696 20935 tablet_copy_client.cc:783] T 693a8666bbe54ae1b774f7f4da5ba67e P 90e329b457c24caf93a9f880345eb288: tablet copy: Starting download of 0 data blocks...
I20240502 21:26:10.888020 20935 tablet_copy_client.cc:647] T 693a8666bbe54ae1b774f7f4da5ba67e P 90e329b457c24caf93a9f880345eb288: tablet copy: Starting download of 1 WAL segments...
I20240502 21:26:10.890533 20935 tablet_copy_client.cc:523] T 693a8666bbe54ae1b774f7f4da5ba67e P 90e329b457c24caf93a9f880345eb288: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:26:10.893894 20935 tablet_bootstrap.cc:492] T 693a8666bbe54ae1b774f7f4da5ba67e P 90e329b457c24caf93a9f880345eb288: Bootstrap starting.
I20240502 21:26:10.898932 20304 consensus_queue.cc:1035] T 37e9dcc6191c42d3a5c1503dedc5511e P c75831edb3b44dd59c00ef151adca49d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 135, Last known committed idx: 134, Time since last communication: 0.000s
I20240502 21:26:10.899354 20885 consensus_queue.cc:1035] T 37e9dcc6191c42d3a5c1503dedc5511e P c75831edb3b44dd59c00ef151adca49d [LEADER]: Connected to new peer: Peer: permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 135, Last known committed idx: 134, Time since last communication: 0.000s
I20240502 21:26:10.911053 19139 raft_consensus.cc:2932] T 37e9dcc6191c42d3a5c1503dedc5511e P 2877108756304e05bd60ba1186b870da [term 1 FOLLOWER]: Committing config change with OpId 1.135: config changed from index 117 to 135, faadabfb099b4abf90a466395be585b5 (127.15.255.129) changed from NON_VOTER to VOTER. New config: { opid_index: 135 OBSOLETE_local: false peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } attrs { replace: true } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } attrs { promote: false } } }
I20240502 21:26:10.919463 18611 raft_consensus.cc:2932] T 37e9dcc6191c42d3a5c1503dedc5511e P 82f50f4b66f3450aa9f58a0b317f5cea [term 1 FOLLOWER]: Committing config change with OpId 1.135: config changed from index 117 to 135, faadabfb099b4abf90a466395be585b5 (127.15.255.129) changed from NON_VOTER to VOTER. New config: { opid_index: 135 OBSOLETE_local: false peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } attrs { replace: true } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } attrs { promote: false } } }
W20240502 21:26:11.027769 18406 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.135:43395, user_credentials={real_user=slave}} blocked reactor thread for 229892us
W20240502 21:26:11.029265 18407 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.130:45139, user_credentials={real_user=slave}} blocked reactor thread for 88024.7us
I20240502 21:26:11.110561 18878 consensus_queue.cc:237] T 37e9dcc6191c42d3a5c1503dedc5511e P c75831edb3b44dd59c00ef151adca49d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 136, Committed index: 136, Last appended: 1.136, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 137 OBSOLETE_local: false peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } attrs { promote: false } }
I20240502 21:26:11.113466 19139 raft_consensus.cc:1261] T 37e9dcc6191c42d3a5c1503dedc5511e P 2877108756304e05bd60ba1186b870da [term 1 FOLLOWER]: Refusing update from remote peer c75831edb3b44dd59c00ef151adca49d: Log matching property violated. Preceding OpId in replica: term: 1 index: 136. Preceding OpId from leader: term: 1 index: 137. (index mismatch)
I20240502 21:26:11.114012 20885 consensus_queue.cc:1035] T 37e9dcc6191c42d3a5c1503dedc5511e P c75831edb3b44dd59c00ef151adca49d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 137, Last known committed idx: 136, Time since last communication: 0.000s
I20240502 21:26:11.114205 18472 raft_consensus.cc:1261] T 37e9dcc6191c42d3a5c1503dedc5511e P faadabfb099b4abf90a466395be585b5 [term 1 FOLLOWER]: Refusing update from remote peer c75831edb3b44dd59c00ef151adca49d: Log matching property violated. Preceding OpId in replica: term: 1 index: 136. Preceding OpId from leader: term: 1 index: 137. (index mismatch)
I20240502 21:26:11.115499 20885 consensus_queue.cc:1035] T 37e9dcc6191c42d3a5c1503dedc5511e P c75831edb3b44dd59c00ef151adca49d [LEADER]: Connected to new peer: Peer: permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 137, Last known committed idx: 136, Time since last communication: 0.000s
I20240502 21:26:11.118373 20885 raft_consensus.cc:2932] T 37e9dcc6191c42d3a5c1503dedc5511e P c75831edb3b44dd59c00ef151adca49d [term 1 LEADER]: Committing config change with OpId 1.137: config changed from index 135 to 137, VOTER 501e3c0d44fe4d70af9ff926d9622417 (127.15.255.131) evicted. New config: { opid_index: 137 OBSOLETE_local: false peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } attrs { promote: false } } }
I20240502 21:26:11.118969 19005 raft_consensus.cc:1261] T 37e9dcc6191c42d3a5c1503dedc5511e P 9be399440ded4e2ebe6b6e0bf486cbb1 [term 1 FOLLOWER]: Refusing update from remote peer c75831edb3b44dd59c00ef151adca49d: Log matching property violated. Preceding OpId in replica: term: 1 index: 136. Preceding OpId from leader: term: 1 index: 137. (index mismatch)
I20240502 21:26:11.128690 19140 raft_consensus.cc:2932] T 37e9dcc6191c42d3a5c1503dedc5511e P 2877108756304e05bd60ba1186b870da [term 1 FOLLOWER]: Committing config change with OpId 1.137: config changed from index 135 to 137, VOTER 501e3c0d44fe4d70af9ff926d9622417 (127.15.255.131) evicted. New config: { opid_index: 137 OBSOLETE_local: false peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } attrs { promote: false } } }
I20240502 21:26:11.129439 18314 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 37e9dcc6191c42d3a5c1503dedc5511e with cas_config_opid_index 135: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:26:11.134145 18611 raft_consensus.cc:1261] T 37e9dcc6191c42d3a5c1503dedc5511e P 82f50f4b66f3450aa9f58a0b317f5cea [term 1 FOLLOWER]: Refusing update from remote peer c75831edb3b44dd59c00ef151adca49d: Log matching property violated. Preceding OpId in replica: term: 1 index: 136. Preceding OpId from leader: term: 1 index: 137. (index mismatch)
I20240502 21:26:11.136809 20963 consensus_queue.cc:1035] T 37e9dcc6191c42d3a5c1503dedc5511e P c75831edb3b44dd59c00ef151adca49d [LEADER]: Connected to new peer: Peer: permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 137, Last known committed idx: 136, Time since last communication: 0.000s
I20240502 21:26:11.139173 20963 consensus_queue.cc:1035] T 37e9dcc6191c42d3a5c1503dedc5511e P c75831edb3b44dd59c00ef151adca49d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 137, Last known committed idx: 136, Time since last communication: 0.000s
I20240502 21:26:11.143990 18611 raft_consensus.cc:2932] T 37e9dcc6191c42d3a5c1503dedc5511e P 82f50f4b66f3450aa9f58a0b317f5cea [term 1 FOLLOWER]: Committing config change with OpId 1.137: config changed from index 135 to 137, VOTER 501e3c0d44fe4d70af9ff926d9622417 (127.15.255.131) evicted. New config: { opid_index: 137 OBSOLETE_local: false peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } attrs { promote: false } } }
I20240502 21:26:11.152402 18327 catalog_manager.cc:5484] T 37e9dcc6191c42d3a5c1503dedc5511e P 2877108756304e05bd60ba1186b870da reported cstate change: config changed from index 135 to 137, VOTER 501e3c0d44fe4d70af9ff926d9622417 (127.15.255.131) evicted. New cstate: current_term: 1 leader_uuid: "c75831edb3b44dd59c00ef151adca49d" committed_config { opid_index: 137 OBSOLETE_local: false peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } attrs { promote: false } } }
I20240502 21:26:11.157310 18722 tablet_service.cc:1514] Processing DeleteTablet for tablet 37e9dcc6191c42d3a5c1503dedc5511e with delete_type TABLET_DATA_TOMBSTONED (TS 501e3c0d44fe4d70af9ff926d9622417 not found in new config with opid_index 137) from {username='slave'} at 127.0.0.1:56558
I20240502 21:26:11.157377 18474 raft_consensus.cc:2932] T 37e9dcc6191c42d3a5c1503dedc5511e P faadabfb099b4abf90a466395be585b5 [term 1 FOLLOWER]: Committing config change with OpId 1.137: config changed from index 135 to 137, VOTER 501e3c0d44fe4d70af9ff926d9622417 (127.15.255.131) evicted. New config: { opid_index: 137 OBSOLETE_local: false peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } attrs { promote: false } } }
I20240502 21:26:11.186728 19008 raft_consensus.cc:2932] T 37e9dcc6191c42d3a5c1503dedc5511e P 9be399440ded4e2ebe6b6e0bf486cbb1 [term 1 FOLLOWER]: Committing config change with OpId 1.137: config changed from index 135 to 137, VOTER 501e3c0d44fe4d70af9ff926d9622417 (127.15.255.131) evicted. New config: { opid_index: 137 OBSOLETE_local: false peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } attrs { promote: false } } }
I20240502 21:26:11.234838 20997 tablet_replica.cc:330] T 37e9dcc6191c42d3a5c1503dedc5511e P 501e3c0d44fe4d70af9ff926d9622417: stopping tablet replica
I20240502 21:26:11.248245 20997 raft_consensus.cc:2229] T 37e9dcc6191c42d3a5c1503dedc5511e P 501e3c0d44fe4d70af9ff926d9622417 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:26:11.255538 20997 raft_consensus.cc:2258] T 37e9dcc6191c42d3a5c1503dedc5511e P 501e3c0d44fe4d70af9ff926d9622417 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:11.316267 20997 ts_tablet_manager.cc:1875] T 37e9dcc6191c42d3a5c1503dedc5511e P 501e3c0d44fe4d70af9ff926d9622417: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20240502 21:26:11.349949 18680 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.134:43635, user_credentials={real_user=slave}} blocked reactor thread for 57218.3us
W20240502 21:26:11.427122 19077 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.129:32983, user_credentials={real_user=slave}} blocked reactor thread for 77702.2us
W20240502 21:26:11.445698 20935 log.cc:924] Time spent T 693a8666bbe54ae1b774f7f4da5ba67e P 90e329b457c24caf93a9f880345eb288: Append to log took a long time: real 0.063s user 0.000s sys 0.001s
I20240502 21:26:11.527841 20997 ts_tablet_manager.cc:1888] T 37e9dcc6191c42d3a5c1503dedc5511e P 501e3c0d44fe4d70af9ff926d9622417: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.136
I20240502 21:26:11.528143 20997 log.cc:1196] T 37e9dcc6191c42d3a5c1503dedc5511e P 501e3c0d44fe4d70af9ff926d9622417: Deleting WAL directory at /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-2/wal/wals/37e9dcc6191c42d3a5c1503dedc5511e
I20240502 21:26:11.532992 18312 catalog_manager.cc:4830] TS 501e3c0d44fe4d70af9ff926d9622417 (127.15.255.131:42501): tablet 37e9dcc6191c42d3a5c1503dedc5511e (table rebalance_test_table_1 [id=76deee1e4597471ebedfc30f6d635eaa]) successfully deleted
W20240502 21:26:11.610824 20935 log.cc:924] Time spent T 693a8666bbe54ae1b774f7f4da5ba67e P 90e329b457c24caf93a9f880345eb288: Append to log took a long time: real 0.072s user 0.001s sys 0.001s
W20240502 21:26:12.230955 20935 log.cc:924] Time spent T 693a8666bbe54ae1b774f7f4da5ba67e P 90e329b457c24caf93a9f880345eb288: Append to log took a long time: real 0.064s user 0.000s sys 0.000s
W20240502 21:26:12.371594 18406 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.132:43607, user_credentials={real_user=slave}} blocked reactor thread for 145388us
W20240502 21:26:12.618655 18945 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.132:43607, user_credentials={real_user=slave}} blocked reactor thread for 47592.8us
W20240502 21:26:12.755466 18407 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.130:45139, user_credentials={real_user=slave}} blocked reactor thread for 44313us
W20240502 21:26:12.879060 18812 outbound_call.cc:322] RPC callback for RPC call kudu.master.MasterService.TSHeartbeat -> {remote=127.15.255.190:36947, user_credentials={real_user=slave}} blocked reactor thread for 54688.3us
I20240502 21:26:12.948940 20935 tablet_bootstrap.cc:492] T 693a8666bbe54ae1b774f7f4da5ba67e P 90e329b457c24caf93a9f880345eb288: Bootstrap replayed 1/1 log segments. Stats: ops{read=151 overwritten=0 applied=151 ignored=0} inserts{seen=782 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:26:12.949564 20935 tablet_bootstrap.cc:492] T 693a8666bbe54ae1b774f7f4da5ba67e P 90e329b457c24caf93a9f880345eb288: Bootstrap complete.
I20240502 21:26:12.950057 20935 ts_tablet_manager.cc:1367] T 693a8666bbe54ae1b774f7f4da5ba67e P 90e329b457c24caf93a9f880345eb288: Time spent bootstrapping tablet: real 2.056s user 0.214s sys 0.042s
I20240502 21:26:12.951032 20935 raft_consensus.cc:348] T 693a8666bbe54ae1b774f7f4da5ba67e P 90e329b457c24caf93a9f880345eb288 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 135 OBSOLETE_local: false peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } attrs { replace: true } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: NON_VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: true } }
I20240502 21:26:12.951871 20935 raft_consensus.cc:729] T 693a8666bbe54ae1b774f7f4da5ba67e P 90e329b457c24caf93a9f880345eb288 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 90e329b457c24caf93a9f880345eb288, State: Initialized, Role: LEARNER
I20240502 21:26:12.952252 20935 consensus_queue.cc:260] T 693a8666bbe54ae1b774f7f4da5ba67e P 90e329b457c24caf93a9f880345eb288 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 151, Last appended: 1.151, Last appended by leader: 151, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 135 OBSOLETE_local: false peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } attrs { replace: true } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: NON_VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: true } }
I20240502 21:26:12.961241 20935 ts_tablet_manager.cc:1398] T 693a8666bbe54ae1b774f7f4da5ba67e P 90e329b457c24caf93a9f880345eb288: Time spent starting tablet: real 0.011s user 0.006s sys 0.000s
I20240502 21:26:12.962252 18496 tablet_copy_service.cc:342] P faadabfb099b4abf90a466395be585b5: Request end of tablet copy session 90e329b457c24caf93a9f880345eb288-693a8666bbe54ae1b774f7f4da5ba67e received from {username='slave'} at 127.15.255.135:49783
I20240502 21:26:12.962416 18496 tablet_copy_service.cc:433] P faadabfb099b4abf90a466395be585b5: ending tablet copy session 90e329b457c24caf93a9f880345eb288-693a8666bbe54ae1b774f7f4da5ba67e on tablet 693a8666bbe54ae1b774f7f4da5ba67e with peer 90e329b457c24caf93a9f880345eb288
I20240502 21:26:13.057503 20166 raft_consensus.cc:1203] T 693a8666bbe54ae1b774f7f4da5ba67e P 90e329b457c24caf93a9f880345eb288 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.134->[1.135-1.176] Dedup: 1.151->[1.152-1.176]
I20240502 21:26:13.202244 18471 consensus_queue.cc:237] T 32c9fc5a18c54207881124fda86f42ff P faadabfb099b4abf90a466395be585b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 167, Committed index: 167, Last appended: 1.167, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 168 OBSOLETE_local: false peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } attrs { replace: true } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: NON_VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: true } }
I20240502 21:26:13.206426 18611 raft_consensus.cc:1261] T 32c9fc5a18c54207881124fda86f42ff P 82f50f4b66f3450aa9f58a0b317f5cea [term 1 FOLLOWER]: Refusing update from remote peer faadabfb099b4abf90a466395be585b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 167. Preceding OpId from leader: term: 1 index: 168. (index mismatch)
I20240502 21:26:13.208355 18738 raft_consensus.cc:1261] T 32c9fc5a18c54207881124fda86f42ff P 501e3c0d44fe4d70af9ff926d9622417 [term 1 FOLLOWER]: Refusing update from remote peer faadabfb099b4abf90a466395be585b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 167. Preceding OpId from leader: term: 1 index: 168. (index mismatch)
I20240502 21:26:13.208410 20953 consensus_queue.cc:1035] T 32c9fc5a18c54207881124fda86f42ff P faadabfb099b4abf90a466395be585b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 168, Last known committed idx: 167, Time since last communication: 0.000s
I20240502 21:26:13.209064 21054 consensus_queue.cc:1035] T 32c9fc5a18c54207881124fda86f42ff P faadabfb099b4abf90a466395be585b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 168, Last known committed idx: 167, Time since last communication: 0.000s
W20240502 21:26:13.211378 18406 consensus_peers.cc:485] T 32c9fc5a18c54207881124fda86f42ff P faadabfb099b4abf90a466395be585b5 -> Peer 90e329b457c24caf93a9f880345eb288 (127.15.255.135:43395): Couldn't send request to peer 90e329b457c24caf93a9f880345eb288. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 32c9fc5a18c54207881124fda86f42ff. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:26:13.211427 20953 raft_consensus.cc:2932] T 32c9fc5a18c54207881124fda86f42ff P faadabfb099b4abf90a466395be585b5 [term 1 LEADER]: Committing config change with OpId 1.168: config changed from index -1 to 168, NON_VOTER 90e329b457c24caf93a9f880345eb288 (127.15.255.135) added. New config: { opid_index: 168 OBSOLETE_local: false peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } attrs { replace: true } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: NON_VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: true } } }
I20240502 21:26:13.219139 18610 raft_consensus.cc:2932] T 32c9fc5a18c54207881124fda86f42ff P 82f50f4b66f3450aa9f58a0b317f5cea [term 1 FOLLOWER]: Committing config change with OpId 1.168: config changed from index -1 to 168, NON_VOTER 90e329b457c24caf93a9f880345eb288 (127.15.255.135) added. New config: { opid_index: 168 OBSOLETE_local: false peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } attrs { replace: true } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: NON_VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: true } } }
I20240502 21:26:13.219669 19141 raft_consensus.cc:1261] T 32c9fc5a18c54207881124fda86f42ff P 2877108756304e05bd60ba1186b870da [term 1 FOLLOWER]: Refusing update from remote peer faadabfb099b4abf90a466395be585b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 167. Preceding OpId from leader: term: 1 index: 168. (index mismatch)
I20240502 21:26:13.218468 18738 raft_consensus.cc:2932] T 32c9fc5a18c54207881124fda86f42ff P 501e3c0d44fe4d70af9ff926d9622417 [term 1 FOLLOWER]: Committing config change with OpId 1.168: config changed from index -1 to 168, NON_VOTER 90e329b457c24caf93a9f880345eb288 (127.15.255.135) added. New config: { opid_index: 168 OBSOLETE_local: false peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } attrs { replace: true } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: NON_VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: true } } }
I20240502 21:26:13.221084 18327 catalog_manager.cc:5484] T 32c9fc5a18c54207881124fda86f42ff P faadabfb099b4abf90a466395be585b5 reported cstate change: config changed from index -1 to 168, NON_VOTER 90e329b457c24caf93a9f880345eb288 (127.15.255.135) added. New cstate: current_term: 1 leader_uuid: "faadabfb099b4abf90a466395be585b5" committed_config { opid_index: 168 OBSOLETE_local: false peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: NON_VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 21:26:13.257786 20953 consensus_queue.cc:1035] T 32c9fc5a18c54207881124fda86f42ff P faadabfb099b4abf90a466395be585b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 168, Last known committed idx: 167, Time since last communication: 0.000s
W20240502 21:26:13.259295 18810 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.133:34217, user_credentials={real_user=slave}} blocked reactor thread for 75818.5us
I20240502 21:26:13.276830 18873 raft_consensus.cc:1261] T 32c9fc5a18c54207881124fda86f42ff P c75831edb3b44dd59c00ef151adca49d [term 1 FOLLOWER]: Refusing update from remote peer faadabfb099b4abf90a466395be585b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 166. Preceding OpId from leader: term: 1 index: 168. (index mismatch)
I20240502 21:26:13.283105 21055 consensus_queue.cc:1035] T 32c9fc5a18c54207881124fda86f42ff P faadabfb099b4abf90a466395be585b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 168, Last known committed idx: 165, Time since last communication: 0.000s
I20240502 21:26:13.327971 19143 raft_consensus.cc:2932] T 32c9fc5a18c54207881124fda86f42ff P 2877108756304e05bd60ba1186b870da [term 1 FOLLOWER]: Committing config change with OpId 1.168: config changed from index -1 to 168, NON_VOTER 90e329b457c24caf93a9f880345eb288 (127.15.255.135) added. New config: { opid_index: 168 OBSOLETE_local: false peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } attrs { replace: true } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: NON_VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: true } } }
I20240502 21:26:13.425191 18873 raft_consensus.cc:2932] T 32c9fc5a18c54207881124fda86f42ff P c75831edb3b44dd59c00ef151adca49d [term 1 FOLLOWER]: Committing config change with OpId 1.168: config changed from index -1 to 168, NON_VOTER 90e329b457c24caf93a9f880345eb288 (127.15.255.135) added. New config: { opid_index: 168 OBSOLETE_local: false peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } attrs { replace: true } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: NON_VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: true } } }
W20240502 21:26:13.466014 19079 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.130:45139, user_credentials={real_user=slave}} blocked reactor thread for 63548.2us
I20240502 21:26:13.470492 21055 raft_consensus.cc:1050] T 693a8666bbe54ae1b774f7f4da5ba67e P faadabfb099b4abf90a466395be585b5: attempting to promote NON_VOTER 90e329b457c24caf93a9f880345eb288 to VOTER
I20240502 21:26:13.471752 21055 consensus_queue.cc:237] T 693a8666bbe54ae1b774f7f4da5ba67e P faadabfb099b4abf90a466395be585b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 179, Committed index: 179, Last appended: 1.179, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 180 OBSOLETE_local: false peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } attrs { replace: true } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } }
I20240502 21:26:13.476620 20166 raft_consensus.cc:1261] T 693a8666bbe54ae1b774f7f4da5ba67e P 90e329b457c24caf93a9f880345eb288 [term 1 LEARNER]: Refusing update from remote peer faadabfb099b4abf90a466395be585b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 179. Preceding OpId from leader: term: 1 index: 180. (index mismatch)
I20240502 21:26:13.477452 21054 consensus_queue.cc:1035] T 693a8666bbe54ae1b774f7f4da5ba67e P faadabfb099b4abf90a466395be585b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 180, Last known committed idx: 179, Time since last communication: 0.000s
I20240502 21:26:13.477596 18738 raft_consensus.cc:1261] T 693a8666bbe54ae1b774f7f4da5ba67e P 501e3c0d44fe4d70af9ff926d9622417 [term 1 FOLLOWER]: Refusing update from remote peer faadabfb099b4abf90a466395be585b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 179. Preceding OpId from leader: term: 1 index: 180. (index mismatch)
I20240502 21:26:13.478269 21054 consensus_queue.cc:1035] T 693a8666bbe54ae1b774f7f4da5ba67e P faadabfb099b4abf90a466395be585b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 180, Last known committed idx: 179, Time since last communication: 0.000s
I20240502 21:26:13.481113 19143 raft_consensus.cc:1261] T 693a8666bbe54ae1b774f7f4da5ba67e P 2877108756304e05bd60ba1186b870da [term 1 FOLLOWER]: Refusing update from remote peer faadabfb099b4abf90a466395be585b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 179. Preceding OpId from leader: term: 1 index: 180. (index mismatch)
W20240502 21:26:13.483702 19077 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.133:34217, user_credentials={real_user=slave}} blocked reactor thread for 93593.9us
I20240502 21:26:13.484663 21048 consensus_queue.cc:1035] T 693a8666bbe54ae1b774f7f4da5ba67e P faadabfb099b4abf90a466395be585b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 180, Last known committed idx: 179, Time since last communication: 0.000s
I20240502 21:26:13.486645 18609 raft_consensus.cc:1261] T 693a8666bbe54ae1b774f7f4da5ba67e P 82f50f4b66f3450aa9f58a0b317f5cea [term 1 FOLLOWER]: Refusing update from remote peer faadabfb099b4abf90a466395be585b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 179. Preceding OpId from leader: term: 1 index: 180. (index mismatch)
I20240502 21:26:13.490763 19010 consensus_queue.cc:237] T 4aff104e3b6b424baa628bbe612b62e8 P 9be399440ded4e2ebe6b6e0bf486cbb1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 157, Committed index: 157, Last appended: 1.157, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 158 OBSOLETE_local: false peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } attrs { replace: true } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: NON_VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: true } }
I20240502 21:26:13.494757 19139 raft_consensus.cc:1261] T 4aff104e3b6b424baa628bbe612b62e8 P 2877108756304e05bd60ba1186b870da [term 1 FOLLOWER]: Refusing update from remote peer 9be399440ded4e2ebe6b6e0bf486cbb1: Log matching property violated. Preceding OpId in replica: term: 1 index: 157. Preceding OpId from leader: term: 1 index: 158. (index mismatch)
I20240502 21:26:13.496315 21004 consensus_queue.cc:1035] T 4aff104e3b6b424baa628bbe612b62e8 P 9be399440ded4e2ebe6b6e0bf486cbb1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 158, Last known committed idx: 157, Time since last communication: 0.000s
I20240502 21:26:13.496646 18740 raft_consensus.cc:1261] T 4aff104e3b6b424baa628bbe612b62e8 P 501e3c0d44fe4d70af9ff926d9622417 [term 1 FOLLOWER]: Refusing update from remote peer 9be399440ded4e2ebe6b6e0bf486cbb1: Log matching property violated. Preceding OpId in replica: term: 1 index: 157. Preceding OpId from leader: term: 1 index: 158. (index mismatch)
I20240502 21:26:13.497303 21004 consensus_queue.cc:1035] T 4aff104e3b6b424baa628bbe612b62e8 P 9be399440ded4e2ebe6b6e0bf486cbb1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 158, Last known committed idx: 157, Time since last communication: 0.000s
I20240502 21:26:13.497690 19008 raft_consensus.cc:1261] T 693a8666bbe54ae1b774f7f4da5ba67e P 9be399440ded4e2ebe6b6e0bf486cbb1 [term 1 FOLLOWER]: Refusing update from remote peer faadabfb099b4abf90a466395be585b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 179. Preceding OpId from leader: term: 1 index: 180. (index mismatch)
I20240502 21:26:13.500782 21048 consensus_queue.cc:1035] T 693a8666bbe54ae1b774f7f4da5ba67e P faadabfb099b4abf90a466395be585b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 180, Last known committed idx: 179, Time since last communication: 0.000s
I20240502 21:26:13.500856 18878 raft_consensus.cc:1261] T 4aff104e3b6b424baa628bbe612b62e8 P c75831edb3b44dd59c00ef151adca49d [term 1 FOLLOWER]: Refusing update from remote peer 9be399440ded4e2ebe6b6e0bf486cbb1: Log matching property violated. Preceding OpId in replica: term: 1 index: 157. Preceding OpId from leader: term: 1 index: 158. (index mismatch)
I20240502 21:26:13.502044 21004 raft_consensus.cc:2932] T 4aff104e3b6b424baa628bbe612b62e8 P 9be399440ded4e2ebe6b6e0bf486cbb1 [term 1 LEADER]: Committing config change with OpId 1.158: config changed from index -1 to 158, NON_VOTER 90e329b457c24caf93a9f880345eb288 (127.15.255.135) added. New config: { opid_index: 158 OBSOLETE_local: false peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } attrs { replace: true } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: NON_VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: true } } }
W20240502 21:26:13.503118 18945 consensus_peers.cc:485] T 4aff104e3b6b424baa628bbe612b62e8 P 9be399440ded4e2ebe6b6e0bf486cbb1 -> Peer 90e329b457c24caf93a9f880345eb288 (127.15.255.135:43395): Couldn't send request to peer 90e329b457c24caf93a9f880345eb288. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4aff104e3b6b424baa628bbe612b62e8. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:26:13.504525 21016 consensus_queue.cc:1035] T 4aff104e3b6b424baa628bbe612b62e8 P 9be399440ded4e2ebe6b6e0bf486cbb1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 158, Last known committed idx: 157, Time since last communication: 0.000s
I20240502 21:26:13.506070 18740 raft_consensus.cc:2932] T 4aff104e3b6b424baa628bbe612b62e8 P 501e3c0d44fe4d70af9ff926d9622417 [term 1 FOLLOWER]: Committing config change with OpId 1.158: config changed from index -1 to 158, NON_VOTER 90e329b457c24caf93a9f880345eb288 (127.15.255.135) added. New config: { opid_index: 158 OBSOLETE_local: false peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } attrs { replace: true } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: NON_VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: true } } }
I20240502 21:26:13.513329 18320 catalog_manager.cc:5484] T 4aff104e3b6b424baa628bbe612b62e8 P 501e3c0d44fe4d70af9ff926d9622417 reported cstate change: config changed from index -1 to 158, NON_VOTER 90e329b457c24caf93a9f880345eb288 (127.15.255.135) added. New cstate: current_term: 1 leader_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" committed_config { opid_index: 158 OBSOLETE_local: false peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } attrs { replace: true } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: NON_VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: true } } }
I20240502 21:26:13.517903 19143 raft_consensus.cc:2932] T 4aff104e3b6b424baa628bbe612b62e8 P 2877108756304e05bd60ba1186b870da [term 1 FOLLOWER]: Committing config change with OpId 1.158: config changed from index -1 to 158, NON_VOTER 90e329b457c24caf93a9f880345eb288 (127.15.255.135) added. New config: { opid_index: 158 OBSOLETE_local: false peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } attrs { replace: true } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: NON_VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: true } } }
I20240502 21:26:13.518996 21055 consensus_queue.cc:1035] T 693a8666bbe54ae1b774f7f4da5ba67e P faadabfb099b4abf90a466395be585b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 180, Last known committed idx: 179, Time since last communication: 0.000s
I20240502 21:26:13.522452 18878 raft_consensus.cc:2932] T 4aff104e3b6b424baa628bbe612b62e8 P c75831edb3b44dd59c00ef151adca49d [term 1 FOLLOWER]: Committing config change with OpId 1.158: config changed from index -1 to 158, NON_VOTER 90e329b457c24caf93a9f880345eb288 (127.15.255.135) added. New config: { opid_index: 158 OBSOLETE_local: false peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } attrs { replace: true } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: NON_VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: true } } }
I20240502 21:26:13.523700 21054 raft_consensus.cc:2932] T 693a8666bbe54ae1b774f7f4da5ba67e P faadabfb099b4abf90a466395be585b5 [term 1 LEADER]: Committing config change with OpId 1.180: config changed from index 135 to 180, 90e329b457c24caf93a9f880345eb288 (127.15.255.135) changed from NON_VOTER to VOTER. New config: { opid_index: 180 OBSOLETE_local: false peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } attrs { replace: true } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } } }
I20240502 21:26:13.527736 18738 raft_consensus.cc:2932] T 693a8666bbe54ae1b774f7f4da5ba67e P 501e3c0d44fe4d70af9ff926d9622417 [term 1 FOLLOWER]: Committing config change with OpId 1.180: config changed from index 135 to 180, 90e329b457c24caf93a9f880345eb288 (127.15.255.135) changed from NON_VOTER to VOTER. New config: { opid_index: 180 OBSOLETE_local: false peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } attrs { replace: true } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } } }
I20240502 21:26:13.531913 20169 raft_consensus.cc:2932] T 693a8666bbe54ae1b774f7f4da5ba67e P 90e329b457c24caf93a9f880345eb288 [term 1 FOLLOWER]: Committing config change with OpId 1.180: config changed from index 135 to 180, 90e329b457c24caf93a9f880345eb288 (127.15.255.135) changed from NON_VOTER to VOTER. New config: { opid_index: 180 OBSOLETE_local: false peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } attrs { replace: true } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } } }
I20240502 21:26:13.535851 18472 raft_consensus.cc:1261] T 4aff104e3b6b424baa628bbe612b62e8 P faadabfb099b4abf90a466395be585b5 [term 1 FOLLOWER]: Refusing update from remote peer 9be399440ded4e2ebe6b6e0bf486cbb1: Log matching property violated. Preceding OpId in replica: term: 1 index: 157. Preceding OpId from leader: term: 1 index: 158. (index mismatch)
I20240502 21:26:13.536708 19143 raft_consensus.cc:2932] T 693a8666bbe54ae1b774f7f4da5ba67e P 2877108756304e05bd60ba1186b870da [term 1 FOLLOWER]: Committing config change with OpId 1.180: config changed from index 135 to 180, 90e329b457c24caf93a9f880345eb288 (127.15.255.135) changed from NON_VOTER to VOTER. New config: { opid_index: 180 OBSOLETE_local: false peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } attrs { replace: true } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } } }
I20240502 21:26:13.539253 18320 catalog_manager.cc:5484] T 693a8666bbe54ae1b774f7f4da5ba67e P 501e3c0d44fe4d70af9ff926d9622417 reported cstate change: config changed from index 135 to 180, 90e329b457c24caf93a9f880345eb288 (127.15.255.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "faadabfb099b4abf90a466395be585b5" committed_config { opid_index: 180 OBSOLETE_local: false peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } attrs { replace: true } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } } }
I20240502 21:26:13.541309 21100 consensus_queue.cc:1035] T 4aff104e3b6b424baa628bbe612b62e8 P 9be399440ded4e2ebe6b6e0bf486cbb1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 158, Last known committed idx: 157, Time since last communication: 0.000s
I20240502 21:26:13.543453 19010 raft_consensus.cc:2932] T 693a8666bbe54ae1b774f7f4da5ba67e P 9be399440ded4e2ebe6b6e0bf486cbb1 [term 1 FOLLOWER]: Committing config change with OpId 1.180: config changed from index 135 to 180, 90e329b457c24caf93a9f880345eb288 (127.15.255.135) changed from NON_VOTER to VOTER. New config: { opid_index: 180 OBSOLETE_local: false peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } attrs { replace: true } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } } }
I20240502 21:26:13.548643 18472 raft_consensus.cc:2932] T 4aff104e3b6b424baa628bbe612b62e8 P faadabfb099b4abf90a466395be585b5 [term 1 FOLLOWER]: Committing config change with OpId 1.158: config changed from index -1 to 158, NON_VOTER 90e329b457c24caf93a9f880345eb288 (127.15.255.135) added. New config: { opid_index: 158 OBSOLETE_local: false peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } attrs { replace: true } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: NON_VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: true } } }
I20240502 21:26:13.549397 18470 consensus_queue.cc:237] T 693a8666bbe54ae1b774f7f4da5ba67e P faadabfb099b4abf90a466395be585b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 180, Committed index: 180, Last appended: 1.180, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 181 OBSOLETE_local: false peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } }
W20240502 21:26:13.557447 18810 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.133:34217, user_credentials={real_user=slave}} blocked reactor thread for 74277.9us
I20240502 21:26:13.577983 20167 raft_consensus.cc:1261] T 693a8666bbe54ae1b774f7f4da5ba67e P 90e329b457c24caf93a9f880345eb288 [term 1 FOLLOWER]: Refusing update from remote peer faadabfb099b4abf90a466395be585b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 180. Preceding OpId from leader: term: 1 index: 181. (index mismatch)
I20240502 21:26:13.582031 18738 raft_consensus.cc:1261] T 693a8666bbe54ae1b774f7f4da5ba67e P 501e3c0d44fe4d70af9ff926d9622417 [term 1 FOLLOWER]: Refusing update from remote peer faadabfb099b4abf90a466395be585b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 180. Preceding OpId from leader: term: 1 index: 181. (index mismatch)
I20240502 21:26:13.592547 20856 consensus_queue.cc:1035] T 693a8666bbe54ae1b774f7f4da5ba67e P faadabfb099b4abf90a466395be585b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 181, Last known committed idx: 180, Time since last communication: 0.000s
I20240502 21:26:13.596264 18611 raft_consensus.cc:2932] T 693a8666bbe54ae1b774f7f4da5ba67e P 82f50f4b66f3450aa9f58a0b317f5cea [term 1 FOLLOWER]: Committing config change with OpId 1.180: config changed from index 135 to 180, 90e329b457c24caf93a9f880345eb288 (127.15.255.135) changed from NON_VOTER to VOTER. New config: { opid_index: 180 OBSOLETE_local: false peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } attrs { replace: true } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } } }
I20240502 21:26:13.603046 19143 raft_consensus.cc:1261] T 693a8666bbe54ae1b774f7f4da5ba67e P 2877108756304e05bd60ba1186b870da [term 1 FOLLOWER]: Refusing update from remote peer faadabfb099b4abf90a466395be585b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 180. Preceding OpId from leader: term: 1 index: 181. (index mismatch)
I20240502 21:26:13.603878 21055 consensus_queue.cc:1035] T 693a8666bbe54ae1b774f7f4da5ba67e P faadabfb099b4abf90a466395be585b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 181, Last known committed idx: 180, Time since last communication: 0.000s
I20240502 21:26:13.611125 19008 raft_consensus.cc:1261] T 693a8666bbe54ae1b774f7f4da5ba67e P 9be399440ded4e2ebe6b6e0bf486cbb1 [term 1 FOLLOWER]: Refusing update from remote peer faadabfb099b4abf90a466395be585b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 180. Preceding OpId from leader: term: 1 index: 181. (index mismatch)
I20240502 21:26:13.611584 21048 raft_consensus.cc:2932] T 693a8666bbe54ae1b774f7f4da5ba67e P faadabfb099b4abf90a466395be585b5 [term 1 LEADER]: Committing config change with OpId 1.181: config changed from index 180 to 181, VOTER 82f50f4b66f3450aa9f58a0b317f5cea (127.15.255.130) evicted. New config: { opid_index: 181 OBSOLETE_local: false peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } } }
I20240502 21:26:13.620889 20953 consensus_queue.cc:1035] T 693a8666bbe54ae1b774f7f4da5ba67e P faadabfb099b4abf90a466395be585b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 181, Last known committed idx: 180, Time since last communication: 0.000s
I20240502 21:26:13.642791 20953 consensus_queue.cc:1035] T 693a8666bbe54ae1b774f7f4da5ba67e P faadabfb099b4abf90a466395be585b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 181, Last known committed idx: 180, Time since last communication: 0.000s
I20240502 21:26:13.648697 19139 raft_consensus.cc:2932] T 693a8666bbe54ae1b774f7f4da5ba67e P 2877108756304e05bd60ba1186b870da [term 1 FOLLOWER]: Committing config change with OpId 1.181: config changed from index 180 to 181, VOTER 82f50f4b66f3450aa9f58a0b317f5cea (127.15.255.130) evicted. New config: { opid_index: 181 OBSOLETE_local: false peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } } }
I20240502 21:26:13.658056 21115 ts_tablet_manager.cc:902] T 32c9fc5a18c54207881124fda86f42ff P 90e329b457c24caf93a9f880345eb288: Initiating tablet copy from peer faadabfb099b4abf90a466395be585b5 (127.15.255.129:32983)
I20240502 21:26:13.666697 18313 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 693a8666bbe54ae1b774f7f4da5ba67e with cas_config_opid_index 180: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:26:13.671888 18740 raft_consensus.cc:2932] T 693a8666bbe54ae1b774f7f4da5ba67e P 501e3c0d44fe4d70af9ff926d9622417 [term 1 FOLLOWER]: Committing config change with OpId 1.181: config changed from index 180 to 181, VOTER 82f50f4b66f3450aa9f58a0b317f5cea (127.15.255.130) evicted. New config: { opid_index: 181 OBSOLETE_local: false peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } } }
I20240502 21:26:13.673455 18327 catalog_manager.cc:5484] T 693a8666bbe54ae1b774f7f4da5ba67e P faadabfb099b4abf90a466395be585b5 reported cstate change: config changed from index 180 to 181, VOTER 82f50f4b66f3450aa9f58a0b317f5cea (127.15.255.130) evicted. New cstate: current_term: 1 leader_uuid: "faadabfb099b4abf90a466395be585b5" committed_config { opid_index: 181 OBSOLETE_local: false peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 21:26:13.709988 20169 raft_consensus.cc:2932] T 693a8666bbe54ae1b774f7f4da5ba67e P 90e329b457c24caf93a9f880345eb288 [term 1 FOLLOWER]: Committing config change with OpId 1.181: config changed from index 180 to 181, VOTER 82f50f4b66f3450aa9f58a0b317f5cea (127.15.255.130) evicted. New config: { opid_index: 181 OBSOLETE_local: false peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } } }
I20240502 21:26:13.718143 19005 raft_consensus.cc:2932] T 693a8666bbe54ae1b774f7f4da5ba67e P 9be399440ded4e2ebe6b6e0bf486cbb1 [term 1 FOLLOWER]: Committing config change with OpId 1.181: config changed from index 180 to 181, VOTER 82f50f4b66f3450aa9f58a0b317f5cea (127.15.255.130) evicted. New config: { opid_index: 181 OBSOLETE_local: false peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } } }
I20240502 21:26:13.750728 18590 tablet_service.cc:1514] Processing DeleteTablet for tablet 693a8666bbe54ae1b774f7f4da5ba67e with delete_type TABLET_DATA_TOMBSTONED (TS 82f50f4b66f3450aa9f58a0b317f5cea not found in new config with opid_index 181) from {username='slave'} at 127.0.0.1:33750
I20240502 21:26:13.778779 21117 tablet_replica.cc:330] T 693a8666bbe54ae1b774f7f4da5ba67e P 82f50f4b66f3450aa9f58a0b317f5cea: stopping tablet replica
I20240502 21:26:13.779168 21117 raft_consensus.cc:2229] T 693a8666bbe54ae1b774f7f4da5ba67e P 82f50f4b66f3450aa9f58a0b317f5cea [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:26:13.782927 21117 raft_consensus.cc:2258] T 693a8666bbe54ae1b774f7f4da5ba67e P 82f50f4b66f3450aa9f58a0b317f5cea [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:13.784065 21117 ts_tablet_manager.cc:1875] T 693a8666bbe54ae1b774f7f4da5ba67e P 82f50f4b66f3450aa9f58a0b317f5cea: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:26:13.814327 21117 ts_tablet_manager.cc:1888] T 693a8666bbe54ae1b774f7f4da5ba67e P 82f50f4b66f3450aa9f58a0b317f5cea: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.180
I20240502 21:26:13.816560 21117 log.cc:1196] T 693a8666bbe54ae1b774f7f4da5ba67e P 82f50f4b66f3450aa9f58a0b317f5cea: Deleting WAL directory at /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-1/wal/wals/693a8666bbe54ae1b774f7f4da5ba67e
I20240502 21:26:13.818461 18315 catalog_manager.cc:4830] TS 82f50f4b66f3450aa9f58a0b317f5cea (127.15.255.130:45139): tablet 693a8666bbe54ae1b774f7f4da5ba67e (table rebalance_test_table_2 [id=59b7e3f7d6ce4a008110576d81d3cf39]) successfully deleted
W20240502 21:26:13.870055 18405 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.133:34217, user_credentials={real_user=slave}} blocked reactor thread for 67296.8us
W20240502 21:26:13.870404 18406 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.132:43607, user_credentials={real_user=slave}} blocked reactor thread for 69220.1us
W20240502 21:26:13.872690 18407 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.134:43635, user_credentials={real_user=slave}} blocked reactor thread for 62919us
I20240502 21:26:14.193105 21125 ts_tablet_manager.cc:902] T 4aff104e3b6b424baa628bbe612b62e8 P 90e329b457c24caf93a9f880345eb288: Initiating tablet copy from peer 9be399440ded4e2ebe6b6e0bf486cbb1 (127.15.255.133:34217)
W20240502 21:26:14.195734 19077 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.133:34217, user_credentials={real_user=slave}} blocked reactor thread for 157870us
W20240502 21:26:14.205016 18945 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.132:43607, user_credentials={real_user=slave}} blocked reactor thread for 63759.6us
I20240502 21:26:14.236050 18878 consensus_queue.cc:237] T 51edafdf2a9348548a494119668bfd31 P c75831edb3b44dd59c00ef151adca49d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 191, Committed index: 191, Last appended: 1.191, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 192 OBSOLETE_local: false peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } attrs { replace: true } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: NON_VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: true } }
I20240502 21:26:14.241433 19007 raft_consensus.cc:1261] T 51edafdf2a9348548a494119668bfd31 P 9be399440ded4e2ebe6b6e0bf486cbb1 [term 1 FOLLOWER]: Refusing update from remote peer c75831edb3b44dd59c00ef151adca49d: Log matching property violated. Preceding OpId in replica: term: 1 index: 191. Preceding OpId from leader: term: 1 index: 193. (index mismatch)
I20240502 21:26:14.242789 18470 raft_consensus.cc:1261] T 51edafdf2a9348548a494119668bfd31 P faadabfb099b4abf90a466395be585b5 [term 1 FOLLOWER]: Refusing update from remote peer c75831edb3b44dd59c00ef151adca49d: Log matching property violated. Preceding OpId in replica: term: 1 index: 191. Preceding OpId from leader: term: 1 index: 193. (index mismatch)
I20240502 21:26:14.243655 20800 consensus_queue.cc:1035] T 51edafdf2a9348548a494119668bfd31 P c75831edb3b44dd59c00ef151adca49d [LEADER]: Connected to new peer: Peer: permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 192, Last known committed idx: 191, Time since last communication: 0.000s
I20240502 21:26:14.246671 20800 consensus_queue.cc:1035] T 51edafdf2a9348548a494119668bfd31 P c75831edb3b44dd59c00ef151adca49d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 192, Last known committed idx: 191, Time since last communication: 0.000s
I20240502 21:26:14.250403 20959 raft_consensus.cc:2932] T 51edafdf2a9348548a494119668bfd31 P c75831edb3b44dd59c00ef151adca49d [term 1 LEADER]: Committing config change with OpId 1.192: config changed from index -1 to 192, NON_VOTER 90e329b457c24caf93a9f880345eb288 (127.15.255.135) added. New config: { opid_index: 192 OBSOLETE_local: false peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } attrs { replace: true } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: NON_VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: true } } }
I20240502 21:26:14.253633 18474 raft_consensus.cc:2932] T 51edafdf2a9348548a494119668bfd31 P faadabfb099b4abf90a466395be585b5 [term 1 FOLLOWER]: Committing config change with OpId 1.192: config changed from index -1 to 192, NON_VOTER 90e329b457c24caf93a9f880345eb288 (127.15.255.135) added. New config: { opid_index: 192 OBSOLETE_local: false peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } attrs { replace: true } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: NON_VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: true } } }
W20240502 21:26:14.257644 18811 consensus_peers.cc:485] T 51edafdf2a9348548a494119668bfd31 P c75831edb3b44dd59c00ef151adca49d -> Peer 90e329b457c24caf93a9f880345eb288 (127.15.255.135:43395): Couldn't send request to peer 90e329b457c24caf93a9f880345eb288. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 51edafdf2a9348548a494119668bfd31. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:26:14.260502 19007 raft_consensus.cc:2932] T 51edafdf2a9348548a494119668bfd31 P 9be399440ded4e2ebe6b6e0bf486cbb1 [term 1 FOLLOWER]: Committing config change with OpId 1.192: config changed from index -1 to 192, NON_VOTER 90e329b457c24caf93a9f880345eb288 (127.15.255.135) added. New config: { opid_index: 192 OBSOLETE_local: false peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } attrs { replace: true } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: NON_VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: true } } }
I20240502 21:26:14.264052 18320 catalog_manager.cc:5484] T 51edafdf2a9348548a494119668bfd31 P faadabfb099b4abf90a466395be585b5 reported cstate change: config changed from index -1 to 192, NON_VOTER 90e329b457c24caf93a9f880345eb288 (127.15.255.135) added. New cstate: current_term: 1 leader_uuid: "c75831edb3b44dd59c00ef151adca49d" committed_config { opid_index: 192 OBSOLETE_local: false peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } attrs { replace: true } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: NON_VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: true } } }
I20240502 21:26:14.333865 18611 raft_consensus.cc:1261] T 51edafdf2a9348548a494119668bfd31 P 82f50f4b66f3450aa9f58a0b317f5cea [term 1 FOLLOWER]: Refusing update from remote peer c75831edb3b44dd59c00ef151adca49d: Log matching property violated. Preceding OpId in replica: term: 1 index: 191. Preceding OpId from leader: term: 1 index: 193. (index mismatch)
I20240502 21:26:14.358342 19140 raft_consensus.cc:1261] T 51edafdf2a9348548a494119668bfd31 P 2877108756304e05bd60ba1186b870da [term 1 FOLLOWER]: Refusing update from remote peer c75831edb3b44dd59c00ef151adca49d: Log matching property violated. Preceding OpId in replica: term: 1 index: 191. Preceding OpId from leader: term: 1 index: 193. (index mismatch)
I20240502 21:26:14.424525 20963 consensus_queue.cc:1035] T 51edafdf2a9348548a494119668bfd31 P c75831edb3b44dd59c00ef151adca49d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 192, Last known committed idx: 191, Time since last communication: 0.000s
I20240502 21:26:14.588248 19140 raft_consensus.cc:2932] T 51edafdf2a9348548a494119668bfd31 P 2877108756304e05bd60ba1186b870da [term 1 FOLLOWER]: Committing config change with OpId 1.192: config changed from index -1 to 192, NON_VOTER 90e329b457c24caf93a9f880345eb288 (127.15.255.135) added. New config: { opid_index: 192 OBSOLETE_local: false peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } attrs { replace: true } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: NON_VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: true } } }
I20240502 21:26:14.599658 20963 consensus_queue.cc:1035] T 51edafdf2a9348548a494119668bfd31 P c75831edb3b44dd59c00ef151adca49d [LEADER]: Connected to new peer: Peer: permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 192, Last known committed idx: 191, Time since last communication: 0.000s
W20240502 21:26:14.624210 18945 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.135:43395, user_credentials={real_user=slave}} blocked reactor thread for 44557.2us
W20240502 21:26:14.656409 18812 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.134:43635, user_credentials={real_user=slave}} blocked reactor thread for 56818us
I20240502 21:26:14.687880 18611 raft_consensus.cc:2932] T 51edafdf2a9348548a494119668bfd31 P 82f50f4b66f3450aa9f58a0b317f5cea [term 1 FOLLOWER]: Committing config change with OpId 1.192: config changed from index -1 to 192, NON_VOTER 90e329b457c24caf93a9f880345eb288 (127.15.255.135) added. New config: { opid_index: 192 OBSOLETE_local: false peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } attrs { replace: true } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: NON_VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: true } } }
I20240502 21:26:14.735486 19139 consensus_queue.cc:237] T b0852f54089944ce935f8ee4cbf2bc5a P 2877108756304e05bd60ba1186b870da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 146, Committed index: 146, Last appended: 1.147, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 148 OBSOLETE_local: false peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } attrs { replace: true } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: NON_VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: true } }
I20240502 21:26:14.761407 18611 raft_consensus.cc:1261] T b0852f54089944ce935f8ee4cbf2bc5a P 82f50f4b66f3450aa9f58a0b317f5cea [term 1 FOLLOWER]: Refusing update from remote peer 2877108756304e05bd60ba1186b870da: Log matching property violated. Preceding OpId in replica: term: 1 index: 147. Preceding OpId from leader: term: 1 index: 148. (index mismatch)
I20240502 21:26:14.762668 21007 consensus_queue.cc:1035] T b0852f54089944ce935f8ee4cbf2bc5a P 2877108756304e05bd60ba1186b870da [LEADER]: Connected to new peer: Peer: permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 148, Last known committed idx: 146, Time since last communication: 0.000s
I20240502 21:26:14.770342 18470 raft_consensus.cc:1261] T b0852f54089944ce935f8ee4cbf2bc5a P faadabfb099b4abf90a466395be585b5 [term 1 FOLLOWER]: Refusing update from remote peer 2877108756304e05bd60ba1186b870da: Log matching property violated. Preceding OpId in replica: term: 1 index: 147. Preceding OpId from leader: term: 1 index: 148. (index mismatch)
I20240502 21:26:14.771392 21007 consensus_queue.cc:1035] T b0852f54089944ce935f8ee4cbf2bc5a P 2877108756304e05bd60ba1186b870da [LEADER]: Connected to new peer: Peer: permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 148, Last known committed idx: 146, Time since last communication: 0.000s
I20240502 21:26:14.781888 18470 raft_consensus.cc:2932] T b0852f54089944ce935f8ee4cbf2bc5a P faadabfb099b4abf90a466395be585b5 [term 1 FOLLOWER]: Committing config change with OpId 1.148: config changed from index -1 to 148, NON_VOTER 90e329b457c24caf93a9f880345eb288 (127.15.255.135) added. New config: { opid_index: 148 OBSOLETE_local: false peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } attrs { replace: true } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: NON_VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: true } } }
I20240502 21:26:14.835181 18743 raft_consensus.cc:1261] T b0852f54089944ce935f8ee4cbf2bc5a P 501e3c0d44fe4d70af9ff926d9622417 [term 1 FOLLOWER]: Refusing update from remote peer 2877108756304e05bd60ba1186b870da: Log matching property violated. Preceding OpId in replica: term: 1 index: 147. Preceding OpId from leader: term: 1 index: 148. (index mismatch)
I20240502 21:26:14.847064 21127 consensus_queue.cc:1035] T b0852f54089944ce935f8ee4cbf2bc5a P 2877108756304e05bd60ba1186b870da [LEADER]: Connected to new peer: Peer: permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 148, Last known committed idx: 146, Time since last communication: 0.000s
I20240502 21:26:14.848950 18743 raft_consensus.cc:2932] T b0852f54089944ce935f8ee4cbf2bc5a P 501e3c0d44fe4d70af9ff926d9622417 [term 1 FOLLOWER]: Committing config change with OpId 1.148: config changed from index -1 to 148, NON_VOTER 90e329b457c24caf93a9f880345eb288 (127.15.255.135) added. New config: { opid_index: 148 OBSOLETE_local: false peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } attrs { replace: true } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: NON_VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: true } } }
I20240502 21:26:14.856657 18321 catalog_manager.cc:5484] T b0852f54089944ce935f8ee4cbf2bc5a P 501e3c0d44fe4d70af9ff926d9622417 reported cstate change: config changed from index -1 to 148, NON_VOTER 90e329b457c24caf93a9f880345eb288 (127.15.255.135) added. New cstate: current_term: 1 leader_uuid: "2877108756304e05bd60ba1186b870da" committed_config { opid_index: 148 OBSOLETE_local: false peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } attrs { replace: true } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: NON_VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: true } } }
W20240502 21:26:14.873311 18812 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.130:45139, user_credentials={real_user=slave}} blocked reactor thread for 44154.4us
W20240502 21:26:14.878858 21125 thread.cc:636] tablet-download-pool-4aff104e3b6b424baa628bbe612b62e8 [worker] (thread pool) Time spent creating pthread: real 0.631s user 0.061s sys 0.124s
W20240502 21:26:14.879467 21125 thread.cc:602] tablet-download-pool-4aff104e3b6b424baa628bbe612b62e8 [worker] (thread pool) Time spent starting thread: real 0.632s user 0.061s sys 0.124s
I20240502 21:26:14.880012 21125 tablet_copy_client.cc:308] T 4aff104e3b6b424baa628bbe612b62e8 P 90e329b457c24caf93a9f880345eb288: tablet copy: Beginning tablet copy session from remote peer at address 127.15.255.133:34217
I20240502 21:26:14.886965 21163 ts_tablet_manager.cc:902] T 51edafdf2a9348548a494119668bfd31 P 90e329b457c24caf93a9f880345eb288: Initiating tablet copy from peer c75831edb3b44dd59c00ef151adca49d (127.15.255.132:43607)
I20240502 21:26:14.909862 19008 raft_consensus.cc:1261] T b0852f54089944ce935f8ee4cbf2bc5a P 9be399440ded4e2ebe6b6e0bf486cbb1 [term 1 FOLLOWER]: Refusing update from remote peer 2877108756304e05bd60ba1186b870da: Log matching property violated. Preceding OpId in replica: term: 1 index: 147. Preceding OpId from leader: term: 1 index: 148. (index mismatch)
I20240502 21:26:14.909335 21128 raft_consensus.cc:2932] T b0852f54089944ce935f8ee4cbf2bc5a P 2877108756304e05bd60ba1186b870da [term 1 LEADER]: Committing config change with OpId 1.148: config changed from index -1 to 148, NON_VOTER 90e329b457c24caf93a9f880345eb288 (127.15.255.135) added. New config: { opid_index: 148 OBSOLETE_local: false peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } attrs { replace: true } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: NON_VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: true } } }
I20240502 21:26:14.909648 18608 raft_consensus.cc:2932] T b0852f54089944ce935f8ee4cbf2bc5a P 82f50f4b66f3450aa9f58a0b317f5cea [term 1 FOLLOWER]: Committing config change with OpId 1.148: config changed from index -1 to 148, NON_VOTER 90e329b457c24caf93a9f880345eb288 (127.15.255.135) added. New config: { opid_index: 148 OBSOLETE_local: false peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } attrs { replace: true } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: NON_VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: true } } }
I20240502 21:26:14.910548 21133 consensus_queue.cc:1035] T b0852f54089944ce935f8ee4cbf2bc5a P 2877108756304e05bd60ba1186b870da [LEADER]: Connected to new peer: Peer: permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 148, Last known committed idx: 147, Time since last communication: 0.000s
I20240502 21:26:14.913110 19008 raft_consensus.cc:2932] T b0852f54089944ce935f8ee4cbf2bc5a P 9be399440ded4e2ebe6b6e0bf486cbb1 [term 1 FOLLOWER]: Committing config change with OpId 1.148: config changed from index -1 to 148, NON_VOTER 90e329b457c24caf93a9f880345eb288 (127.15.255.135) added. New config: { opid_index: 148 OBSOLETE_local: false peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } attrs { replace: true } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: NON_VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: true } } }
I20240502 21:26:14.964643 19033 tablet_copy_service.cc:140] P 9be399440ded4e2ebe6b6e0bf486cbb1: Received BeginTabletCopySession request for tablet 4aff104e3b6b424baa628bbe612b62e8 from peer 90e329b457c24caf93a9f880345eb288 ({username='slave'} at 127.15.255.135:33929)
I20240502 21:26:14.964831 19033 tablet_copy_service.cc:161] P 9be399440ded4e2ebe6b6e0bf486cbb1: Beginning new tablet copy session on tablet 4aff104e3b6b424baa628bbe612b62e8 from peer 90e329b457c24caf93a9f880345eb288 at {username='slave'} at 127.15.255.135:33929: session id = 90e329b457c24caf93a9f880345eb288-4aff104e3b6b424baa628bbe612b62e8
I20240502 21:26:14.967581 19033 tablet_copy_source_session.cc:205] T 4aff104e3b6b424baa628bbe612b62e8 P 9be399440ded4e2ebe6b6e0bf486cbb1: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:26:14.973332 21125 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4aff104e3b6b424baa628bbe612b62e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:14.979970 21125 tablet_copy_client.cc:783] T 4aff104e3b6b424baa628bbe612b62e8 P 90e329b457c24caf93a9f880345eb288: tablet copy: Starting download of 0 data blocks...
I20240502 21:26:14.980321 21125 tablet_copy_client.cc:647] T 4aff104e3b6b424baa628bbe612b62e8 P 90e329b457c24caf93a9f880345eb288: tablet copy: Starting download of 1 WAL segments...
W20240502 21:26:15.000176 21115 thread.cc:636] tablet-download-pool-32c9fc5a18c54207881124fda86f42ff [worker] (thread pool) Time spent creating pthread: real 0.910s user 0.101s sys 0.175s
W20240502 21:26:15.000814 21115 thread.cc:602] tablet-download-pool-32c9fc5a18c54207881124fda86f42ff [worker] (thread pool) Time spent starting thread: real 0.911s user 0.101s sys 0.176s
I20240502 21:26:15.001610 21115 tablet_copy_client.cc:308] T 32c9fc5a18c54207881124fda86f42ff P 90e329b457c24caf93a9f880345eb288: tablet copy: Beginning tablet copy session from remote peer at address 127.15.255.129:32983
I20240502 21:26:15.003289 18496 tablet_copy_service.cc:140] P faadabfb099b4abf90a466395be585b5: Received BeginTabletCopySession request for tablet 32c9fc5a18c54207881124fda86f42ff from peer 90e329b457c24caf93a9f880345eb288 ({username='slave'} at 127.15.255.135:49783)
I20240502 21:26:15.003492 18496 tablet_copy_service.cc:161] P faadabfb099b4abf90a466395be585b5: Beginning new tablet copy session on tablet 32c9fc5a18c54207881124fda86f42ff from peer 90e329b457c24caf93a9f880345eb288 at {username='slave'} at 127.15.255.135:49783: session id = 90e329b457c24caf93a9f880345eb288-32c9fc5a18c54207881124fda86f42ff
I20240502 21:26:15.027969 18496 tablet_copy_source_session.cc:205] T 32c9fc5a18c54207881124fda86f42ff P faadabfb099b4abf90a466395be585b5: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:26:15.049502 21115 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32c9fc5a18c54207881124fda86f42ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:15.056629 21115 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.StartTabletCopy from 127.15.255.129:52763 (request call id 236) took 1410 ms. Trace:
I20240502 21:26:15.056948 21115 rpcz_store.cc:276] 0502 21:26:13.646449 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 21:26:13.646502 (+ 53us) service_pool.cc:225] Handling call
0502 21:26:13.658359 (+ 11857us) ts_tablet_manager.cc:903] T 32c9fc5a18c54207881124fda86f42ff P 90e329b457c24caf93a9f880345eb288: Initiating tablet copy from peer faadabfb099b4abf90a466395be585b5 (127.15.255.129:32983)
0502 21:26:15.048938 (+1390579us) tablet_copy_client.cc:353] Tablet copy session begun
0502 21:26:15.053667 (+ 4729us) tablet_metadata.cc:641] Metadata flushed
0502 21:26:15.053801 (+ 134us) tablet_copy_client.cc:482] Wrote new tablet metadata
0502 21:26:15.056338 (+ 2537us) tablet_copy_client.cc:403] Wrote new consensus metadata
0502 21:26:15.056475 (+ 137us) tablet_replica.cc:205] Creating consensus instance
0502 21:26:15.056570 (+ 95us) ts_tablet_manager.cc:958] Replica registered.
0502 21:26:15.056592 (+ 22us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":94,"tablet-copy.queue_time_us":10852,"thread_start_us":1142521,"threads_started":5}
I20240502 21:26:15.058226 21115 tablet_copy_client.cc:783] T 32c9fc5a18c54207881124fda86f42ff P 90e329b457c24caf93a9f880345eb288: tablet copy: Starting download of 0 data blocks...
I20240502 21:26:15.058533 21115 tablet_copy_client.cc:647] T 32c9fc5a18c54207881124fda86f42ff P 90e329b457c24caf93a9f880345eb288: tablet copy: Starting download of 1 WAL segments...
I20240502 21:26:15.070164 21115 tablet_copy_client.cc:523] T 32c9fc5a18c54207881124fda86f42ff P 90e329b457c24caf93a9f880345eb288: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:26:15.073907 21115 tablet_bootstrap.cc:492] T 32c9fc5a18c54207881124fda86f42ff P 90e329b457c24caf93a9f880345eb288: Bootstrap starting.
W20240502 21:26:15.092250 19078 consensus_peers.cc:485] T b0852f54089944ce935f8ee4cbf2bc5a P 2877108756304e05bd60ba1186b870da -> Peer 90e329b457c24caf93a9f880345eb288 (127.15.255.135:43395): Couldn't send request to peer 90e329b457c24caf93a9f880345eb288. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b0852f54089944ce935f8ee4cbf2bc5a. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:26:15.120168 21125 tablet_copy_client.cc:523] T 4aff104e3b6b424baa628bbe612b62e8 P 90e329b457c24caf93a9f880345eb288: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:26:15.123518 21125 tablet_bootstrap.cc:492] T 4aff104e3b6b424baa628bbe612b62e8 P 90e329b457c24caf93a9f880345eb288: Bootstrap starting.
I20240502 21:26:15.159209 21175 ts_tablet_manager.cc:902] T b0852f54089944ce935f8ee4cbf2bc5a P 90e329b457c24caf93a9f880345eb288: Initiating tablet copy from peer 2877108756304e05bd60ba1186b870da (127.15.255.134:43635)
W20240502 21:26:15.236583 18945 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.132:43607, user_credentials={real_user=slave}} blocked reactor thread for 102188us
W20240502 21:26:15.429116 21163 thread.cc:636] tablet-download-pool-51edafdf2a9348548a494119668bfd31 [worker] (thread pool) Time spent creating pthread: real 0.542s user 0.026s sys 0.073s
W20240502 21:26:15.509588 18945 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.132:43607, user_credentials={real_user=slave}} blocked reactor thread for 43686.9us
W20240502 21:26:15.536020 21163 thread.cc:602] tablet-download-pool-51edafdf2a9348548a494119668bfd31 [worker] (thread pool) Time spent starting thread: real 0.543s user 0.026s sys 0.073s
I20240502 21:26:15.743468 21175 tablet_copy_client.cc:308] T b0852f54089944ce935f8ee4cbf2bc5a P 90e329b457c24caf93a9f880345eb288: tablet copy: Beginning tablet copy session from remote peer at address 127.15.255.134:43635
W20240502 21:26:15.757728 18545 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.132:43607, user_credentials={real_user=slave}} blocked reactor thread for 212941us
I20240502 21:26:15.780799 19166 tablet_copy_service.cc:140] P 2877108756304e05bd60ba1186b870da: Received BeginTabletCopySession request for tablet b0852f54089944ce935f8ee4cbf2bc5a from peer 90e329b457c24caf93a9f880345eb288 ({username='slave'} at 127.15.255.135:43445)
I20240502 21:26:15.781033 19166 tablet_copy_service.cc:161] P 2877108756304e05bd60ba1186b870da: Beginning new tablet copy session on tablet b0852f54089944ce935f8ee4cbf2bc5a from peer 90e329b457c24caf93a9f880345eb288 at {username='slave'} at 127.15.255.135:43445: session id = 90e329b457c24caf93a9f880345eb288-b0852f54089944ce935f8ee4cbf2bc5a
I20240502 21:26:15.787339 19166 tablet_copy_source_session.cc:205] T b0852f54089944ce935f8ee4cbf2bc5a P 2877108756304e05bd60ba1186b870da: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:26:15.794656 21175 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0852f54089944ce935f8ee4cbf2bc5a. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 21:26:15.920495 18544 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.129:32983, user_credentials={real_user=slave}} blocked reactor thread for 49356.4us
I20240502 21:26:16.286715 21175 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.StartTabletCopy from 127.15.255.134:43957 (request call id 1) took 1130 ms. Trace:
I20240502 21:26:16.300191 21175 rpcz_store.cc:276] 0502 21:26:15.155833 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 21:26:15.155937 (+ 104us) service_pool.cc:225] Handling call
0502 21:26:15.159848 (+ 3911us) ts_tablet_manager.cc:903] T b0852f54089944ce935f8ee4cbf2bc5a P 90e329b457c24caf93a9f880345eb288: Initiating tablet copy from peer 2877108756304e05bd60ba1186b870da (127.15.255.134:43635)
0502 21:26:15.794084 (+634236us) tablet_copy_client.cc:353] Tablet copy session begun
0502 21:26:15.811931 (+ 17847us) tablet_metadata.cc:641] Metadata flushed
0502 21:26:15.812053 (+ 122us) tablet_copy_client.cc:482] Wrote new tablet metadata
0502 21:26:16.285902 (+473849us) tablet_copy_client.cc:403] Wrote new consensus metadata
0502 21:26:16.286048 (+ 146us) tablet_replica.cc:205] Creating consensus instance
0502 21:26:16.286137 (+ 89us) ts_tablet_manager.cc:958] Replica registered.
0502 21:26:16.286660 (+ 523us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":1616,"tablet-copy.queue_time_us":2955,"thread_start_us":588062,"threads_started":5}
I20240502 21:26:16.313282 21163 tablet_copy_client.cc:308] T 51edafdf2a9348548a494119668bfd31 P 90e329b457c24caf93a9f880345eb288: tablet copy: Beginning tablet copy session from remote peer at address 127.15.255.132:43607
I20240502 21:26:16.314217 18898 tablet_copy_service.cc:140] P c75831edb3b44dd59c00ef151adca49d: Received BeginTabletCopySession request for tablet 51edafdf2a9348548a494119668bfd31 from peer 90e329b457c24caf93a9f880345eb288 ({username='slave'} at 127.15.255.135:46619)
I20240502 21:26:16.314369 18898 tablet_copy_service.cc:161] P c75831edb3b44dd59c00ef151adca49d: Beginning new tablet copy session on tablet 51edafdf2a9348548a494119668bfd31 from peer 90e329b457c24caf93a9f880345eb288 at {username='slave'} at 127.15.255.135:46619: session id = 90e329b457c24caf93a9f880345eb288-51edafdf2a9348548a494119668bfd31
I20240502 21:26:16.317263 18898 tablet_copy_source_session.cc:205] T 51edafdf2a9348548a494119668bfd31 P c75831edb3b44dd59c00ef151adca49d: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:26:16.322532 21163 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51edafdf2a9348548a494119668bfd31. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:16.358184 21175 tablet_copy_client.cc:783] T b0852f54089944ce935f8ee4cbf2bc5a P 90e329b457c24caf93a9f880345eb288: tablet copy: Starting download of 0 data blocks...
I20240502 21:26:16.358780 21175 tablet_copy_client.cc:647] T b0852f54089944ce935f8ee4cbf2bc5a P 90e329b457c24caf93a9f880345eb288: tablet copy: Starting download of 1 WAL segments...
I20240502 21:26:16.399232 21163 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.StartTabletCopy from 127.15.255.132:37735 (request call id 71) took 1537 ms. Trace:
I20240502 21:26:16.399472 21163 rpcz_store.cc:276] 0502 21:26:14.862096 (+ 0us) service_pool.cc:168] Inserting onto call queue
0502 21:26:14.862180 (+ 84us) service_pool.cc:225] Handling call
0502 21:26:14.887205 (+ 25025us) ts_tablet_manager.cc:903] T 51edafdf2a9348548a494119668bfd31 P 90e329b457c24caf93a9f880345eb288: Initiating tablet copy from peer c75831edb3b44dd59c00ef151adca49d (127.15.255.132:43607)
0502 21:26:16.322075 (+1434870us) tablet_copy_client.cc:353] Tablet copy session begun
0502 21:26:16.325688 (+ 3613us) tablet_metadata.cc:641] Metadata flushed
0502 21:26:16.325800 (+ 112us) tablet_copy_client.cc:482] Wrote new tablet metadata
0502 21:26:16.398935 (+ 73135us) tablet_copy_client.cc:403] Wrote new consensus metadata
0502 21:26:16.399072 (+ 137us) tablet_replica.cc:205] Creating consensus instance
0502 21:26:16.399159 (+ 87us) ts_tablet_manager.cc:958] Replica registered.
0502 21:26:16.399181 (+ 22us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":87,"tablet-copy.queue_time_us":24505,"thread_start_us":1450407,"threads_started":5}
I20240502 21:26:16.408315 21175 tablet_copy_client.cc:523] T b0852f54089944ce935f8ee4cbf2bc5a P 90e329b457c24caf93a9f880345eb288: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:26:16.449335 21163 tablet_copy_client.cc:783] T 51edafdf2a9348548a494119668bfd31 P 90e329b457c24caf93a9f880345eb288: tablet copy: Starting download of 0 data blocks...
I20240502 21:26:16.459470 21163 tablet_copy_client.cc:647] T 51edafdf2a9348548a494119668bfd31 P 90e329b457c24caf93a9f880345eb288: tablet copy: Starting download of 1 WAL segments...
I20240502 21:26:16.474978 21163 tablet_copy_client.cc:523] T 51edafdf2a9348548a494119668bfd31 P 90e329b457c24caf93a9f880345eb288: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:26:16.453507 21175 tablet_bootstrap.cc:492] T b0852f54089944ce935f8ee4cbf2bc5a P 90e329b457c24caf93a9f880345eb288: Bootstrap starting.
W20240502 21:26:16.503341 18944 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.129:32983, user_credentials={real_user=slave}} blocked reactor thread for 80366.2us
W20240502 21:26:16.506381 18945 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.132:43607, user_credentials={real_user=slave}} blocked reactor thread for 60613.9us
W20240502 21:26:16.513262 18943 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.131:42501, user_credentials={real_user=slave}} blocked reactor thread for 61704.7us
I20240502 21:26:16.533008 21163 tablet_bootstrap.cc:492] T 51edafdf2a9348548a494119668bfd31 P 90e329b457c24caf93a9f880345eb288: Bootstrap starting.
W20240502 21:26:16.734086 18812 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.130:45139, user_credentials={real_user=slave}} blocked reactor thread for 58330.4us
W20240502 21:26:16.989837 18812 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.130:45139, user_credentials={real_user=slave}} blocked reactor thread for 47342.9us
W20240502 21:26:17.128975 18945 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.132:43607, user_credentials={real_user=slave}} blocked reactor thread for 54507us
W20240502 21:26:17.418138 18545 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.132:43607, user_credentials={real_user=slave}} blocked reactor thread for 364408us
W20240502 21:26:17.434010 18812 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.134:43635, user_credentials={real_user=slave}} blocked reactor thread for 50048us
W20240502 21:26:17.475732 18406 consensus_peers.cc:485] T 32c9fc5a18c54207881124fda86f42ff P faadabfb099b4abf90a466395be585b5 -> Peer 90e329b457c24caf93a9f880345eb288 (127.15.255.135:43395): Couldn't send request to peer 90e329b457c24caf93a9f880345eb288. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 6: this message will repeat every 5th retry.
W20240502 21:26:17.478056 18945 consensus_peers.cc:485] T 4aff104e3b6b424baa628bbe612b62e8 P 9be399440ded4e2ebe6b6e0bf486cbb1 -> Peer 90e329b457c24caf93a9f880345eb288 (127.15.255.135:43395): Couldn't send request to peer 90e329b457c24caf93a9f880345eb288. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 6: this message will repeat every 5th retry.
I20240502 21:26:17.525712 21125 tablet_bootstrap.cc:492] T 4aff104e3b6b424baa628bbe612b62e8 P 90e329b457c24caf93a9f880345eb288: Bootstrap replayed 1/1 log segments. Stats: ops{read=164 overwritten=0 applied=164 ignored=0} inserts{seen=795 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:26:17.626873 21125 tablet_bootstrap.cc:492] T 4aff104e3b6b424baa628bbe612b62e8 P 90e329b457c24caf93a9f880345eb288: Bootstrap complete.
I20240502 21:26:17.643407 21125 ts_tablet_manager.cc:1367] T 4aff104e3b6b424baa628bbe612b62e8 P 90e329b457c24caf93a9f880345eb288: Time spent bootstrapping tablet: real 2.504s user 0.231s sys 0.027s
I20240502 21:26:17.649518 21125 raft_consensus.cc:348] T 4aff104e3b6b424baa628bbe612b62e8 P 90e329b457c24caf93a9f880345eb288 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 158 OBSOLETE_local: false peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } attrs { replace: true } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: NON_VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: true } }
I20240502 21:26:17.652098 21125 raft_consensus.cc:729] T 4aff104e3b6b424baa628bbe612b62e8 P 90e329b457c24caf93a9f880345eb288 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 90e329b457c24caf93a9f880345eb288, State: Initialized, Role: LEARNER
I20240502 21:26:17.652410 21125 consensus_queue.cc:260] T 4aff104e3b6b424baa628bbe612b62e8 P 90e329b457c24caf93a9f880345eb288 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 164, Last appended: 1.164, Last appended by leader: 164, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 158 OBSOLETE_local: false peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } attrs { replace: true } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: NON_VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: true } }
W20240502 21:26:17.758670 18812 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.130:45139, user_credentials={real_user=slave}} blocked reactor thread for 72774.1us
W20240502 21:26:17.798000 18407 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.130:45139, user_credentials={real_user=slave}} blocked reactor thread for 166981us
I20240502 21:26:18.051839 21125 ts_tablet_manager.cc:1398] T 4aff104e3b6b424baa628bbe612b62e8 P 90e329b457c24caf93a9f880345eb288: Time spent starting tablet: real 0.408s user 0.028s sys 0.039s
I20240502 21:26:18.056337 19033 tablet_copy_service.cc:342] P 9be399440ded4e2ebe6b6e0bf486cbb1: Request end of tablet copy session 90e329b457c24caf93a9f880345eb288-4aff104e3b6b424baa628bbe612b62e8 received from {username='slave'} at 127.15.255.135:33929
I20240502 21:26:18.056489 19033 tablet_copy_service.cc:433] P 9be399440ded4e2ebe6b6e0bf486cbb1: ending tablet copy session 90e329b457c24caf93a9f880345eb288-4aff104e3b6b424baa628bbe612b62e8 on tablet 4aff104e3b6b424baa628bbe612b62e8 with peer 90e329b457c24caf93a9f880345eb288
I20240502 21:26:18.315755 20169 raft_consensus.cc:1203] T 4aff104e3b6b424baa628bbe612b62e8 P 90e329b457c24caf93a9f880345eb288 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.157->[1.158-1.179] Dedup: 1.164->[1.165-1.179]
I20240502 21:26:18.460165 21290 raft_consensus.cc:1050] T 4aff104e3b6b424baa628bbe612b62e8 P 9be399440ded4e2ebe6b6e0bf486cbb1: attempting to promote NON_VOTER 90e329b457c24caf93a9f880345eb288 to VOTER
I20240502 21:26:18.461211 21290 consensus_queue.cc:237] T 4aff104e3b6b424baa628bbe612b62e8 P 9be399440ded4e2ebe6b6e0bf486cbb1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 181, Committed index: 181, Last appended: 1.181, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 182 OBSOLETE_local: false peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } attrs { replace: true } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } }
I20240502 21:26:18.465188 20164 raft_consensus.cc:1261] T 4aff104e3b6b424baa628bbe612b62e8 P 90e329b457c24caf93a9f880345eb288 [term 1 LEARNER]: Refusing update from remote peer 9be399440ded4e2ebe6b6e0bf486cbb1: Log matching property violated. Preceding OpId in replica: term: 1 index: 181. Preceding OpId from leader: term: 1 index: 182. (index mismatch)
I20240502 21:26:18.465797 21221 consensus_queue.cc:1035] T 4aff104e3b6b424baa628bbe612b62e8 P 9be399440ded4e2ebe6b6e0bf486cbb1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 182, Last known committed idx: 181, Time since last communication: 0.000s
I20240502 21:26:18.468336 19146 raft_consensus.cc:1261] T 4aff104e3b6b424baa628bbe612b62e8 P 2877108756304e05bd60ba1186b870da [term 1 FOLLOWER]: Refusing update from remote peer 9be399440ded4e2ebe6b6e0bf486cbb1: Log matching property violated. Preceding OpId in replica: term: 1 index: 181. Preceding OpId from leader: term: 1 index: 182. (index mismatch)
I20240502 21:26:18.468901 21173 consensus_queue.cc:1035] T 4aff104e3b6b424baa628bbe612b62e8 P 9be399440ded4e2ebe6b6e0bf486cbb1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 182, Last known committed idx: 181, Time since last communication: 0.000s
I20240502 21:26:18.472756 18470 raft_consensus.cc:1261] T 4aff104e3b6b424baa628bbe612b62e8 P faadabfb099b4abf90a466395be585b5 [term 1 FOLLOWER]: Refusing update from remote peer 9be399440ded4e2ebe6b6e0bf486cbb1: Log matching property violated. Preceding OpId in replica: term: 1 index: 181. Preceding OpId from leader: term: 1 index: 182. (index mismatch)
I20240502 21:26:18.472826 18740 raft_consensus.cc:1261] T 4aff104e3b6b424baa628bbe612b62e8 P 501e3c0d44fe4d70af9ff926d9622417 [term 1 FOLLOWER]: Refusing update from remote peer 9be399440ded4e2ebe6b6e0bf486cbb1: Log matching property violated. Preceding OpId in replica: term: 1 index: 181. Preceding OpId from leader: term: 1 index: 182. (index mismatch)
I20240502 21:26:18.473263 21290 consensus_queue.cc:1035] T 4aff104e3b6b424baa628bbe612b62e8 P 9be399440ded4e2ebe6b6e0bf486cbb1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 182, Last known committed idx: 181, Time since last communication: 0.000s
I20240502 21:26:18.473656 21221 consensus_queue.cc:1035] T 4aff104e3b6b424baa628bbe612b62e8 P 9be399440ded4e2ebe6b6e0bf486cbb1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 182, Last known committed idx: 181, Time since last communication: 0.000s
I20240502 21:26:18.477972 21173 raft_consensus.cc:2932] T 4aff104e3b6b424baa628bbe612b62e8 P 9be399440ded4e2ebe6b6e0bf486cbb1 [term 1 LEADER]: Committing config change with OpId 1.182: config changed from index 158 to 182, 90e329b457c24caf93a9f880345eb288 (127.15.255.135) changed from NON_VOTER to VOTER. New config: { opid_index: 182 OBSOLETE_local: false peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } attrs { replace: true } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } } }
I20240502 21:26:18.480686 18743 raft_consensus.cc:2932] T 4aff104e3b6b424baa628bbe612b62e8 P 501e3c0d44fe4d70af9ff926d9622417 [term 1 FOLLOWER]: Committing config change with OpId 1.182: config changed from index 158 to 182, 90e329b457c24caf93a9f880345eb288 (127.15.255.135) changed from NON_VOTER to VOTER. New config: { opid_index: 182 OBSOLETE_local: false peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } attrs { replace: true } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } } }
I20240502 21:26:18.489753 20164 raft_consensus.cc:2932] T 4aff104e3b6b424baa628bbe612b62e8 P 90e329b457c24caf93a9f880345eb288 [term 1 FOLLOWER]: Committing config change with OpId 1.182: config changed from index 158 to 182, 90e329b457c24caf93a9f880345eb288 (127.15.255.135) changed from NON_VOTER to VOTER. New config: { opid_index: 182 OBSOLETE_local: false peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } attrs { replace: true } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } } }
I20240502 21:26:18.489812 19146 raft_consensus.cc:2932] T 4aff104e3b6b424baa628bbe612b62e8 P 2877108756304e05bd60ba1186b870da [term 1 FOLLOWER]: Committing config change with OpId 1.182: config changed from index 158 to 182, 90e329b457c24caf93a9f880345eb288 (127.15.255.135) changed from NON_VOTER to VOTER. New config: { opid_index: 182 OBSOLETE_local: false peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } attrs { replace: true } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } } }
I20240502 21:26:18.494110 18472 raft_consensus.cc:2932] T 4aff104e3b6b424baa628bbe612b62e8 P faadabfb099b4abf90a466395be585b5 [term 1 FOLLOWER]: Committing config change with OpId 1.182: config changed from index 158 to 182, 90e329b457c24caf93a9f880345eb288 (127.15.255.135) changed from NON_VOTER to VOTER. New config: { opid_index: 182 OBSOLETE_local: false peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } attrs { replace: true } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } } }
I20240502 21:26:18.494519 18321 catalog_manager.cc:5484] T 4aff104e3b6b424baa628bbe612b62e8 P 9be399440ded4e2ebe6b6e0bf486cbb1 reported cstate change: config changed from index 158 to 182, 90e329b457c24caf93a9f880345eb288 (127.15.255.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" committed_config { opid_index: 182 OBSOLETE_local: false peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:26:18.509013 18873 raft_consensus.cc:1261] T 4aff104e3b6b424baa628bbe612b62e8 P c75831edb3b44dd59c00ef151adca49d [term 1 FOLLOWER]: Refusing update from remote peer 9be399440ded4e2ebe6b6e0bf486cbb1: Log matching property violated. Preceding OpId in replica: term: 1 index: 181. Preceding OpId from leader: term: 1 index: 182. (index mismatch)
I20240502 21:26:18.512449 21290 consensus_queue.cc:1035] T 4aff104e3b6b424baa628bbe612b62e8 P 9be399440ded4e2ebe6b6e0bf486cbb1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 182, Last known committed idx: 181, Time since last communication: 0.000s
I20240502 21:26:18.523440 18873 raft_consensus.cc:2932] T 4aff104e3b6b424baa628bbe612b62e8 P c75831edb3b44dd59c00ef151adca49d [term 1 FOLLOWER]: Committing config change with OpId 1.182: config changed from index 158 to 182, 90e329b457c24caf93a9f880345eb288 (127.15.255.135) changed from NON_VOTER to VOTER. New config: { opid_index: 182 OBSOLETE_local: false peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } attrs { replace: true } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } } }
W20240502 21:26:18.621598 19078 consensus_peers.cc:485] T b0852f54089944ce935f8ee4cbf2bc5a P 2877108756304e05bd60ba1186b870da -> Peer 90e329b457c24caf93a9f880345eb288 (127.15.255.135:43395): Couldn't send request to peer 90e329b457c24caf93a9f880345eb288. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 6: this message will repeat every 5th retry.
W20240502 21:26:18.721838 18810 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.133:34217, user_credentials={real_user=slave}} blocked reactor thread for 68144.6us
W20240502 21:26:18.764583 18812 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.130:45139, user_credentials={real_user=slave}} blocked reactor thread for 131135us
W20240502 21:26:18.846122 18811 consensus_peers.cc:485] T 51edafdf2a9348548a494119668bfd31 P c75831edb3b44dd59c00ef151adca49d -> Peer 90e329b457c24caf93a9f880345eb288 (127.15.255.135:43395): Couldn't send request to peer 90e329b457c24caf93a9f880345eb288. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 6: this message will repeat every 5th retry.
W20240502 21:26:18.924631 18811 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.135:43395, user_credentials={real_user=slave}} blocked reactor thread for 78528.1us
W20240502 21:26:18.960248 18407 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.130:45139, user_credentials={real_user=slave}} blocked reactor thread for 225276us
W20240502 21:26:19.102188 18406 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.132:43607, user_credentials={real_user=slave}} blocked reactor thread for 141071us
I20240502 21:26:19.118034 19005 consensus_queue.cc:237] T 4aff104e3b6b424baa628bbe612b62e8 P 9be399440ded4e2ebe6b6e0bf486cbb1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 182, Committed index: 182, Last appended: 1.182, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 183 OBSOLETE_local: false peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } }
I20240502 21:26:19.179740 20168 raft_consensus.cc:1261] T 4aff104e3b6b424baa628bbe612b62e8 P 90e329b457c24caf93a9f880345eb288 [term 1 FOLLOWER]: Refusing update from remote peer 9be399440ded4e2ebe6b6e0bf486cbb1: Log matching property violated. Preceding OpId in replica: term: 1 index: 182. Preceding OpId from leader: term: 1 index: 183. (index mismatch)
I20240502 21:26:19.297740 21221 consensus_queue.cc:1035] T 4aff104e3b6b424baa628bbe612b62e8 P 9be399440ded4e2ebe6b6e0bf486cbb1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 183, Last known committed idx: 182, Time since last communication: 0.000s
I20240502 21:26:19.301627 19146 raft_consensus.cc:1261] T 4aff104e3b6b424baa628bbe612b62e8 P 2877108756304e05bd60ba1186b870da [term 1 FOLLOWER]: Refusing update from remote peer 9be399440ded4e2ebe6b6e0bf486cbb1: Log matching property violated. Preceding OpId in replica: term: 1 index: 182. Preceding OpId from leader: term: 1 index: 184. (index mismatch)
W20240502 21:26:19.302443 18945 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.135:43395, user_credentials={real_user=slave}} blocked reactor thread for 116700us
I20240502 21:26:19.302325 21299 consensus_queue.cc:1035] T 4aff104e3b6b424baa628bbe612b62e8 P 9be399440ded4e2ebe6b6e0bf486cbb1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 183, Last known committed idx: 182, Time since last communication: 0.000s
I20240502 21:26:19.303855 18874 raft_consensus.cc:1261] T 4aff104e3b6b424baa628bbe612b62e8 P c75831edb3b44dd59c00ef151adca49d [term 1 FOLLOWER]: Refusing update from remote peer 9be399440ded4e2ebe6b6e0bf486cbb1: Log matching property violated. Preceding OpId in replica: term: 1 index: 182. Preceding OpId from leader: term: 1 index: 184. (index mismatch)
I20240502 21:26:19.316383 18470 raft_consensus.cc:1261] T 4aff104e3b6b424baa628bbe612b62e8 P faadabfb099b4abf90a466395be585b5 [term 1 FOLLOWER]: Refusing update from remote peer 9be399440ded4e2ebe6b6e0bf486cbb1: Log matching property violated. Preceding OpId in replica: term: 1 index: 182. Preceding OpId from leader: term: 1 index: 184. (index mismatch)
I20240502 21:26:19.317368 21221 consensus_queue.cc:1035] T 4aff104e3b6b424baa628bbe612b62e8 P 9be399440ded4e2ebe6b6e0bf486cbb1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 183, Last known committed idx: 182, Time since last communication: 0.000s
I20240502 21:26:19.319434 21173 consensus_queue.cc:1035] T 4aff104e3b6b424baa628bbe612b62e8 P 9be399440ded4e2ebe6b6e0bf486cbb1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 183, Last known committed idx: 182, Time since last communication: 0.000s
I20240502 21:26:19.320168 21286 raft_consensus.cc:2932] T 4aff104e3b6b424baa628bbe612b62e8 P 9be399440ded4e2ebe6b6e0bf486cbb1 [term 1 LEADER]: Committing config change with OpId 1.183: config changed from index 182 to 183, VOTER 501e3c0d44fe4d70af9ff926d9622417 (127.15.255.131) evicted. New config: { opid_index: 183 OBSOLETE_local: false peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } } }
I20240502 21:26:19.325783 20168 raft_consensus.cc:2932] T 4aff104e3b6b424baa628bbe612b62e8 P 90e329b457c24caf93a9f880345eb288 [term 1 FOLLOWER]: Committing config change with OpId 1.183: config changed from index 182 to 183, VOTER 501e3c0d44fe4d70af9ff926d9622417 (127.15.255.131) evicted. New config: { opid_index: 183 OBSOLETE_local: false peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } } }
I20240502 21:26:19.327617 18313 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 4aff104e3b6b424baa628bbe612b62e8 with cas_config_opid_index 182: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:26:19.329439 18329 catalog_manager.cc:5484] T 4aff104e3b6b424baa628bbe612b62e8 P 9be399440ded4e2ebe6b6e0bf486cbb1 reported cstate change: config changed from index 182 to 183, VOTER 501e3c0d44fe4d70af9ff926d9622417 (127.15.255.131) evicted. New cstate: current_term: 1 leader_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" committed_config { opid_index: 183 OBSOLETE_local: false peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:26:19.339138 19146 raft_consensus.cc:2932] T 4aff104e3b6b424baa628bbe612b62e8 P 2877108756304e05bd60ba1186b870da [term 1 FOLLOWER]: Committing config change with OpId 1.183: config changed from index 182 to 183, VOTER 501e3c0d44fe4d70af9ff926d9622417 (127.15.255.131) evicted. New config: { opid_index: 183 OBSOLETE_local: false peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } } }
I20240502 21:26:19.358954 18471 raft_consensus.cc:2932] T 4aff104e3b6b424baa628bbe612b62e8 P faadabfb099b4abf90a466395be585b5 [term 1 FOLLOWER]: Committing config change with OpId 1.183: config changed from index 182 to 183, VOTER 501e3c0d44fe4d70af9ff926d9622417 (127.15.255.131) evicted. New config: { opid_index: 183 OBSOLETE_local: false peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } } }
I20240502 21:26:19.367326 18874 raft_consensus.cc:2932] T 4aff104e3b6b424baa628bbe612b62e8 P c75831edb3b44dd59c00ef151adca49d [term 1 FOLLOWER]: Committing config change with OpId 1.183: config changed from index 182 to 183, VOTER 501e3c0d44fe4d70af9ff926d9622417 (127.15.255.131) evicted. New config: { opid_index: 183 OBSOLETE_local: false peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } } }
W20240502 21:26:19.398710 18679 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.132:43607, user_credentials={real_user=slave}} blocked reactor thread for 51814.8us
I20240502 21:26:19.469136 18722 tablet_service.cc:1514] Processing DeleteTablet for tablet 4aff104e3b6b424baa628bbe612b62e8 with delete_type TABLET_DATA_TOMBSTONED (TS 501e3c0d44fe4d70af9ff926d9622417 not found in new config with opid_index 183) from {username='slave'} at 127.0.0.1:56558
I20240502 21:26:19.491050 21360 tablet_replica.cc:330] T 4aff104e3b6b424baa628bbe612b62e8 P 501e3c0d44fe4d70af9ff926d9622417: stopping tablet replica
I20240502 21:26:19.491281 21360 raft_consensus.cc:2229] T 4aff104e3b6b424baa628bbe612b62e8 P 501e3c0d44fe4d70af9ff926d9622417 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:26:19.492098 21360 raft_consensus.cc:2258] T 4aff104e3b6b424baa628bbe612b62e8 P 501e3c0d44fe4d70af9ff926d9622417 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:19.492892 21360 ts_tablet_manager.cc:1875] T 4aff104e3b6b424baa628bbe612b62e8 P 501e3c0d44fe4d70af9ff926d9622417: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:26:19.499858 21360 ts_tablet_manager.cc:1888] T 4aff104e3b6b424baa628bbe612b62e8 P 501e3c0d44fe4d70af9ff926d9622417: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.182
I20240502 21:26:19.500051 21360 log.cc:1196] T 4aff104e3b6b424baa628bbe612b62e8 P 501e3c0d44fe4d70af9ff926d9622417: Deleting WAL directory at /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-2/wal/wals/4aff104e3b6b424baa628bbe612b62e8
I20240502 21:26:19.501312 18312 catalog_manager.cc:4830] TS 501e3c0d44fe4d70af9ff926d9622417 (127.15.255.131:42501): tablet 4aff104e3b6b424baa628bbe612b62e8 (table rebalance_test_table_3 [id=5af23c390f054d07b5a7985c2611ca30]) successfully deleted
I20240502 21:26:19.586252 21175 tablet_bootstrap.cc:492] T b0852f54089944ce935f8ee4cbf2bc5a P 90e329b457c24caf93a9f880345eb288: Bootstrap replayed 1/1 log segments. Stats: ops{read=157 overwritten=0 applied=156 ignored=0} inserts{seen=787 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240502 21:26:19.586730 21175 tablet_bootstrap.cc:492] T b0852f54089944ce935f8ee4cbf2bc5a P 90e329b457c24caf93a9f880345eb288: Bootstrap complete.
I20240502 21:26:19.587018 21175 ts_tablet_manager.cc:1367] T b0852f54089944ce935f8ee4cbf2bc5a P 90e329b457c24caf93a9f880345eb288: Time spent bootstrapping tablet: real 3.134s user 0.210s sys 0.029s
I20240502 21:26:19.587720 21175 raft_consensus.cc:348] T b0852f54089944ce935f8ee4cbf2bc5a P 90e329b457c24caf93a9f880345eb288 [term 1 LEARNER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 148 OBSOLETE_local: false peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } attrs { replace: true } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: NON_VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: true } }
I20240502 21:26:19.588335 21175 raft_consensus.cc:729] T b0852f54089944ce935f8ee4cbf2bc5a P 90e329b457c24caf93a9f880345eb288 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 90e329b457c24caf93a9f880345eb288, State: Initialized, Role: LEARNER
I20240502 21:26:19.641449 21175 consensus_queue.cc:260] T b0852f54089944ce935f8ee4cbf2bc5a P 90e329b457c24caf93a9f880345eb288 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 156, Last appended: 1.157, Last appended by leader: 157, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 148 OBSOLETE_local: false peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } attrs { replace: true } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: NON_VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: true } }
I20240502 21:26:19.646440 21175 ts_tablet_manager.cc:1398] T b0852f54089944ce935f8ee4cbf2bc5a P 90e329b457c24caf93a9f880345eb288: Time spent starting tablet: real 0.055s user 0.004s sys 0.000s
I20240502 21:26:19.650997 19166 tablet_copy_service.cc:342] P 2877108756304e05bd60ba1186b870da: Request end of tablet copy session 90e329b457c24caf93a9f880345eb288-b0852f54089944ce935f8ee4cbf2bc5a received from {username='slave'} at 127.15.255.135:43445
I20240502 21:26:19.651213 19166 tablet_copy_service.cc:433] P 2877108756304e05bd60ba1186b870da: ending tablet copy session 90e329b457c24caf93a9f880345eb288-b0852f54089944ce935f8ee4cbf2bc5a on tablet b0852f54089944ce935f8ee4cbf2bc5a with peer 90e329b457c24caf93a9f880345eb288
I20240502 21:26:19.661134 21115 tablet_bootstrap.cc:492] T 32c9fc5a18c54207881124fda86f42ff P 90e329b457c24caf93a9f880345eb288: Bootstrap replayed 1/1 log segments. Stats: ops{read=182 overwritten=0 applied=182 ignored=0} inserts{seen=867 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:26:19.667527 21115 tablet_bootstrap.cc:492] T 32c9fc5a18c54207881124fda86f42ff P 90e329b457c24caf93a9f880345eb288: Bootstrap complete.
I20240502 21:26:19.669915 21115 ts_tablet_manager.cc:1367] T 32c9fc5a18c54207881124fda86f42ff P 90e329b457c24caf93a9f880345eb288: Time spent bootstrapping tablet: real 4.596s user 0.240s sys 0.040s
I20240502 21:26:19.671859 21115 raft_consensus.cc:348] T 32c9fc5a18c54207881124fda86f42ff P 90e329b457c24caf93a9f880345eb288 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 168 OBSOLETE_local: false peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } attrs { replace: true } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: NON_VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: true } }
I20240502 21:26:19.678596 21115 raft_consensus.cc:729] T 32c9fc5a18c54207881124fda86f42ff P 90e329b457c24caf93a9f880345eb288 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 90e329b457c24caf93a9f880345eb288, State: Initialized, Role: LEARNER
I20240502 21:26:19.678864 21115 consensus_queue.cc:260] T 32c9fc5a18c54207881124fda86f42ff P 90e329b457c24caf93a9f880345eb288 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 182, Last appended: 1.182, Last appended by leader: 182, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 168 OBSOLETE_local: false peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } attrs { replace: true } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: NON_VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: true } }
I20240502 21:26:19.682492 21115 ts_tablet_manager.cc:1398] T 32c9fc5a18c54207881124fda86f42ff P 90e329b457c24caf93a9f880345eb288: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20240502 21:26:19.685345 18496 tablet_copy_service.cc:342] P faadabfb099b4abf90a466395be585b5: Request end of tablet copy session 90e329b457c24caf93a9f880345eb288-32c9fc5a18c54207881124fda86f42ff received from {username='slave'} at 127.15.255.135:49783
I20240502 21:26:19.685462 18496 tablet_copy_service.cc:433] P faadabfb099b4abf90a466395be585b5: ending tablet copy session 90e329b457c24caf93a9f880345eb288-32c9fc5a18c54207881124fda86f42ff on tablet 32c9fc5a18c54207881124fda86f42ff with peer 90e329b457c24caf93a9f880345eb288
I20240502 21:26:19.710434 21163 tablet_bootstrap.cc:492] T 51edafdf2a9348548a494119668bfd31 P 90e329b457c24caf93a9f880345eb288: Bootstrap replayed 1/1 log segments. Stats: ops{read=209 overwritten=0 applied=209 ignored=0} inserts{seen=881 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:26:19.711652 21163 tablet_bootstrap.cc:492] T 51edafdf2a9348548a494119668bfd31 P 90e329b457c24caf93a9f880345eb288: Bootstrap complete.
I20240502 21:26:19.713989 21163 ts_tablet_manager.cc:1367] T 51edafdf2a9348548a494119668bfd31 P 90e329b457c24caf93a9f880345eb288: Time spent bootstrapping tablet: real 3.236s user 0.298s sys 0.027s
I20240502 21:26:19.714987 21163 raft_consensus.cc:348] T 51edafdf2a9348548a494119668bfd31 P 90e329b457c24caf93a9f880345eb288 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 192 OBSOLETE_local: false peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } attrs { replace: true } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: NON_VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: true } }
I20240502 21:26:19.715334 21163 raft_consensus.cc:729] T 51edafdf2a9348548a494119668bfd31 P 90e329b457c24caf93a9f880345eb288 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 90e329b457c24caf93a9f880345eb288, State: Initialized, Role: LEARNER
I20240502 21:26:19.715564 21163 consensus_queue.cc:260] T 51edafdf2a9348548a494119668bfd31 P 90e329b457c24caf93a9f880345eb288 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 209, Last appended: 1.209, Last appended by leader: 209, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 192 OBSOLETE_local: false peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } attrs { replace: true } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: NON_VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: true } }
I20240502 21:26:19.720279 21163 ts_tablet_manager.cc:1398] T 51edafdf2a9348548a494119668bfd31 P 90e329b457c24caf93a9f880345eb288: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20240502 21:26:19.723169 18898 tablet_copy_service.cc:342] P c75831edb3b44dd59c00ef151adca49d: Request end of tablet copy session 90e329b457c24caf93a9f880345eb288-51edafdf2a9348548a494119668bfd31 received from {username='slave'} at 127.15.255.135:46619
I20240502 21:26:19.723304 18898 tablet_copy_service.cc:433] P c75831edb3b44dd59c00ef151adca49d: ending tablet copy session 90e329b457c24caf93a9f880345eb288-51edafdf2a9348548a494119668bfd31 on tablet 51edafdf2a9348548a494119668bfd31 with peer 90e329b457c24caf93a9f880345eb288
I20240502 21:26:19.741408 20168 raft_consensus.cc:1203] T b0852f54089944ce935f8ee4cbf2bc5a P 90e329b457c24caf93a9f880345eb288 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.147->[1.148-1.201] Dedup: 1.157->[1.158-1.201]
I20240502 21:26:19.924964 20164 raft_consensus.cc:1203] T 51edafdf2a9348548a494119668bfd31 P 90e329b457c24caf93a9f880345eb288 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.191->[1.192-1.244] Dedup: 1.209->[1.210-1.244]
I20240502 21:26:20.094303 20166 raft_consensus.cc:1203] T 32c9fc5a18c54207881124fda86f42ff P 90e329b457c24caf93a9f880345eb288 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.167->[1.168-1.219] Dedup: 1.182->[1.183-1.219]
I20240502 21:26:20.252753 21112 raft_consensus.cc:1050] T 32c9fc5a18c54207881124fda86f42ff P faadabfb099b4abf90a466395be585b5: attempting to promote NON_VOTER 90e329b457c24caf93a9f880345eb288 to VOTER
I20240502 21:26:20.253762 21112 consensus_queue.cc:237] T 32c9fc5a18c54207881124fda86f42ff P faadabfb099b4abf90a466395be585b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 221, Committed index: 221, Last appended: 1.221, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 222 OBSOLETE_local: false peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } attrs { replace: true } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } }
I20240502 21:26:20.261180 19146 raft_consensus.cc:1261] T 32c9fc5a18c54207881124fda86f42ff P 2877108756304e05bd60ba1186b870da [term 1 FOLLOWER]: Refusing update from remote peer faadabfb099b4abf90a466395be585b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 221. Preceding OpId from leader: term: 1 index: 222. (index mismatch)
I20240502 21:26:20.275319 18606 raft_consensus.cc:1261] T 32c9fc5a18c54207881124fda86f42ff P 82f50f4b66f3450aa9f58a0b317f5cea [term 1 FOLLOWER]: Refusing update from remote peer faadabfb099b4abf90a466395be585b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 221. Preceding OpId from leader: term: 1 index: 222. (index mismatch)
I20240502 21:26:20.278748 21200 consensus_queue.cc:1035] T 32c9fc5a18c54207881124fda86f42ff P faadabfb099b4abf90a466395be585b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 222, Last known committed idx: 221, Time since last communication: 0.000s
I20240502 21:26:20.285712 18738 raft_consensus.cc:1261] T 32c9fc5a18c54207881124fda86f42ff P 501e3c0d44fe4d70af9ff926d9622417 [term 1 FOLLOWER]: Refusing update from remote peer faadabfb099b4abf90a466395be585b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 221. Preceding OpId from leader: term: 1 index: 222. (index mismatch)
I20240502 21:26:20.286573 21112 consensus_queue.cc:1035] T 32c9fc5a18c54207881124fda86f42ff P faadabfb099b4abf90a466395be585b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 222, Last known committed idx: 221, Time since last communication: 0.000s
I20240502 21:26:20.290534 21182 consensus_queue.cc:1035] T 32c9fc5a18c54207881124fda86f42ff P faadabfb099b4abf90a466395be585b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 222, Last known committed idx: 221, Time since last communication: 0.000s
I20240502 21:26:20.307053 21055 raft_consensus.cc:2932] T 32c9fc5a18c54207881124fda86f42ff P faadabfb099b4abf90a466395be585b5 [term 1 LEADER]: Committing config change with OpId 1.222: config changed from index 168 to 222, 90e329b457c24caf93a9f880345eb288 (127.15.255.135) changed from NON_VOTER to VOTER. New config: { opid_index: 222 OBSOLETE_local: false peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } attrs { replace: true } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } } }
I20240502 21:26:20.314172 18743 raft_consensus.cc:2932] T 32c9fc5a18c54207881124fda86f42ff P 501e3c0d44fe4d70af9ff926d9622417 [term 1 FOLLOWER]: Committing config change with OpId 1.222: config changed from index 168 to 222, 90e329b457c24caf93a9f880345eb288 (127.15.255.135) changed from NON_VOTER to VOTER. New config: { opid_index: 222 OBSOLETE_local: false peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } attrs { replace: true } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } } }
I20240502 21:26:20.315286 18606 raft_consensus.cc:2932] T 32c9fc5a18c54207881124fda86f42ff P 82f50f4b66f3450aa9f58a0b317f5cea [term 1 FOLLOWER]: Committing config change with OpId 1.222: config changed from index 168 to 222, 90e329b457c24caf93a9f880345eb288 (127.15.255.135) changed from NON_VOTER to VOTER. New config: { opid_index: 222 OBSOLETE_local: false peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } attrs { replace: true } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } } }
I20240502 21:26:20.316744 18329 catalog_manager.cc:5484] T 32c9fc5a18c54207881124fda86f42ff P faadabfb099b4abf90a466395be585b5 reported cstate change: config changed from index 168 to 222, 90e329b457c24caf93a9f880345eb288 (127.15.255.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "faadabfb099b4abf90a466395be585b5" committed_config { opid_index: 222 OBSOLETE_local: false peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 21:26:20.318562 19144 raft_consensus.cc:2932] T 32c9fc5a18c54207881124fda86f42ff P 2877108756304e05bd60ba1186b870da [term 1 FOLLOWER]: Committing config change with OpId 1.222: config changed from index 168 to 222, 90e329b457c24caf93a9f880345eb288 (127.15.255.135) changed from NON_VOTER to VOTER. New config: { opid_index: 222 OBSOLETE_local: false peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } attrs { replace: true } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } } }
I20240502 21:26:20.336922 21127 raft_consensus.cc:1050] T b0852f54089944ce935f8ee4cbf2bc5a P 2877108756304e05bd60ba1186b870da: attempting to promote NON_VOTER 90e329b457c24caf93a9f880345eb288 to VOTER
I20240502 21:26:20.338135 21127 consensus_queue.cc:237] T b0852f54089944ce935f8ee4cbf2bc5a P 2877108756304e05bd60ba1186b870da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 203, Committed index: 203, Last appended: 1.203, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 204 OBSOLETE_local: false peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } attrs { replace: true } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } }
I20240502 21:26:20.341302 20169 raft_consensus.cc:1261] T b0852f54089944ce935f8ee4cbf2bc5a P 90e329b457c24caf93a9f880345eb288 [term 1 LEARNER]: Refusing update from remote peer 2877108756304e05bd60ba1186b870da: Log matching property violated. Preceding OpId in replica: term: 1 index: 203. Preceding OpId from leader: term: 1 index: 204. (index mismatch)
I20240502 21:26:20.342114 21316 consensus_queue.cc:1035] T b0852f54089944ce935f8ee4cbf2bc5a P 2877108756304e05bd60ba1186b870da [LEADER]: Connected to new peer: Peer: permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 204, Last known committed idx: 203, Time since last communication: 0.000s
I20240502 21:26:20.346554 18606 raft_consensus.cc:1261] T b0852f54089944ce935f8ee4cbf2bc5a P 82f50f4b66f3450aa9f58a0b317f5cea [term 1 FOLLOWER]: Refusing update from remote peer 2877108756304e05bd60ba1186b870da: Log matching property violated. Preceding OpId in replica: term: 1 index: 203. Preceding OpId from leader: term: 1 index: 204. (index mismatch)
I20240502 21:26:20.347621 21316 consensus_queue.cc:1035] T b0852f54089944ce935f8ee4cbf2bc5a P 2877108756304e05bd60ba1186b870da [LEADER]: Connected to new peer: Peer: permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 204, Last known committed idx: 203, Time since last communication: 0.000s
I20240502 21:26:20.349700 19007 raft_consensus.cc:1261] T b0852f54089944ce935f8ee4cbf2bc5a P 9be399440ded4e2ebe6b6e0bf486cbb1 [term 1 FOLLOWER]: Refusing update from remote peer 2877108756304e05bd60ba1186b870da: Log matching property violated. Preceding OpId in replica: term: 1 index: 203. Preceding OpId from leader: term: 1 index: 204. (index mismatch)
I20240502 21:26:20.350334 21316 consensus_queue.cc:1035] T b0852f54089944ce935f8ee4cbf2bc5a P 2877108756304e05bd60ba1186b870da [LEADER]: Connected to new peer: Peer: permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 204, Last known committed idx: 203, Time since last communication: 0.000s
I20240502 21:26:20.350546 18738 raft_consensus.cc:1261] T b0852f54089944ce935f8ee4cbf2bc5a P 501e3c0d44fe4d70af9ff926d9622417 [term 1 FOLLOWER]: Refusing update from remote peer 2877108756304e05bd60ba1186b870da: Log matching property violated. Preceding OpId in replica: term: 1 index: 203. Preceding OpId from leader: term: 1 index: 204. (index mismatch)
I20240502 21:26:20.351042 21316 consensus_queue.cc:1035] T b0852f54089944ce935f8ee4cbf2bc5a P 2877108756304e05bd60ba1186b870da [LEADER]: Connected to new peer: Peer: permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 204, Last known committed idx: 203, Time since last communication: 0.000s
I20240502 21:26:20.361697 21316 raft_consensus.cc:2932] T b0852f54089944ce935f8ee4cbf2bc5a P 2877108756304e05bd60ba1186b870da [term 1 LEADER]: Committing config change with OpId 1.204: config changed from index 148 to 204, 90e329b457c24caf93a9f880345eb288 (127.15.255.135) changed from NON_VOTER to VOTER. New config: { opid_index: 204 OBSOLETE_local: false peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } attrs { replace: true } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } } }
I20240502 21:26:20.366034 20166 raft_consensus.cc:1261] T 32c9fc5a18c54207881124fda86f42ff P 90e329b457c24caf93a9f880345eb288 [term 1 LEARNER]: Refusing update from remote peer faadabfb099b4abf90a466395be585b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 221. Preceding OpId from leader: term: 1 index: 222. (index mismatch)
I20240502 21:26:20.369458 20169 raft_consensus.cc:2932] T b0852f54089944ce935f8ee4cbf2bc5a P 90e329b457c24caf93a9f880345eb288 [term 1 FOLLOWER]: Committing config change with OpId 1.204: config changed from index 148 to 204, 90e329b457c24caf93a9f880345eb288 (127.15.255.135) changed from NON_VOTER to VOTER. New config: { opid_index: 204 OBSOLETE_local: false peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } attrs { replace: true } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } } }
I20240502 21:26:20.375852 18740 raft_consensus.cc:2932] T b0852f54089944ce935f8ee4cbf2bc5a P 501e3c0d44fe4d70af9ff926d9622417 [term 1 FOLLOWER]: Committing config change with OpId 1.204: config changed from index 148 to 204, 90e329b457c24caf93a9f880345eb288 (127.15.255.135) changed from NON_VOTER to VOTER. New config: { opid_index: 204 OBSOLETE_local: false peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } attrs { replace: true } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } } }
I20240502 21:26:20.377779 19007 raft_consensus.cc:2932] T b0852f54089944ce935f8ee4cbf2bc5a P 9be399440ded4e2ebe6b6e0bf486cbb1 [term 1 FOLLOWER]: Committing config change with OpId 1.204: config changed from index 148 to 204, 90e329b457c24caf93a9f880345eb288 (127.15.255.135) changed from NON_VOTER to VOTER. New config: { opid_index: 204 OBSOLETE_local: false peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } attrs { replace: true } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } } }
I20240502 21:26:20.384495 18611 raft_consensus.cc:2932] T b0852f54089944ce935f8ee4cbf2bc5a P 82f50f4b66f3450aa9f58a0b317f5cea [term 1 FOLLOWER]: Committing config change with OpId 1.204: config changed from index 148 to 204, 90e329b457c24caf93a9f880345eb288 (127.15.255.135) changed from NON_VOTER to VOTER. New config: { opid_index: 204 OBSOLETE_local: false peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } attrs { replace: true } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } } }
I20240502 21:26:20.406031 18320 catalog_manager.cc:5484] T b0852f54089944ce935f8ee4cbf2bc5a P 9be399440ded4e2ebe6b6e0bf486cbb1 reported cstate change: config changed from index 148 to 204, 90e329b457c24caf93a9f880345eb288 (127.15.255.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "2877108756304e05bd60ba1186b870da" committed_config { opid_index: 204 OBSOLETE_local: false peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } attrs { replace: true } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } } }
I20240502 21:26:20.478646 18873 raft_consensus.cc:1261] T 32c9fc5a18c54207881124fda86f42ff P c75831edb3b44dd59c00ef151adca49d [term 1 FOLLOWER]: Refusing update from remote peer faadabfb099b4abf90a466395be585b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 221. Preceding OpId from leader: term: 1 index: 222. (index mismatch)
I20240502 21:26:20.480185 18474 raft_consensus.cc:1261] T b0852f54089944ce935f8ee4cbf2bc5a P faadabfb099b4abf90a466395be585b5 [term 1 FOLLOWER]: Refusing update from remote peer 2877108756304e05bd60ba1186b870da: Log matching property violated. Preceding OpId in replica: term: 1 index: 203. Preceding OpId from leader: term: 1 index: 204. (index mismatch)
I20240502 21:26:20.485821 21200 consensus_queue.cc:1035] T 32c9fc5a18c54207881124fda86f42ff P faadabfb099b4abf90a466395be585b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 222, Last known committed idx: 221, Time since last communication: 0.000s
I20240502 21:26:20.487718 21316 consensus_queue.cc:1035] T b0852f54089944ce935f8ee4cbf2bc5a P 2877108756304e05bd60ba1186b870da [LEADER]: Connected to new peer: Peer: permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 204, Last known committed idx: 203, Time since last communication: 0.000s
W20240502 21:26:20.585618 18812 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.134:43635, user_credentials={real_user=slave}} blocked reactor thread for 78407.9us
I20240502 21:26:20.696913 20166 raft_consensus.cc:2932] T 32c9fc5a18c54207881124fda86f42ff P 90e329b457c24caf93a9f880345eb288 [term 1 FOLLOWER]: Committing config change with OpId 1.222: config changed from index 168 to 222, 90e329b457c24caf93a9f880345eb288 (127.15.255.135) changed from NON_VOTER to VOTER. New config: { opid_index: 222 OBSOLETE_local: false peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } attrs { replace: true } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } } }
I20240502 21:26:20.750641 21182 consensus_queue.cc:1035] T 32c9fc5a18c54207881124fda86f42ff P faadabfb099b4abf90a466395be585b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 222, Last known committed idx: 221, Time since last communication: 0.000s
I20240502 21:26:20.779021 18474 raft_consensus.cc:2932] T b0852f54089944ce935f8ee4cbf2bc5a P faadabfb099b4abf90a466395be585b5 [term 1 FOLLOWER]: Committing config change with OpId 1.204: config changed from index 148 to 204, 90e329b457c24caf93a9f880345eb288 (127.15.255.135) changed from NON_VOTER to VOTER. New config: { opid_index: 204 OBSOLETE_local: false peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } attrs { replace: true } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } } }
W20240502 21:26:20.828102 21387 log.cc:924] Time spent T 51edafdf2a9348548a494119668bfd31 P 90e329b457c24caf93a9f880345eb288: Append to log took a long time: real 0.074s user 0.000s sys 0.000s
I20240502 21:26:20.881590 18871 raft_consensus.cc:2932] T 32c9fc5a18c54207881124fda86f42ff P c75831edb3b44dd59c00ef151adca49d [term 1 FOLLOWER]: Committing config change with OpId 1.222: config changed from index 168 to 222, 90e329b457c24caf93a9f880345eb288 (127.15.255.135) changed from NON_VOTER to VOTER. New config: { opid_index: 222 OBSOLETE_local: false peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } attrs { replace: true } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } } }
I20240502 21:26:20.897790 20963 raft_consensus.cc:1050] T 51edafdf2a9348548a494119668bfd31 P c75831edb3b44dd59c00ef151adca49d: attempting to promote NON_VOTER 90e329b457c24caf93a9f880345eb288 to VOTER
I20240502 21:26:20.898736 20963 consensus_queue.cc:237] T 51edafdf2a9348548a494119668bfd31 P c75831edb3b44dd59c00ef151adca49d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 250, Committed index: 250, Last appended: 1.250, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 251 OBSOLETE_local: false peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } attrs { replace: true } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } }
I20240502 21:26:20.902932 19005 raft_consensus.cc:1261] T 51edafdf2a9348548a494119668bfd31 P 9be399440ded4e2ebe6b6e0bf486cbb1 [term 1 FOLLOWER]: Refusing update from remote peer c75831edb3b44dd59c00ef151adca49d: Log matching property violated. Preceding OpId in replica: term: 1 index: 250. Preceding OpId from leader: term: 1 index: 251. (index mismatch)
I20240502 21:26:20.903782 21375 consensus_queue.cc:1035] T 51edafdf2a9348548a494119668bfd31 P c75831edb3b44dd59c00ef151adca49d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 251, Last known committed idx: 250, Time since last communication: 0.000s
I20240502 21:26:20.904016 18473 raft_consensus.cc:1261] T 51edafdf2a9348548a494119668bfd31 P faadabfb099b4abf90a466395be585b5 [term 1 FOLLOWER]: Refusing update from remote peer c75831edb3b44dd59c00ef151adca49d: Log matching property violated. Preceding OpId in replica: term: 1 index: 250. Preceding OpId from leader: term: 1 index: 251. (index mismatch)
I20240502 21:26:20.905405 20963 consensus_queue.cc:1035] T 51edafdf2a9348548a494119668bfd31 P c75831edb3b44dd59c00ef151adca49d [LEADER]: Connected to new peer: Peer: permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 251, Last known committed idx: 250, Time since last communication: 0.000s
I20240502 21:26:20.905970 20169 raft_consensus.cc:1261] T 51edafdf2a9348548a494119668bfd31 P 90e329b457c24caf93a9f880345eb288 [term 1 LEARNER]: Refusing update from remote peer c75831edb3b44dd59c00ef151adca49d: Log matching property violated. Preceding OpId in replica: term: 1 index: 250. Preceding OpId from leader: term: 1 index: 251. (index mismatch)
I20240502 21:26:20.906919 21375 consensus_queue.cc:1035] T 51edafdf2a9348548a494119668bfd31 P c75831edb3b44dd59c00ef151adca49d [LEADER]: Connected to new peer: Peer: permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 251, Last known committed idx: 250, Time since last communication: 0.000s
I20240502 21:26:20.912835 21375 raft_consensus.cc:2932] T 51edafdf2a9348548a494119668bfd31 P c75831edb3b44dd59c00ef151adca49d [term 1 LEADER]: Committing config change with OpId 1.251: config changed from index 192 to 251, 90e329b457c24caf93a9f880345eb288 (127.15.255.135) changed from NON_VOTER to VOTER. New config: { opid_index: 251 OBSOLETE_local: false peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } attrs { replace: true } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } } }
I20240502 21:26:20.913962 20169 raft_consensus.cc:2932] T 51edafdf2a9348548a494119668bfd31 P 90e329b457c24caf93a9f880345eb288 [term 1 FOLLOWER]: Committing config change with OpId 1.251: config changed from index 192 to 251, 90e329b457c24caf93a9f880345eb288 (127.15.255.135) changed from NON_VOTER to VOTER. New config: { opid_index: 251 OBSOLETE_local: false peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } attrs { replace: true } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } } }
I20240502 21:26:20.928753 18610 raft_consensus.cc:1261] T 51edafdf2a9348548a494119668bfd31 P 82f50f4b66f3450aa9f58a0b317f5cea [term 1 FOLLOWER]: Refusing update from remote peer c75831edb3b44dd59c00ef151adca49d: Log matching property violated. Preceding OpId in replica: term: 1 index: 250. Preceding OpId from leader: term: 1 index: 251. (index mismatch)
I20240502 21:26:20.936453 19140 raft_consensus.cc:1261] T 51edafdf2a9348548a494119668bfd31 P 2877108756304e05bd60ba1186b870da [term 1 FOLLOWER]: Refusing update from remote peer c75831edb3b44dd59c00ef151adca49d: Log matching property violated. Preceding OpId in replica: term: 1 index: 250. Preceding OpId from leader: term: 1 index: 251. (index mismatch)
I20240502 21:26:20.994836 21081 consensus_queue.cc:1035] T 51edafdf2a9348548a494119668bfd31 P c75831edb3b44dd59c00ef151adca49d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 251, Last known committed idx: 249, Time since last communication: 0.000s
I20240502 21:26:21.010671 21375 consensus_queue.cc:1035] T 51edafdf2a9348548a494119668bfd31 P c75831edb3b44dd59c00ef151adca49d [LEADER]: Connected to new peer: Peer: permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 251, Last known committed idx: 250, Time since last communication: 0.000s
I20240502 21:26:21.003942 19008 raft_consensus.cc:2932] T 51edafdf2a9348548a494119668bfd31 P 9be399440ded4e2ebe6b6e0bf486cbb1 [term 1 FOLLOWER]: Committing config change with OpId 1.251: config changed from index 192 to 251, 90e329b457c24caf93a9f880345eb288 (127.15.255.135) changed from NON_VOTER to VOTER. New config: { opid_index: 251 OBSOLETE_local: false peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } attrs { replace: true } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } } }
I20240502 21:26:21.027068 18474 raft_consensus.cc:2932] T 51edafdf2a9348548a494119668bfd31 P faadabfb099b4abf90a466395be585b5 [term 1 FOLLOWER]: Committing config change with OpId 1.251: config changed from index 192 to 251, 90e329b457c24caf93a9f880345eb288 (127.15.255.135) changed from NON_VOTER to VOTER. New config: { opid_index: 251 OBSOLETE_local: false peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } attrs { replace: true } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } } }
I20240502 21:26:21.061875 19144 raft_consensus.cc:2932] T 51edafdf2a9348548a494119668bfd31 P 2877108756304e05bd60ba1186b870da [term 1 FOLLOWER]: Committing config change with OpId 1.251: config changed from index 192 to 251, 90e329b457c24caf93a9f880345eb288 (127.15.255.135) changed from NON_VOTER to VOTER. New config: { opid_index: 251 OBSOLETE_local: false peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } attrs { replace: true } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } } }
I20240502 21:26:21.073225 18610 raft_consensus.cc:2932] T 51edafdf2a9348548a494119668bfd31 P 82f50f4b66f3450aa9f58a0b317f5cea [term 1 FOLLOWER]: Committing config change with OpId 1.251: config changed from index 192 to 251, 90e329b457c24caf93a9f880345eb288 (127.15.255.135) changed from NON_VOTER to VOTER. New config: { opid_index: 251 OBSOLETE_local: false peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } attrs { replace: true } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } } }
I20240502 21:26:21.120848 18327 catalog_manager.cc:5484] T 51edafdf2a9348548a494119668bfd31 P 90e329b457c24caf93a9f880345eb288 reported cstate change: config changed from index 192 to 251, 90e329b457c24caf93a9f880345eb288 (127.15.255.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c75831edb3b44dd59c00ef151adca49d" committed_config { opid_index: 251 OBSOLETE_local: false peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } attrs { replace: true } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } } }
I20240502 21:26:21.138608 18472 consensus_queue.cc:237] T 32c9fc5a18c54207881124fda86f42ff P faadabfb099b4abf90a466395be585b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 226, Committed index: 226, Last appended: 1.227, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 228 OBSOLETE_local: false peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } }
I20240502 21:26:21.160324 18606 raft_consensus.cc:1261] T 32c9fc5a18c54207881124fda86f42ff P 82f50f4b66f3450aa9f58a0b317f5cea [term 1 FOLLOWER]: Refusing update from remote peer faadabfb099b4abf90a466395be585b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 227. Preceding OpId from leader: term: 1 index: 228. (index mismatch)
I20240502 21:26:21.161756 21200 consensus_queue.cc:1035] T 32c9fc5a18c54207881124fda86f42ff P faadabfb099b4abf90a466395be585b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 228, Last known committed idx: 226, Time since last communication: 0.000s
I20240502 21:26:21.169198 18740 raft_consensus.cc:1261] T 32c9fc5a18c54207881124fda86f42ff P 501e3c0d44fe4d70af9ff926d9622417 [term 1 FOLLOWER]: Refusing update from remote peer faadabfb099b4abf90a466395be585b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 226. Preceding OpId from leader: term: 1 index: 228. (index mismatch)
I20240502 21:26:21.171027 21055 consensus_queue.cc:1035] T 32c9fc5a18c54207881124fda86f42ff P faadabfb099b4abf90a466395be585b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 228, Last known committed idx: 226, Time since last communication: 0.000s
I20240502 21:26:21.174386 18739 raft_consensus.cc:1203] T 32c9fc5a18c54207881124fda86f42ff P 501e3c0d44fe4d70af9ff926d9622417 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.226->[1.227-1.228] Dedup: 1.227->[1.228-1.228]
I20240502 21:26:21.183980 19144 consensus_queue.cc:237] T b0852f54089944ce935f8ee4cbf2bc5a P 2877108756304e05bd60ba1186b870da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 205, Committed index: 205, Last appended: 1.205, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 206 OBSOLETE_local: false peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } }
I20240502 21:26:21.197715 18739 raft_consensus.cc:1261] T b0852f54089944ce935f8ee4cbf2bc5a P 501e3c0d44fe4d70af9ff926d9622417 [term 1 FOLLOWER]: Refusing update from remote peer 2877108756304e05bd60ba1186b870da: Log matching property violated. Preceding OpId in replica: term: 1 index: 205. Preceding OpId from leader: term: 1 index: 206. (index mismatch)
I20240502 21:26:21.197810 20169 raft_consensus.cc:1261] T b0852f54089944ce935f8ee4cbf2bc5a P 90e329b457c24caf93a9f880345eb288 [term 1 FOLLOWER]: Refusing update from remote peer 2877108756304e05bd60ba1186b870da: Log matching property violated. Preceding OpId in replica: term: 1 index: 205. Preceding OpId from leader: term: 1 index: 206. (index mismatch)
I20240502 21:26:21.197886 18606 raft_consensus.cc:1261] T b0852f54089944ce935f8ee4cbf2bc5a P 82f50f4b66f3450aa9f58a0b317f5cea [term 1 FOLLOWER]: Refusing update from remote peer 2877108756304e05bd60ba1186b870da: Log matching property violated. Preceding OpId in replica: term: 1 index: 205. Preceding OpId from leader: term: 1 index: 206. (index mismatch)
I20240502 21:26:21.198513 21419 consensus_queue.cc:1035] T b0852f54089944ce935f8ee4cbf2bc5a P 2877108756304e05bd60ba1186b870da [LEADER]: Connected to new peer: Peer: permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 206, Last known committed idx: 205, Time since last communication: 0.000s
I20240502 21:26:21.198984 21316 consensus_queue.cc:1035] T b0852f54089944ce935f8ee4cbf2bc5a P 2877108756304e05bd60ba1186b870da [LEADER]: Connected to new peer: Peer: permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 206, Last known committed idx: 205, Time since last communication: 0.000s
I20240502 21:26:21.207307 20166 raft_consensus.cc:1261] T 32c9fc5a18c54207881124fda86f42ff P 90e329b457c24caf93a9f880345eb288 [term 1 FOLLOWER]: Refusing update from remote peer faadabfb099b4abf90a466395be585b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 226. Preceding OpId from leader: term: 1 index: 228. (index mismatch)
I20240502 21:26:21.209056 21112 consensus_queue.cc:1035] T 32c9fc5a18c54207881124fda86f42ff P faadabfb099b4abf90a466395be585b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 228, Last known committed idx: 226, Time since last communication: 0.000s
I20240502 21:26:21.210552 20164 raft_consensus.cc:1203] T 32c9fc5a18c54207881124fda86f42ff P 90e329b457c24caf93a9f880345eb288 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.226->[1.227-1.228] Dedup: 1.227->[1.228-1.228]
I20240502 21:26:21.211956 21329 raft_consensus.cc:2932] T 32c9fc5a18c54207881124fda86f42ff P faadabfb099b4abf90a466395be585b5 [term 1 LEADER]: Committing config change with OpId 1.228: config changed from index 222 to 228, VOTER 2877108756304e05bd60ba1186b870da (127.15.255.134) evicted. New config: { opid_index: 228 OBSOLETE_local: false peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } } }
I20240502 21:26:21.212627 20164 raft_consensus.cc:2932] T 32c9fc5a18c54207881124fda86f42ff P 90e329b457c24caf93a9f880345eb288 [term 1 FOLLOWER]: Committing config change with OpId 1.228: config changed from index 222 to 228, VOTER 2877108756304e05bd60ba1186b870da (127.15.255.134) evicted. New config: { opid_index: 228 OBSOLETE_local: false peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } } }
I20240502 21:26:21.220983 18319 catalog_manager.cc:5484] T 32c9fc5a18c54207881124fda86f42ff P 90e329b457c24caf93a9f880345eb288 reported cstate change: config changed from index 222 to 228, VOTER 2877108756304e05bd60ba1186b870da (127.15.255.134) evicted. New cstate: current_term: 1 leader_uuid: "faadabfb099b4abf90a466395be585b5" committed_config { opid_index: 228 OBSOLETE_local: false peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } } }
I20240502 21:26:21.223021 18313 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 32c9fc5a18c54207881124fda86f42ff with cas_config_opid_index 222: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:26:21.227535 21418 raft_consensus.cc:2932] T b0852f54089944ce935f8ee4cbf2bc5a P 2877108756304e05bd60ba1186b870da [term 1 LEADER]: Committing config change with OpId 1.206: config changed from index 204 to 206, VOTER faadabfb099b4abf90a466395be585b5 (127.15.255.129) evicted. New config: { opid_index: 206 OBSOLETE_local: false peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } } }
I20240502 21:26:21.228281 18739 raft_consensus.cc:2932] T 32c9fc5a18c54207881124fda86f42ff P 501e3c0d44fe4d70af9ff926d9622417 [term 1 FOLLOWER]: Committing config change with OpId 1.228: config changed from index 222 to 228, VOTER 2877108756304e05bd60ba1186b870da (127.15.255.134) evicted. New config: { opid_index: 228 OBSOLETE_local: false peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } } }
I20240502 21:26:21.230382 18611 raft_consensus.cc:2932] T b0852f54089944ce935f8ee4cbf2bc5a P 82f50f4b66f3450aa9f58a0b317f5cea [term 1 FOLLOWER]: Committing config change with OpId 1.206: config changed from index 204 to 206, VOTER faadabfb099b4abf90a466395be585b5 (127.15.255.129) evicted. New config: { opid_index: 206 OBSOLETE_local: false peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } } }
I20240502 21:26:21.231199 19124 tablet_service.cc:1514] Processing DeleteTablet for tablet 32c9fc5a18c54207881124fda86f42ff with delete_type TABLET_DATA_TOMBSTONED (TS 2877108756304e05bd60ba1186b870da not found in new config with opid_index 228) from {username='slave'} at 127.0.0.1:57946
I20240502 21:26:21.231613 18738 raft_consensus.cc:2932] T b0852f54089944ce935f8ee4cbf2bc5a P 501e3c0d44fe4d70af9ff926d9622417 [term 1 FOLLOWER]: Committing config change with OpId 1.206: config changed from index 204 to 206, VOTER faadabfb099b4abf90a466395be585b5 (127.15.255.129) evicted. New config: { opid_index: 206 OBSOLETE_local: false peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } } }
I20240502 21:26:21.239276 18315 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet b0852f54089944ce935f8ee4cbf2bc5a with cas_config_opid_index 204: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:26:21.240414 18321 catalog_manager.cc:5484] T b0852f54089944ce935f8ee4cbf2bc5a P 501e3c0d44fe4d70af9ff926d9622417 reported cstate change: config changed from index 204 to 206, VOTER faadabfb099b4abf90a466395be585b5 (127.15.255.129) evicted. New cstate: current_term: 1 leader_uuid: "2877108756304e05bd60ba1186b870da" committed_config { opid_index: 206 OBSOLETE_local: false peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } } }
I20240502 21:26:21.245322 18451 tablet_service.cc:1514] Processing DeleteTablet for tablet b0852f54089944ce935f8ee4cbf2bc5a with delete_type TABLET_DATA_TOMBSTONED (TS faadabfb099b4abf90a466395be585b5 not found in new config with opid_index 206) from {username='slave'} at 127.0.0.1:39378
I20240502 21:26:21.267271 21425 tablet_replica.cc:330] T b0852f54089944ce935f8ee4cbf2bc5a P faadabfb099b4abf90a466395be585b5: stopping tablet replica
I20240502 21:26:21.268884 21425 raft_consensus.cc:2229] T b0852f54089944ce935f8ee4cbf2bc5a P faadabfb099b4abf90a466395be585b5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:26:21.270018 21425 raft_consensus.cc:2258] T b0852f54089944ce935f8ee4cbf2bc5a P faadabfb099b4abf90a466395be585b5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:21.271266 21425 ts_tablet_manager.cc:1875] T b0852f54089944ce935f8ee4cbf2bc5a P faadabfb099b4abf90a466395be585b5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:26:21.291684 19007 raft_consensus.cc:1261] T b0852f54089944ce935f8ee4cbf2bc5a P 9be399440ded4e2ebe6b6e0bf486cbb1 [term 1 FOLLOWER]: Refusing update from remote peer 2877108756304e05bd60ba1186b870da: Log matching property violated. Preceding OpId in replica: term: 1 index: 205. Preceding OpId from leader: term: 1 index: 207. (index mismatch)
I20240502 21:26:21.292765 21316 consensus_queue.cc:1035] T b0852f54089944ce935f8ee4cbf2bc5a P 2877108756304e05bd60ba1186b870da [LEADER]: Connected to new peer: Peer: permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 206, Last known committed idx: 205, Time since last communication: 0.000s
I20240502 21:26:21.294626 19007 raft_consensus.cc:2932] T b0852f54089944ce935f8ee4cbf2bc5a P 9be399440ded4e2ebe6b6e0bf486cbb1 [term 1 FOLLOWER]: Committing config change with OpId 1.206: config changed from index 204 to 206, VOTER faadabfb099b4abf90a466395be585b5 (127.15.255.129) evicted. New config: { opid_index: 206 OBSOLETE_local: false peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } } }
I20240502 21:26:21.299276 21425 ts_tablet_manager.cc:1888] T b0852f54089944ce935f8ee4cbf2bc5a P faadabfb099b4abf90a466395be585b5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.205
I20240502 21:26:21.308512 21425 log.cc:1196] T b0852f54089944ce935f8ee4cbf2bc5a P faadabfb099b4abf90a466395be585b5: Deleting WAL directory at /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-0/wal/wals/b0852f54089944ce935f8ee4cbf2bc5a
I20240502 21:26:21.311533 18313 catalog_manager.cc:4830] TS faadabfb099b4abf90a466395be585b5 (127.15.255.129:32983): tablet b0852f54089944ce935f8ee4cbf2bc5a (table rebalance_test_table_4 [id=c79c94b660074ffc94fcea415b4caae8]) successfully deleted
I20240502 21:26:21.343580 21419 consensus_queue.cc:1035] T b0852f54089944ce935f8ee4cbf2bc5a P 2877108756304e05bd60ba1186b870da [LEADER]: Connected to new peer: Peer: permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 206, Last known committed idx: 205, Time since last communication: 0.003s
I20240502 21:26:21.348388 20164 raft_consensus.cc:2932] T b0852f54089944ce935f8ee4cbf2bc5a P 90e329b457c24caf93a9f880345eb288 [term 1 FOLLOWER]: Committing config change with OpId 1.206: config changed from index 204 to 206, VOTER faadabfb099b4abf90a466395be585b5 (127.15.255.129) evicted. New config: { opid_index: 206 OBSOLETE_local: false peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } } }
I20240502 21:26:21.406035 18871 raft_consensus.cc:1261] T 32c9fc5a18c54207881124fda86f42ff P c75831edb3b44dd59c00ef151adca49d [term 1 FOLLOWER]: Refusing update from remote peer faadabfb099b4abf90a466395be585b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 226. Preceding OpId from leader: term: 1 index: 228. (index mismatch)
I20240502 21:26:21.483214 21182 consensus_queue.cc:1035] T 32c9fc5a18c54207881124fda86f42ff P faadabfb099b4abf90a466395be585b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 228, Last known committed idx: 226, Time since last communication: 0.000s
I20240502 21:26:21.493057 21424 tablet_replica.cc:330] T 32c9fc5a18c54207881124fda86f42ff P 2877108756304e05bd60ba1186b870da: stopping tablet replica
I20240502 21:26:21.493299 21424 raft_consensus.cc:2229] T 32c9fc5a18c54207881124fda86f42ff P 2877108756304e05bd60ba1186b870da [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:26:21.493490 21424 pending_rounds.cc:62] T 32c9fc5a18c54207881124fda86f42ff P 2877108756304e05bd60ba1186b870da: Trying to abort 1 pending ops.
I20240502 21:26:21.493539 21424 pending_rounds.cc:68] T 32c9fc5a18c54207881124fda86f42ff P 2877108756304e05bd60ba1186b870da: Aborting op as it isn't in flight: id { term: 1 index: 227 } timestamp: 7023350501933809664 op_type: WRITE_OP write_request { tablet_id: "32c9fc5a18c54207881124fda86f42ff" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000\222\350/!\345\2154\247\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""219ad31e7734cf9c" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7023350501794045952 authz_token { token_data: "\010\215\214\320\261\006\"3\n\005slave\022*\n 086ede98fc604333833bce759f342073\020\001\030\001 \001(\001" signature: "<redacted>""}\223\032\242\274\354X\201\n\023\301#;\373\024\245\254\306,\310\333\331\017hS\320\257\377-]\"\305\362\016\342\'\236\002\263\235U\002\033vX\275\"\001+\345\245\364\275=J{\222r\266\272\311\013\t9" signing_key_seq_num: 0 } } request_id { client_id: "2762c302b7494db2a6fdf48475f1ee6b" seq_no: 678 first_incomplete_seq_no: 678 attempt_no: 0 }
I20240502 21:26:21.494124 21424 raft_consensus.cc:2258] T 32c9fc5a18c54207881124fda86f42ff P 2877108756304e05bd60ba1186b870da [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:21.505417 18871 raft_consensus.cc:2932] T 32c9fc5a18c54207881124fda86f42ff P c75831edb3b44dd59c00ef151adca49d [term 1 FOLLOWER]: Committing config change with OpId 1.228: config changed from index 222 to 228, VOTER 2877108756304e05bd60ba1186b870da (127.15.255.134) evicted. New config: { opid_index: 228 OBSOLETE_local: false peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } } }
I20240502 21:26:21.519029 18606 raft_consensus.cc:2932] T 32c9fc5a18c54207881124fda86f42ff P 82f50f4b66f3450aa9f58a0b317f5cea [term 1 FOLLOWER]: Committing config change with OpId 1.228: config changed from index 222 to 228, VOTER 2877108756304e05bd60ba1186b870da (127.15.255.134) evicted. New config: { opid_index: 228 OBSOLETE_local: false peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } } }
I20240502 21:26:21.543361 18871 consensus_queue.cc:237] T 51edafdf2a9348548a494119668bfd31 P c75831edb3b44dd59c00ef151adca49d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 257, Committed index: 257, Last appended: 1.257, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 258 OBSOLETE_local: false peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } }
I20240502 21:26:21.546144 20166 raft_consensus.cc:1261] T 51edafdf2a9348548a494119668bfd31 P 90e329b457c24caf93a9f880345eb288 [term 1 FOLLOWER]: Refusing update from remote peer c75831edb3b44dd59c00ef151adca49d: Log matching property violated. Preceding OpId in replica: term: 1 index: 257. Preceding OpId from leader: term: 1 index: 258. (index mismatch)
I20240502 21:26:21.546758 21081 consensus_queue.cc:1035] T 51edafdf2a9348548a494119668bfd31 P c75831edb3b44dd59c00ef151adca49d [LEADER]: Connected to new peer: Peer: permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 258, Last known committed idx: 257, Time since last communication: 0.000s
I20240502 21:26:21.553969 19005 raft_consensus.cc:1261] T 51edafdf2a9348548a494119668bfd31 P 9be399440ded4e2ebe6b6e0bf486cbb1 [term 1 FOLLOWER]: Refusing update from remote peer c75831edb3b44dd59c00ef151adca49d: Log matching property violated. Preceding OpId in replica: term: 1 index: 257. Preceding OpId from leader: term: 1 index: 258. (index mismatch)
I20240502 21:26:21.554509 21081 consensus_queue.cc:1035] T 51edafdf2a9348548a494119668bfd31 P c75831edb3b44dd59c00ef151adca49d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 258, Last known committed idx: 257, Time since last communication: 0.000s
I20240502 21:26:21.556833 21081 raft_consensus.cc:2932] T 51edafdf2a9348548a494119668bfd31 P c75831edb3b44dd59c00ef151adca49d [term 1 LEADER]: Committing config change with OpId 1.258: config changed from index 251 to 258, VOTER faadabfb099b4abf90a466395be585b5 (127.15.255.129) evicted. New config: { opid_index: 258 OBSOLETE_local: false peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } } }
I20240502 21:26:21.559255 19009 raft_consensus.cc:2932] T 51edafdf2a9348548a494119668bfd31 P 9be399440ded4e2ebe6b6e0bf486cbb1 [term 1 FOLLOWER]: Committing config change with OpId 1.258: config changed from index 251 to 258, VOTER faadabfb099b4abf90a466395be585b5 (127.15.255.129) evicted. New config: { opid_index: 258 OBSOLETE_local: false peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } } }
I20240502 21:26:21.565685 18319 catalog_manager.cc:5484] T 51edafdf2a9348548a494119668bfd31 P 9be399440ded4e2ebe6b6e0bf486cbb1 reported cstate change: config changed from index 251 to 258, VOTER faadabfb099b4abf90a466395be585b5 (127.15.255.129) evicted. New cstate: current_term: 1 leader_uuid: "c75831edb3b44dd59c00ef151adca49d" committed_config { opid_index: 258 OBSOLETE_local: false peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } } }
I20240502 21:26:21.569916 18451 tablet_service.cc:1514] Processing DeleteTablet for tablet 51edafdf2a9348548a494119668bfd31 with delete_type TABLET_DATA_TOMBSTONED (TS faadabfb099b4abf90a466395be585b5 not found in new config with opid_index 258) from {username='slave'} at 127.0.0.1:39378
I20240502 21:26:21.570268 21425 tablet_replica.cc:330] T 51edafdf2a9348548a494119668bfd31 P faadabfb099b4abf90a466395be585b5: stopping tablet replica
I20240502 21:26:21.570446 21425 raft_consensus.cc:2229] T 51edafdf2a9348548a494119668bfd31 P faadabfb099b4abf90a466395be585b5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:26:21.570703 21425 raft_consensus.cc:2258] T 51edafdf2a9348548a494119668bfd31 P faadabfb099b4abf90a466395be585b5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:21.571764 21425 ts_tablet_manager.cc:1875] T 51edafdf2a9348548a494119668bfd31 P faadabfb099b4abf90a466395be585b5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:26:21.573290 20166 raft_consensus.cc:2932] T 51edafdf2a9348548a494119668bfd31 P 90e329b457c24caf93a9f880345eb288 [term 1 FOLLOWER]: Committing config change with OpId 1.258: config changed from index 251 to 258, VOTER faadabfb099b4abf90a466395be585b5 (127.15.255.129) evicted. New config: { opid_index: 258 OBSOLETE_local: false peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } } }
I20240502 21:26:21.578848 21425 ts_tablet_manager.cc:1888] T 51edafdf2a9348548a494119668bfd31 P faadabfb099b4abf90a466395be585b5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.257
I20240502 21:26:21.579021 21425 log.cc:1196] T 51edafdf2a9348548a494119668bfd31 P faadabfb099b4abf90a466395be585b5: Deleting WAL directory at /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-0/wal/wals/51edafdf2a9348548a494119668bfd31
I20240502 21:26:21.583148 18313 catalog_manager.cc:4830] TS faadabfb099b4abf90a466395be585b5 (127.15.255.129:32983): tablet 51edafdf2a9348548a494119668bfd31 (table rebalance_test_table_2 [id=59b7e3f7d6ce4a008110576d81d3cf39]) successfully deleted
I20240502 21:26:21.592155 18314 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 51edafdf2a9348548a494119668bfd31 with cas_config_opid_index 251: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:26:21.595312 18608 raft_consensus.cc:1261] T 51edafdf2a9348548a494119668bfd31 P 82f50f4b66f3450aa9f58a0b317f5cea [term 1 FOLLOWER]: Refusing update from remote peer c75831edb3b44dd59c00ef151adca49d: Log matching property violated. Preceding OpId in replica: term: 1 index: 257. Preceding OpId from leader: term: 1 index: 258. (index mismatch)
I20240502 21:26:21.596678 19139 raft_consensus.cc:1261] T 51edafdf2a9348548a494119668bfd31 P 2877108756304e05bd60ba1186b870da [term 1 FOLLOWER]: Refusing update from remote peer c75831edb3b44dd59c00ef151adca49d: Log matching property violated. Preceding OpId in replica: term: 1 index: 257. Preceding OpId from leader: term: 1 index: 258. (index mismatch)
I20240502 21:26:21.601553 21384 consensus_queue.cc:1035] T 51edafdf2a9348548a494119668bfd31 P c75831edb3b44dd59c00ef151adca49d [LEADER]: Connected to new peer: Peer: permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 258, Last known committed idx: 257, Time since last communication: 0.000s
I20240502 21:26:21.602038 21384 consensus_queue.cc:1035] T 51edafdf2a9348548a494119668bfd31 P c75831edb3b44dd59c00ef151adca49d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 258, Last known committed idx: 257, Time since last communication: 0.000s
I20240502 21:26:21.607511 18608 raft_consensus.cc:2932] T 51edafdf2a9348548a494119668bfd31 P 82f50f4b66f3450aa9f58a0b317f5cea [term 1 FOLLOWER]: Committing config change with OpId 1.258: config changed from index 251 to 258, VOTER faadabfb099b4abf90a466395be585b5 (127.15.255.129) evicted. New config: { opid_index: 258 OBSOLETE_local: false peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } } }
I20240502 21:26:21.609877 19139 raft_consensus.cc:2932] T 51edafdf2a9348548a494119668bfd31 P 2877108756304e05bd60ba1186b870da [term 1 FOLLOWER]: Committing config change with OpId 1.258: config changed from index 251 to 258, VOTER faadabfb099b4abf90a466395be585b5 (127.15.255.129) evicted. New config: { opid_index: 258 OBSOLETE_local: false peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } } }
I20240502 21:26:21.665043 21424 ts_tablet_manager.cc:1875] T 32c9fc5a18c54207881124fda86f42ff P 2877108756304e05bd60ba1186b870da: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20240502 21:26:21.868844 18544 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.129:32983, user_credentials={real_user=slave}} blocked reactor thread for 381150us
I20240502 21:26:21.968484 21424 ts_tablet_manager.cc:1888] T 32c9fc5a18c54207881124fda86f42ff P 2877108756304e05bd60ba1186b870da: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.227
I20240502 21:26:21.968648 21424 log.cc:1196] T 32c9fc5a18c54207881124fda86f42ff P 2877108756304e05bd60ba1186b870da: Deleting WAL directory at /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-5/wal/wals/32c9fc5a18c54207881124fda86f42ff
I20240502 21:26:21.988646 18315 catalog_manager.cc:4830] TS 2877108756304e05bd60ba1186b870da (127.15.255.134:43635): tablet 32c9fc5a18c54207881124fda86f42ff (table rebalance_test_table_0 [id=086ede98fc604333833bce759f342073]) successfully deleted
W20240502 21:26:22.040720 21059 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.553s user 0.009s sys 0.003s
W20240502 21:26:22.079730 21059 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.573s user 0.009s sys 0.003s
W20240502 21:26:22.138703 18812 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.134:43635, user_credentials={real_user=slave}} blocked reactor thread for 78476.4us
W20240502 21:26:22.143162 21254 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.653s user 0.059s sys 0.076s
W20240502 21:26:22.143363 21254 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.653s user 0.059s sys 0.076s
W20240502 21:26:22.205089 18812 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.130:45139, user_credentials={real_user=slave}} blocked reactor thread for 49175.8us
W20240502 21:26:22.528892 18544 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.129:32983, user_credentials={real_user=slave}} blocked reactor thread for 89680.5us
W20240502 21:26:22.825645 18945 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.135:43395, user_credentials={real_user=slave}} blocked reactor thread for 131168us
W20240502 21:26:22.839804 18946 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.134:43635, user_credentials={real_user=slave}} blocked reactor thread for 47871.3us
W20240502 21:26:22.984241 18406 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.132:43607, user_credentials={real_user=slave}} blocked reactor thread for 98786.5us
W20240502 21:26:23.032248 18406 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.132:43607, user_credentials={real_user=slave}} blocked reactor thread for 46356.4us
W20240502 21:26:23.120965 18945 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.135:43395, user_credentials={real_user=slave}} blocked reactor thread for 64677.4us
W20240502 21:26:23.312402 21405 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.682s user 0.015s sys 0.020s
W20240502 21:26:23.462036 18406 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.135:43395, user_credentials={real_user=slave}} blocked reactor thread for 336989us
W20240502 21:26:23.482043 21405 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.852s user 0.015s sys 0.020s
W20240502 21:26:23.662000 18406 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.135:43395, user_credentials={real_user=slave}} blocked reactor thread for 91823.7us
W20240502 21:26:23.824620 18406 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.132:43607, user_credentials={real_user=slave}} blocked reactor thread for 124481us
W20240502 21:26:23.894452 18405 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.133:34217, user_credentials={real_user=slave}} blocked reactor thread for 121568us
W20240502 21:26:24.250628 18393 kernel_stack_watchdog.cc:198] Thread 21506 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:637 for 351ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x7f66fe762980 (unknown) at ??:0
@ 0x90ef77 __sanitizer::internal_sched_yield() at ??:?
@ 0x87462d __interceptor_pthread_create at ??:?
@ 0x7f66fb3d0d11 kudu::Thread::StartThread() at ??:0
@ 0xa4d187 kudu::Thread::Create() at ??:?
@ 0x7f66fb3f17ab kudu::ThreadPool::CreateThread() at ??:0
@ 0x7f66fb3ed3ad kudu::ThreadPool::DoSubmit() at ??:0
@ 0x7f66fb3eb84f kudu::ThreadPoolToken::Submit() at ??:0
@ 0x7f6706b1bd65 kudu::consensus::Peer::SignalRequest() at ??:0
@ 0x7f6706baf47d kudu::consensus::PeerManager::SignalRequest() at ??:0
@ 0x7f6706bf21d4 kudu::consensus::RaftConsensus::Replicate() at ??:0
@ 0x7f6707f5ab03 kudu::tablet::OpDriver::Prepare() at ??:0
@ 0x7f6707f5962a kudu::tablet::OpDriver::PrepareTask() at ??:0
@ 0x7f6707f6153c kudu::tablet::OpDriver::ExecuteAsync()::$_1::operator()() at ??:0
@ 0x7f6707f613b9 std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
W20240502 21:26:24.381824 21506 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.608s user 0.001s sys 0.012s
W20240502 21:26:24.381927 21506 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.608s user 0.001s sys 0.012s
W20240502 21:26:24.760685 18945 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.132:43607, user_credentials={real_user=slave}} blocked reactor thread for 59779.6us
W20240502 21:26:24.768779 18812 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.130:45139, user_credentials={real_user=slave}} blocked reactor thread for 54822us
W20240502 21:26:25.152392 18544 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.129:32983, user_credentials={real_user=slave}} blocked reactor thread for 66950.6us
W20240502 21:26:26.583822 18812 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.130:45139, user_credentials={real_user=slave}} blocked reactor thread for 136020us
W20240502 21:26:27.147639 18811 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.135:43395, user_credentials={real_user=slave}} blocked reactor thread for 282766us
W20240502 21:26:27.470944 18812 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.130:45139, user_credentials={real_user=slave}} blocked reactor thread for 630725us
W20240502 21:26:27.513698 20164 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.609s user 0.007s sys 0.002s
W20240502 21:26:27.513911 20164 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.609s user 0.007s sys 0.002s
W20240502 21:26:27.518431 21594 thread.cc:636] apply [worker] (thread pool) Time spent creating pthread: real 0.632s user 0.001s sys 0.010s
W20240502 21:26:27.518623 21594 thread.cc:602] apply [worker] (thread pool) Time spent starting thread: real 0.634s user 0.001s sys 0.010s
W20240502 21:26:27.524524 21483 thread.cc:636] apply [worker] (thread pool) Time spent creating pthread: real 0.640s user 0.055s sys 0.084s
W20240502 21:26:27.524674 21483 thread.cc:602] apply [worker] (thread pool) Time spent starting thread: real 0.640s user 0.055s sys 0.084s
W20240502 21:26:27.608053 20165 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.718s user 0.005s sys 0.005s
W20240502 21:26:27.608269 20165 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.718s user 0.005s sys 0.005s
W20240502 21:26:27.622653 18812 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.130:45139, user_credentials={real_user=slave}} blocked reactor thread for 90921.2us
I20240502 21:26:27.917753 18722 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:26:27.951081 18590 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:26:28.069749 19124 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 21:26:28.299055 18678 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.129:32983, user_credentials={real_user=slave}} blocked reactor thread for 99696.8us
I20240502 21:26:28.360874 20147 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 21:26:28.364279 18679 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.132:43607, user_credentials={real_user=slave}} blocked reactor thread for 88398.4us
I20240502 21:26:28.401283 18857 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:26:28.415994 18989 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:26:28.611796 18451 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
W20240502 21:26:28.761531 18812 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.134:43635, user_credentials={real_user=slave}} blocked reactor thread for 53121.8us
W20240502 21:26:28.844044 18810 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.129:32983, user_credentials={real_user=slave}} blocked reactor thread for 51908.1us
W20240502 21:26:29.078397 18406 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.135:43395, user_credentials={real_user=slave}} blocked reactor thread for 52546.8us
W20240502 21:26:29.135663 18406 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.135:43395, user_credentials={real_user=slave}} blocked reactor thread for 54087.9us
I20240502 21:26:29.245258 19005 consensus_queue.cc:237] T 4aff104e3b6b424baa628bbe612b62e8 P 9be399440ded4e2ebe6b6e0bf486cbb1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 234, Committed index: 234, Last appended: 1.234, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 235 OBSOLETE_local: false peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } attrs { replace: true } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: NON_VOTER last_known_addr { host: "127.15.255.130" port: 45139 } attrs { promote: true } }
I20240502 21:26:29.255029 18873 raft_consensus.cc:1261] T 4aff104e3b6b424baa628bbe612b62e8 P c75831edb3b44dd59c00ef151adca49d [term 1 FOLLOWER]: Refusing update from remote peer 9be399440ded4e2ebe6b6e0bf486cbb1: Log matching property violated. Preceding OpId in replica: term: 1 index: 234. Preceding OpId from leader: term: 1 index: 235. (index mismatch)
I20240502 21:26:29.258190 20169 raft_consensus.cc:1261] T 4aff104e3b6b424baa628bbe612b62e8 P 90e329b457c24caf93a9f880345eb288 [term 1 FOLLOWER]: Refusing update from remote peer 9be399440ded4e2ebe6b6e0bf486cbb1: Log matching property violated. Preceding OpId in replica: term: 1 index: 234. Preceding OpId from leader: term: 1 index: 235. (index mismatch)
I20240502 21:26:29.258545 21684 consensus_queue.cc:1035] T 4aff104e3b6b424baa628bbe612b62e8 P 9be399440ded4e2ebe6b6e0bf486cbb1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 235, Last known committed idx: 234, Time since last communication: 0.000s
I20240502 21:26:29.264966 19144 raft_consensus.cc:1261] T 4aff104e3b6b424baa628bbe612b62e8 P 2877108756304e05bd60ba1186b870da [term 1 FOLLOWER]: Refusing update from remote peer 9be399440ded4e2ebe6b6e0bf486cbb1: Log matching property violated. Preceding OpId in replica: term: 1 index: 234. Preceding OpId from leader: term: 1 index: 235. (index mismatch)
I20240502 21:26:29.265604 21685 consensus_queue.cc:1035] T 4aff104e3b6b424baa628bbe612b62e8 P 9be399440ded4e2ebe6b6e0bf486cbb1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 235, Last known committed idx: 234, Time since last communication: 0.000s
I20240502 21:26:29.268283 21685 consensus_queue.cc:1035] T 4aff104e3b6b424baa628bbe612b62e8 P 9be399440ded4e2ebe6b6e0bf486cbb1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 235, Last known committed idx: 234, Time since last communication: 0.000s
I20240502 21:26:29.273416 21645 raft_consensus.cc:2932] T 4aff104e3b6b424baa628bbe612b62e8 P 9be399440ded4e2ebe6b6e0bf486cbb1 [term 1 LEADER]: Committing config change with OpId 1.235: config changed from index 183 to 235, NON_VOTER 82f50f4b66f3450aa9f58a0b317f5cea (127.15.255.130) added. New config: { opid_index: 235 OBSOLETE_local: false peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } attrs { replace: true } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: NON_VOTER last_known_addr { host: "127.15.255.130" port: 45139 } attrs { promote: true } } }
I20240502 21:26:29.275075 18469 raft_consensus.cc:1261] T 4aff104e3b6b424baa628bbe612b62e8 P faadabfb099b4abf90a466395be585b5 [term 1 FOLLOWER]: Refusing update from remote peer 9be399440ded4e2ebe6b6e0bf486cbb1: Log matching property violated. Preceding OpId in replica: term: 1 index: 234. Preceding OpId from leader: term: 1 index: 235. (index mismatch)
I20240502 21:26:29.278245 19144 raft_consensus.cc:2932] T 4aff104e3b6b424baa628bbe612b62e8 P 2877108756304e05bd60ba1186b870da [term 1 FOLLOWER]: Committing config change with OpId 1.235: config changed from index 183 to 235, NON_VOTER 82f50f4b66f3450aa9f58a0b317f5cea (127.15.255.130) added. New config: { opid_index: 235 OBSOLETE_local: false peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } attrs { replace: true } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: NON_VOTER last_known_addr { host: "127.15.255.130" port: 45139 } attrs { promote: true } } }
I20240502 21:26:29.285239 21570 consensus_queue.cc:1035] T 4aff104e3b6b424baa628bbe612b62e8 P 9be399440ded4e2ebe6b6e0bf486cbb1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 235, Last known committed idx: 234, Time since last communication: 0.000s
I20240502 21:26:29.291025 18469 raft_consensus.cc:2932] T 4aff104e3b6b424baa628bbe612b62e8 P faadabfb099b4abf90a466395be585b5 [term 1 FOLLOWER]: Committing config change with OpId 1.235: config changed from index 183 to 235, NON_VOTER 82f50f4b66f3450aa9f58a0b317f5cea (127.15.255.130) added. New config: { opid_index: 235 OBSOLETE_local: false peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } attrs { replace: true } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: NON_VOTER last_known_addr { host: "127.15.255.130" port: 45139 } attrs { promote: true } } }
I20240502 21:26:29.291625 18877 raft_consensus.cc:2932] T 4aff104e3b6b424baa628bbe612b62e8 P c75831edb3b44dd59c00ef151adca49d [term 1 FOLLOWER]: Committing config change with OpId 1.235: config changed from index 183 to 235, NON_VOTER 82f50f4b66f3450aa9f58a0b317f5cea (127.15.255.130) added. New config: { opid_index: 235 OBSOLETE_local: false peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } attrs { replace: true } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: NON_VOTER last_known_addr { host: "127.15.255.130" port: 45139 } attrs { promote: true } } }
I20240502 21:26:29.293344 18321 catalog_manager.cc:5484] T 4aff104e3b6b424baa628bbe612b62e8 P 9be399440ded4e2ebe6b6e0bf486cbb1 reported cstate change: config changed from index 183 to 235, NON_VOTER 82f50f4b66f3450aa9f58a0b317f5cea (127.15.255.130) added. New cstate: current_term: 1 leader_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" committed_config { opid_index: 235 OBSOLETE_local: false peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: NON_VOTER last_known_addr { host: "127.15.255.130" port: 45139 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 21:26:29.298143 20162 raft_consensus.cc:2932] T 4aff104e3b6b424baa628bbe612b62e8 P 90e329b457c24caf93a9f880345eb288 [term 1 FOLLOWER]: Committing config change with OpId 1.235: config changed from index 183 to 235, NON_VOTER 82f50f4b66f3450aa9f58a0b317f5cea (127.15.255.130) added. New config: { opid_index: 235 OBSOLETE_local: false peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } attrs { replace: true } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: NON_VOTER last_known_addr { host: "127.15.255.130" port: 45139 } attrs { promote: true } } }
W20240502 21:26:29.306385 18946 consensus_peers.cc:485] T 4aff104e3b6b424baa628bbe612b62e8 P 9be399440ded4e2ebe6b6e0bf486cbb1 -> Peer 82f50f4b66f3450aa9f58a0b317f5cea (127.15.255.130:45139): Couldn't send request to peer 82f50f4b66f3450aa9f58a0b317f5cea. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4aff104e3b6b424baa628bbe612b62e8. This is attempt 1: this message will repeat every 5th retry.
W20240502 21:26:29.529912 18404 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.131:42501, user_credentials={real_user=slave}} blocked reactor thread for 72322.6us
W20240502 21:26:29.530354 18405 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.133:34217, user_credentials={real_user=slave}} blocked reactor thread for 70961.3us
W20240502 21:26:29.531085 18406 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.135:43395, user_credentials={real_user=slave}} blocked reactor thread for 75107.2us
W20240502 21:26:29.538475 18407 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.134:43635, user_credentials={real_user=slave}} blocked reactor thread for 79390.3us
I20240502 21:26:29.748234 21716 ts_tablet_manager.cc:902] T 4aff104e3b6b424baa628bbe612b62e8 P 82f50f4b66f3450aa9f58a0b317f5cea: Initiating tablet copy from peer 9be399440ded4e2ebe6b6e0bf486cbb1 (127.15.255.133:34217)
I20240502 21:26:29.989701 21716 tablet_copy_client.cc:308] T 4aff104e3b6b424baa628bbe612b62e8 P 82f50f4b66f3450aa9f58a0b317f5cea: tablet copy: Beginning tablet copy session from remote peer at address 127.15.255.133:34217
I20240502 21:26:29.992164 19033 tablet_copy_service.cc:140] P 9be399440ded4e2ebe6b6e0bf486cbb1: Received BeginTabletCopySession request for tablet 4aff104e3b6b424baa628bbe612b62e8 from peer 82f50f4b66f3450aa9f58a0b317f5cea ({username='slave'} at 127.15.255.130:46223)
I20240502 21:26:29.992305 19033 tablet_copy_service.cc:161] P 9be399440ded4e2ebe6b6e0bf486cbb1: Beginning new tablet copy session on tablet 4aff104e3b6b424baa628bbe612b62e8 from peer 82f50f4b66f3450aa9f58a0b317f5cea at {username='slave'} at 127.15.255.130:46223: session id = 82f50f4b66f3450aa9f58a0b317f5cea-4aff104e3b6b424baa628bbe612b62e8
I20240502 21:26:29.994977 19033 tablet_copy_source_session.cc:205] T 4aff104e3b6b424baa628bbe612b62e8 P 9be399440ded4e2ebe6b6e0bf486cbb1: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:26:30.004105 21716 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4aff104e3b6b424baa628bbe612b62e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:26:30.062433 21716 tablet_copy_client.cc:783] T 4aff104e3b6b424baa628bbe612b62e8 P 82f50f4b66f3450aa9f58a0b317f5cea: tablet copy: Starting download of 0 data blocks...
I20240502 21:26:30.068461 21716 tablet_copy_client.cc:647] T 4aff104e3b6b424baa628bbe612b62e8 P 82f50f4b66f3450aa9f58a0b317f5cea: tablet copy: Starting download of 1 WAL segments...
W20240502 21:26:30.127024 18544 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.133:34217, user_credentials={real_user=slave}} blocked reactor thread for 63166.1us
I20240502 21:26:30.206763 21716 tablet_copy_client.cc:523] T 4aff104e3b6b424baa628bbe612b62e8 P 82f50f4b66f3450aa9f58a0b317f5cea: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:26:30.212651 21716 tablet_bootstrap.cc:492] T 4aff104e3b6b424baa628bbe612b62e8 P 82f50f4b66f3450aa9f58a0b317f5cea: Bootstrap starting.
W20240502 21:26:30.387528 18812 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.130:45139, user_credentials={real_user=slave}} blocked reactor thread for 54940.7us
W20240502 21:26:30.391652 18544 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.129:32983, user_credentials={real_user=slave}} blocked reactor thread for 55392.9us
W20240502 21:26:30.478422 18406 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.135:43395, user_credentials={real_user=slave}} blocked reactor thread for 59359.8us
W20240502 21:26:30.693758 18406 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.135:43395, user_credentials={real_user=slave}} blocked reactor thread for 123239us
W20240502 21:26:30.750088 18406 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.135:43395, user_credentials={real_user=slave}} blocked reactor thread for 47053.3us
W20240502 21:26:30.949347 18680 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.134:43635, user_credentials={real_user=slave}} blocked reactor thread for 56468.8us
I20240502 21:26:32.161378 21716 tablet_bootstrap.cc:492] T 4aff104e3b6b424baa628bbe612b62e8 P 82f50f4b66f3450aa9f58a0b317f5cea: Bootstrap replayed 1/1 log segments. Stats: ops{read=237 overwritten=0 applied=237 ignored=0} inserts{seen=865 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:26:32.168924 21716 tablet_bootstrap.cc:492] T 4aff104e3b6b424baa628bbe612b62e8 P 82f50f4b66f3450aa9f58a0b317f5cea: Bootstrap complete.
W20240502 21:26:32.169155 18946 consensus_peers.cc:485] T 4aff104e3b6b424baa628bbe612b62e8 P 9be399440ded4e2ebe6b6e0bf486cbb1 -> Peer 82f50f4b66f3450aa9f58a0b317f5cea (127.15.255.130:45139): Couldn't send request to peer 82f50f4b66f3450aa9f58a0b317f5cea. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 6: this message will repeat every 5th retry.
I20240502 21:26:32.169958 21716 ts_tablet_manager.cc:1367] T 4aff104e3b6b424baa628bbe612b62e8 P 82f50f4b66f3450aa9f58a0b317f5cea: Time spent bootstrapping tablet: real 1.958s user 0.303s sys 0.022s
I20240502 21:26:32.170856 21716 raft_consensus.cc:348] T 4aff104e3b6b424baa628bbe612b62e8 P 82f50f4b66f3450aa9f58a0b317f5cea [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 235 OBSOLETE_local: false peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } attrs { replace: true } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: NON_VOTER last_known_addr { host: "127.15.255.130" port: 45139 } attrs { promote: true } }
I20240502 21:26:32.171653 21716 raft_consensus.cc:729] T 4aff104e3b6b424baa628bbe612b62e8 P 82f50f4b66f3450aa9f58a0b317f5cea [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 82f50f4b66f3450aa9f58a0b317f5cea, State: Initialized, Role: LEARNER
I20240502 21:26:32.171979 21716 consensus_queue.cc:260] T 4aff104e3b6b424baa628bbe612b62e8 P 82f50f4b66f3450aa9f58a0b317f5cea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 237, Last appended: 1.237, Last appended by leader: 237, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 235 OBSOLETE_local: false peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } attrs { replace: true } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: NON_VOTER last_known_addr { host: "127.15.255.130" port: 45139 } attrs { promote: true } }
I20240502 21:26:32.178092 21716 ts_tablet_manager.cc:1398] T 4aff104e3b6b424baa628bbe612b62e8 P 82f50f4b66f3450aa9f58a0b317f5cea: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
W20240502 21:26:32.190003 18680 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.130:45139, user_credentials={real_user=slave}} blocked reactor thread for 53274.3us
I20240502 21:26:32.200585 19033 tablet_copy_service.cc:342] P 9be399440ded4e2ebe6b6e0bf486cbb1: Request end of tablet copy session 82f50f4b66f3450aa9f58a0b317f5cea-4aff104e3b6b424baa628bbe612b62e8 received from {username='slave'} at 127.15.255.130:46223
I20240502 21:26:32.200736 19033 tablet_copy_service.cc:433] P 9be399440ded4e2ebe6b6e0bf486cbb1: ending tablet copy session 82f50f4b66f3450aa9f58a0b317f5cea-4aff104e3b6b424baa628bbe612b62e8 on tablet 4aff104e3b6b424baa628bbe612b62e8 with peer 82f50f4b66f3450aa9f58a0b317f5cea
W20240502 21:26:32.276826 18544 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.129:32983, user_credentials={real_user=slave}} blocked reactor thread for 74896.7us
W20240502 21:26:32.284292 18680 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.134:43635, user_credentials={real_user=slave}} blocked reactor thread for 66752us
W20240502 21:26:32.423259 18544 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.129:32983, user_credentials={real_user=slave}} blocked reactor thread for 45503.5us
I20240502 21:26:32.821889 18611 raft_consensus.cc:1203] T 4aff104e3b6b424baa628bbe612b62e8 P 82f50f4b66f3450aa9f58a0b317f5cea [term 1 LEARNER]: Deduplicated request from leader. Original: 1.234->[1.235-1.246] Dedup: 1.237->[1.238-1.246]
W20240502 21:26:33.047607 18406 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.135:43395, user_credentials={real_user=slave}} blocked reactor thread for 56428.9us
I20240502 21:26:33.085204 21728 raft_consensus.cc:1050] T 4aff104e3b6b424baa628bbe612b62e8 P 9be399440ded4e2ebe6b6e0bf486cbb1: attempting to promote NON_VOTER 82f50f4b66f3450aa9f58a0b317f5cea to VOTER
I20240502 21:26:33.086093 21728 consensus_queue.cc:237] T 4aff104e3b6b424baa628bbe612b62e8 P 9be399440ded4e2ebe6b6e0bf486cbb1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 247, Committed index: 247, Last appended: 1.247, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 248 OBSOLETE_local: false peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } attrs { replace: true } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } attrs { promote: false } }
I20240502 21:26:33.089138 20168 raft_consensus.cc:1261] T 4aff104e3b6b424baa628bbe612b62e8 P 90e329b457c24caf93a9f880345eb288 [term 1 FOLLOWER]: Refusing update from remote peer 9be399440ded4e2ebe6b6e0bf486cbb1: Log matching property violated. Preceding OpId in replica: term: 1 index: 247. Preceding OpId from leader: term: 1 index: 248. (index mismatch)
I20240502 21:26:33.089890 21570 consensus_queue.cc:1035] T 4aff104e3b6b424baa628bbe612b62e8 P 9be399440ded4e2ebe6b6e0bf486cbb1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 248, Last known committed idx: 247, Time since last communication: 0.000s
I20240502 21:26:33.091109 18609 raft_consensus.cc:1261] T 4aff104e3b6b424baa628bbe612b62e8 P 82f50f4b66f3450aa9f58a0b317f5cea [term 1 LEARNER]: Refusing update from remote peer 9be399440ded4e2ebe6b6e0bf486cbb1: Log matching property violated. Preceding OpId in replica: term: 1 index: 247. Preceding OpId from leader: term: 1 index: 248. (index mismatch)
I20240502 21:26:33.091598 21570 consensus_queue.cc:1035] T 4aff104e3b6b424baa628bbe612b62e8 P 9be399440ded4e2ebe6b6e0bf486cbb1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 248, Last known committed idx: 247, Time since last communication: 0.000s
I20240502 21:26:33.100687 21570 raft_consensus.cc:1013] T 4aff104e3b6b424baa628bbe612b62e8 P 9be399440ded4e2ebe6b6e0bf486cbb1 [term 1 LEADER]: attempt to promote peer 82f50f4b66f3450aa9f58a0b317f5cea: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240502 21:26:33.114818 19146 raft_consensus.cc:1261] T 4aff104e3b6b424baa628bbe612b62e8 P 2877108756304e05bd60ba1186b870da [term 1 FOLLOWER]: Refusing update from remote peer 9be399440ded4e2ebe6b6e0bf486cbb1: Log matching property violated. Preceding OpId in replica: term: 1 index: 247. Preceding OpId from leader: term: 1 index: 248. (index mismatch)
I20240502 21:26:33.116132 21806 consensus_queue.cc:1035] T 4aff104e3b6b424baa628bbe612b62e8 P 9be399440ded4e2ebe6b6e0bf486cbb1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 248, Last known committed idx: 247, Time since last communication: 0.000s
I20240502 21:26:33.119385 21570 raft_consensus.cc:2932] T 4aff104e3b6b424baa628bbe612b62e8 P 9be399440ded4e2ebe6b6e0bf486cbb1 [term 1 LEADER]: Committing config change with OpId 1.248: config changed from index 235 to 248, 82f50f4b66f3450aa9f58a0b317f5cea (127.15.255.130) changed from NON_VOTER to VOTER. New config: { opid_index: 248 OBSOLETE_local: false peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } attrs { replace: true } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } attrs { promote: false } } }
I20240502 21:26:33.124718 19140 raft_consensus.cc:2932] T 4aff104e3b6b424baa628bbe612b62e8 P 2877108756304e05bd60ba1186b870da [term 1 FOLLOWER]: Committing config change with OpId 1.248: config changed from index 235 to 248, 82f50f4b66f3450aa9f58a0b317f5cea (127.15.255.130) changed from NON_VOTER to VOTER. New config: { opid_index: 248 OBSOLETE_local: false peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } attrs { replace: true } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } attrs { promote: false } } }
I20240502 21:26:33.131172 18327 catalog_manager.cc:5484] T 4aff104e3b6b424baa628bbe612b62e8 P 2877108756304e05bd60ba1186b870da reported cstate change: config changed from index 235 to 248, 82f50f4b66f3450aa9f58a0b317f5cea (127.15.255.130) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" committed_config { opid_index: 248 OBSOLETE_local: false peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } attrs { replace: true } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } attrs { promote: false } } }
W20240502 21:26:33.131827 18407 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.134:43635, user_credentials={real_user=slave}} blocked reactor thread for 48549.2us
I20240502 21:26:33.132858 18469 raft_consensus.cc:1261] T 4aff104e3b6b424baa628bbe612b62e8 P faadabfb099b4abf90a466395be585b5 [term 1 FOLLOWER]: Refusing update from remote peer 9be399440ded4e2ebe6b6e0bf486cbb1: Log matching property violated. Preceding OpId in replica: term: 1 index: 247. Preceding OpId from leader: term: 1 index: 248. (index mismatch)
W20240502 21:26:33.133410 18405 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.133:34217, user_credentials={real_user=slave}} blocked reactor thread for 54293.9us
I20240502 21:26:33.144351 18874 raft_consensus.cc:1261] T 4aff104e3b6b424baa628bbe612b62e8 P c75831edb3b44dd59c00ef151adca49d [term 1 FOLLOWER]: Refusing update from remote peer 9be399440ded4e2ebe6b6e0bf486cbb1: Log matching property violated. Preceding OpId in replica: term: 1 index: 247. Preceding OpId from leader: term: 1 index: 249. (index mismatch)
I20240502 21:26:33.144882 21802 consensus_queue.cc:1035] T 4aff104e3b6b424baa628bbe612b62e8 P 9be399440ded4e2ebe6b6e0bf486cbb1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 248, Last known committed idx: 247, Time since last communication: 0.000s
I20240502 21:26:33.145359 21802 consensus_queue.cc:1035] T 4aff104e3b6b424baa628bbe612b62e8 P 9be399440ded4e2ebe6b6e0bf486cbb1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 248, Last known committed idx: 247, Time since last communication: 0.000s
I20240502 21:26:33.147241 18874 raft_consensus.cc:2932] T 4aff104e3b6b424baa628bbe612b62e8 P c75831edb3b44dd59c00ef151adca49d [term 1 FOLLOWER]: Committing config change with OpId 1.248: config changed from index 235 to 248, 82f50f4b66f3450aa9f58a0b317f5cea (127.15.255.130) changed from NON_VOTER to VOTER. New config: { opid_index: 248 OBSOLETE_local: false peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } attrs { replace: true } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } attrs { promote: false } } }
I20240502 21:26:33.147378 18469 raft_consensus.cc:2932] T 4aff104e3b6b424baa628bbe612b62e8 P faadabfb099b4abf90a466395be585b5 [term 1 FOLLOWER]: Committing config change with OpId 1.248: config changed from index 235 to 248, 82f50f4b66f3450aa9f58a0b317f5cea (127.15.255.130) changed from NON_VOTER to VOTER. New config: { opid_index: 248 OBSOLETE_local: false peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } attrs { replace: true } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } attrs { promote: false } } }
I20240502 21:26:33.155079 20168 raft_consensus.cc:2932] T 4aff104e3b6b424baa628bbe612b62e8 P 90e329b457c24caf93a9f880345eb288 [term 1 FOLLOWER]: Committing config change with OpId 1.248: config changed from index 235 to 248, 82f50f4b66f3450aa9f58a0b317f5cea (127.15.255.130) changed from NON_VOTER to VOTER. New config: { opid_index: 248 OBSOLETE_local: false peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } attrs { replace: true } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } attrs { promote: false } } }
I20240502 21:26:33.163417 18609 raft_consensus.cc:2932] T 4aff104e3b6b424baa628bbe612b62e8 P 82f50f4b66f3450aa9f58a0b317f5cea [term 1 FOLLOWER]: Committing config change with OpId 1.248: config changed from index 235 to 248, 82f50f4b66f3450aa9f58a0b317f5cea (127.15.255.130) changed from NON_VOTER to VOTER. New config: { opid_index: 248 OBSOLETE_local: false peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } attrs { replace: true } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } attrs { promote: false } } }
I20240502 21:26:33.469367 19008 consensus_queue.cc:237] T 4aff104e3b6b424baa628bbe612b62e8 P 9be399440ded4e2ebe6b6e0bf486cbb1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 252, Committed index: 252, Last appended: 1.252, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 253 OBSOLETE_local: false peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } attrs { promote: false } }
I20240502 21:26:33.479256 18610 raft_consensus.cc:1261] T 4aff104e3b6b424baa628bbe612b62e8 P 82f50f4b66f3450aa9f58a0b317f5cea [term 1 FOLLOWER]: Refusing update from remote peer 9be399440ded4e2ebe6b6e0bf486cbb1: Log matching property violated. Preceding OpId in replica: term: 1 index: 252. Preceding OpId from leader: term: 1 index: 253. (index mismatch)
I20240502 21:26:33.479911 21570 consensus_queue.cc:1035] T 4aff104e3b6b424baa628bbe612b62e8 P 9be399440ded4e2ebe6b6e0bf486cbb1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 253, Last known committed idx: 252, Time since last communication: 0.000s
I20240502 21:26:33.487955 19141 raft_consensus.cc:1261] T 4aff104e3b6b424baa628bbe612b62e8 P 2877108756304e05bd60ba1186b870da [term 1 FOLLOWER]: Refusing update from remote peer 9be399440ded4e2ebe6b6e0bf486cbb1: Log matching property violated. Preceding OpId in replica: term: 1 index: 252. Preceding OpId from leader: term: 1 index: 253. (index mismatch)
I20240502 21:26:33.489579 18472 raft_consensus.cc:1261] T 4aff104e3b6b424baa628bbe612b62e8 P faadabfb099b4abf90a466395be585b5 [term 1 FOLLOWER]: Refusing update from remote peer 9be399440ded4e2ebe6b6e0bf486cbb1: Log matching property violated. Preceding OpId in replica: term: 1 index: 252. Preceding OpId from leader: term: 1 index: 253. (index mismatch)
I20240502 21:26:33.490733 21570 consensus_queue.cc:1035] T 4aff104e3b6b424baa628bbe612b62e8 P 9be399440ded4e2ebe6b6e0bf486cbb1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 253, Last known committed idx: 252, Time since last communication: 0.000s
I20240502 21:26:33.491287 21570 consensus_queue.cc:1035] T 4aff104e3b6b424baa628bbe612b62e8 P 9be399440ded4e2ebe6b6e0bf486cbb1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 253, Last known committed idx: 252, Time since last communication: 0.000s
I20240502 21:26:33.496573 20168 raft_consensus.cc:1261] T 4aff104e3b6b424baa628bbe612b62e8 P 90e329b457c24caf93a9f880345eb288 [term 1 FOLLOWER]: Refusing update from remote peer 9be399440ded4e2ebe6b6e0bf486cbb1: Log matching property violated. Preceding OpId in replica: term: 1 index: 251. Preceding OpId from leader: term: 1 index: 253. (index mismatch)
I20240502 21:26:33.496801 21825 raft_consensus.cc:2932] T 4aff104e3b6b424baa628bbe612b62e8 P 9be399440ded4e2ebe6b6e0bf486cbb1 [term 1 LEADER]: Committing config change with OpId 1.253: config changed from index 248 to 253, VOTER c75831edb3b44dd59c00ef151adca49d (127.15.255.132) evicted. New config: { opid_index: 253 OBSOLETE_local: false peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } attrs { promote: false } } }
I20240502 21:26:33.504865 19146 raft_consensus.cc:2932] T 4aff104e3b6b424baa628bbe612b62e8 P 2877108756304e05bd60ba1186b870da [term 1 FOLLOWER]: Committing config change with OpId 1.253: config changed from index 248 to 253, VOTER c75831edb3b44dd59c00ef151adca49d (127.15.255.132) evicted. New config: { opid_index: 253 OBSOLETE_local: false peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } attrs { promote: false } } }
I20240502 21:26:33.505988 18329 catalog_manager.cc:5484] T 4aff104e3b6b424baa628bbe612b62e8 P 9be399440ded4e2ebe6b6e0bf486cbb1 reported cstate change: config changed from index 248 to 253, VOTER c75831edb3b44dd59c00ef151adca49d (127.15.255.132) evicted. New cstate: current_term: 1 leader_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" committed_config { opid_index: 253 OBSOLETE_local: false peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:26:33.511953 18313 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 4aff104e3b6b424baa628bbe612b62e8 with cas_config_opid_index 248: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:26:33.513195 21570 consensus_queue.cc:1035] T 4aff104e3b6b424baa628bbe612b62e8 P 9be399440ded4e2ebe6b6e0bf486cbb1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 253, Last known committed idx: 250, Time since last communication: 0.000s
I20240502 21:26:33.527328 18610 raft_consensus.cc:2932] T 4aff104e3b6b424baa628bbe612b62e8 P 82f50f4b66f3450aa9f58a0b317f5cea [term 1 FOLLOWER]: Committing config change with OpId 1.253: config changed from index 248 to 253, VOTER c75831edb3b44dd59c00ef151adca49d (127.15.255.132) evicted. New config: { opid_index: 253 OBSOLETE_local: false peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } attrs { promote: false } } }
I20240502 21:26:33.560381 18857 tablet_service.cc:1514] Processing DeleteTablet for tablet 4aff104e3b6b424baa628bbe612b62e8 with delete_type TABLET_DATA_TOMBSTONED (TS c75831edb3b44dd59c00ef151adca49d not found in new config with opid_index 253) from {username='slave'} at 127.0.0.1:38486
I20240502 21:26:33.586510 18472 raft_consensus.cc:2932] T 4aff104e3b6b424baa628bbe612b62e8 P faadabfb099b4abf90a466395be585b5 [term 1 FOLLOWER]: Committing config change with OpId 1.253: config changed from index 248 to 253, VOTER c75831edb3b44dd59c00ef151adca49d (127.15.255.132) evicted. New config: { opid_index: 253 OBSOLETE_local: false peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } attrs { promote: false } } }
I20240502 21:26:33.625416 20162 raft_consensus.cc:2932] T 4aff104e3b6b424baa628bbe612b62e8 P 90e329b457c24caf93a9f880345eb288 [term 1 FOLLOWER]: Committing config change with OpId 1.253: config changed from index 248 to 253, VOTER c75831edb3b44dd59c00ef151adca49d (127.15.255.132) evicted. New config: { opid_index: 253 OBSOLETE_local: false peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } attrs { promote: false } } }
I20240502 21:26:33.682416 21839 tablet_replica.cc:330] T 4aff104e3b6b424baa628bbe612b62e8 P c75831edb3b44dd59c00ef151adca49d: stopping tablet replica
I20240502 21:26:33.682678 21839 raft_consensus.cc:2229] T 4aff104e3b6b424baa628bbe612b62e8 P c75831edb3b44dd59c00ef151adca49d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:26:33.682951 21839 raft_consensus.cc:2258] T 4aff104e3b6b424baa628bbe612b62e8 P c75831edb3b44dd59c00ef151adca49d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:26:33.706317 21839 ts_tablet_manager.cc:1875] T 4aff104e3b6b424baa628bbe612b62e8 P c75831edb3b44dd59c00ef151adca49d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:26:33.712311 21839 ts_tablet_manager.cc:1888] T 4aff104e3b6b424baa628bbe612b62e8 P c75831edb3b44dd59c00ef151adca49d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.252
I20240502 21:26:33.779309 21839 log.cc:1196] T 4aff104e3b6b424baa628bbe612b62e8 P c75831edb3b44dd59c00ef151adca49d: Deleting WAL directory at /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-3/wal/wals/4aff104e3b6b424baa628bbe612b62e8
I20240502 21:26:33.782322 18314 catalog_manager.cc:4830] TS c75831edb3b44dd59c00ef151adca49d (127.15.255.132:43607): tablet 4aff104e3b6b424baa628bbe612b62e8 (table rebalance_test_table_3 [id=5af23c390f054d07b5a7985c2611ca30]) successfully deleted
I20240502 21:26:34.557232 18722 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:26:34.678893 20147 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:26:34.706038 18857 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:26:34.813069 18451 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:26:34.969039 18590 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:26:35.061525 19124 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:26:35.116714 18989 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:26:35.446628 19077 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.133:34217, user_credentials={real_user=slave}} blocked reactor thread for 61017.1us
W20240502 21:26:35.549357 18406 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.135:43395, user_credentials={real_user=slave}} blocked reactor thread for 68298.7us
W20240502 21:26:35.583317 19079 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.130:45139, user_credentials={real_user=slave}} blocked reactor thread for 177364us
W20240502 21:26:35.623620 18680 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.130:45139, user_credentials={real_user=slave}} blocked reactor thread for 52698.1us
W20240502 21:26:36.465584 18407 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.134:43635, user_credentials={real_user=slave}} blocked reactor thread for 72411.8us
W20240502 21:26:36.665710 18812 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.134:43635, user_credentials={real_user=slave}} blocked reactor thread for 61942us
W20240502 21:26:36.828148 18406 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.135:43395, user_credentials={real_user=slave}} blocked reactor thread for 72194.6us
W20240502 21:26:36.884827 18407 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.134:43635, user_credentials={real_user=slave}} blocked reactor thread for 43728.9us
W20240502 21:26:37.114555 18406 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.135:43395, user_credentials={real_user=slave}} blocked reactor thread for 56346.3us
W20240502 21:26:37.277989 19077 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.133:34217, user_credentials={real_user=slave}} blocked reactor thread for 64443.7us
W20240502 21:26:37.427698 18543 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.131:42501, user_credentials={real_user=slave}} blocked reactor thread for 66599.9us
W20240502 21:26:37.497022 18406 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.132:43607, user_credentials={real_user=slave}} blocked reactor thread for 101607us
W20240502 21:26:37.618801 18407 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.130:45139, user_credentials={real_user=slave}} blocked reactor thread for 44378.5us
W20240502 21:26:37.720595 18407 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.130:45139, user_credentials={real_user=slave}} blocked reactor thread for 72254.1us
W20240502 21:26:37.721812 18406 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.135:43395, user_credentials={real_user=slave}} blocked reactor thread for 67228.5us
W20240502 21:26:37.812268 22001 log.cc:924] Time spent T 1df9d3ad5ddc438798543900b6c67b46 P 2877108756304e05bd60ba1186b870da: Append to log took a long time: real 0.060s user 0.000s sys 0.000s
W20240502 21:26:38.000561 18407 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.130:45139, user_credentials={real_user=slave}} blocked reactor thread for 90464.1us
W20240502 21:26:38.199257 18680 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.130:45139, user_credentials={real_user=slave}} blocked reactor thread for 361006us
W20240502 21:26:38.277915 21932 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.583s user 0.031s sys 0.085s
W20240502 21:26:38.278098 21932 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.583s user 0.031s sys 0.085s
W20240502 21:26:38.457791 18680 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.134:43635, user_credentials={real_user=slave}} blocked reactor thread for 256232us
W20240502 21:26:38.916930 18407 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.130:45139, user_credentials={real_user=slave}} blocked reactor thread for 44281us
W20240502 21:26:39.070523 18406 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.132:43607, user_credentials={real_user=slave}} blocked reactor thread for 56882.7us
I20240502 21:26:39.295500 18989 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:26:39.314870 18722 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:26:39.369872 18451 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:26:39.382620 18857 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 21:26:39.499037 18678 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.129:32983, user_credentials={real_user=slave}} blocked reactor thread for 52948.8us
W20240502 21:26:39.502465 18680 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.130:45139, user_credentials={real_user=slave}} blocked reactor thread for 64237.5us
I20240502 21:26:39.538635 18590 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 21:26:39.545781 18406 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.135:43395, user_credentials={real_user=slave}} blocked reactor thread for 44161us
I20240502 21:26:39.802052 20147 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:26:39.938341 19124 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 21:26:40.188047 18406 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.135:43395, user_credentials={real_user=slave}} blocked reactor thread for 61522.3us
W20240502 21:26:41.051118 18811 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.135:43395, user_credentials={real_user=slave}} blocked reactor thread for 70348.3us
W20240502 21:26:41.854036 18544 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.129:32983, user_credentials={real_user=slave}} blocked reactor thread for 74721.2us
W20240502 21:26:41.897727 19076 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.131:42501, user_credentials={real_user=slave}} blocked reactor thread for 142231us
W20240502 21:26:41.963449 18544 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.129:32983, user_credentials={real_user=slave}} blocked reactor thread for 51882.6us
W20240502 21:26:41.964382 18812 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.134:43635, user_credentials={real_user=slave}} blocked reactor thread for 49668.6us
W20240502 21:26:42.166388 18544 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.133:34217, user_credentials={real_user=slave}} blocked reactor thread for 77380.9us
W20240502 21:26:42.322450 18544 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.129:32983, user_credentials={real_user=slave}} blocked reactor thread for 68963.6us
W20240502 21:26:42.481212 18544 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.129:32983, user_credentials={real_user=slave}} blocked reactor thread for 81655.8us
W20240502 21:26:42.555224 18544 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.133:34217, user_credentials={real_user=slave}} blocked reactor thread for 73666us
W20240502 21:26:42.710264 18812 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.134:43635, user_credentials={real_user=slave}} blocked reactor thread for 75763.9us
W20240502 21:26:42.793375 18812 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.134:43635, user_credentials={real_user=slave}} blocked reactor thread for 82797us
W20240502 21:26:43.601846 18680 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.134:43635, user_credentials={real_user=slave}} blocked reactor thread for 45866.9us
W20240502 21:26:44.031159 18407 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.130:45139, user_credentials={real_user=slave}} blocked reactor thread for 48179.9us
W20240502 21:26:44.134972 18407 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.130:45139, user_credentials={real_user=slave}} blocked reactor thread for 102637us
W20240502 21:26:44.769538 18404 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.131:42501, user_credentials={real_user=slave}} blocked reactor thread for 44650.2us
W20240502 21:26:45.331221 18405 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.133:34217, user_credentials={real_user=slave}} blocked reactor thread for 188570us
W20240502 21:26:45.619258 18931 kernel_stack_watchdog.cc:198] Thread 18946 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:308 for 152ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x7f4e208ed980 (unknown) at ??:0
@ 0x90ef77 __sanitizer::internal_sched_yield() at ??:?
@ 0x87462d __interceptor_pthread_create at ??:?
@ 0x7f4e1d55bd11 kudu::Thread::StartThread() at ??:0
@ 0xa4d187 kudu::Thread::Create() at ??:?
@ 0x7f4e1d57c7ab kudu::ThreadPool::CreateThread() at ??:0
@ 0x7f4e1d5783ad kudu::ThreadPool::DoSubmit() at ??:0
@ 0x7f4e1d57684f kudu::ThreadPoolToken::Submit() at ??:0
@ 0x7f4e28ca99e8 kudu::consensus::Peer::ProcessResponse() at ??:0
@ 0x7f4e28cb0061 kudu::consensus::Peer::SendNextRequest()::$_3::operator()() at ??:0
@ 0x7f4e28cafe6e std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
@ 0x7f4e241ad8cf kudu::rpc::Proxy::AsyncRequest()::$_1::operator()() at ??:0
@ 0x7f4e241ad4ae std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
@ 0x7f4e241994e0 kudu::rpc::OutboundCall::CallCallback() at ??:0
W20240502 21:26:45.651619 18811 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.135:43395, user_credentials={real_user=slave}} blocked reactor thread for 233227us
W20240502 21:26:45.677384 18871 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.515s user 0.007s sys 0.004s
W20240502 21:26:45.677500 18871 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.515s user 0.007s sys 0.004s
W20240502 21:26:45.695928 18946 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.134:43635, user_credentials={real_user=slave}} blocked reactor thread for 389404us
W20240502 21:26:45.842721 22216 thread.cc:636] apply [worker] (thread pool) Time spent creating pthread: real 0.671s user 0.006s sys 0.004s
W20240502 21:26:45.843024 22216 thread.cc:602] apply [worker] (thread pool) Time spent starting thread: real 0.671s user 0.006s sys 0.004s
W20240502 21:26:45.922098 18878 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.751s user 0.042s sys 0.087s
W20240502 21:26:45.922206 18878 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.751s user 0.042s sys 0.087s
I20240502 21:26:45.958909 19124 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:26:45.991328 18722 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 21:26:46.015165 18812 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.134:43635, user_credentials={real_user=slave}} blocked reactor thread for 44411.6us
I20240502 21:26:46.021116 18590 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 21:26:46.128921 18812 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.130:45139, user_credentials={real_user=slave}} blocked reactor thread for 85523us
I20240502 21:26:46.147835 20147 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 21:26:46.158229 18811 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.135:43395, user_credentials={real_user=slave}} blocked reactor thread for 54730.9us
I20240502 21:26:46.185573 18451 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:26:46.258424 18989 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:26:46.291602 18857 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 21:26:46.956653 18405 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.133:34217, user_credentials={real_user=slave}} blocked reactor thread for 81000.2us
W20240502 21:26:46.966493 18812 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.134:43635, user_credentials={real_user=slave}} blocked reactor thread for 60962.6us
W20240502 21:26:47.368528 18543 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.131:42501, user_credentials={real_user=slave}} blocked reactor thread for 47895.8us
W20240502 21:26:48.022189 18407 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.130:45139, user_credentials={real_user=slave}} blocked reactor thread for 61069.6us
W20240502 21:26:49.424083 18544 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.129:32983, user_credentials={real_user=slave}} blocked reactor thread for 54487.4us
I20240502 21:26:49.521631 19124 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240502 21:26:49.707953 18544 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.129:32983, user_credentials={real_user=slave}} blocked reactor thread for 47797.1us
W20240502 21:26:49.717519 18405 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.133:34217, user_credentials={real_user=slave}} blocked reactor thread for 77051.7us
I20240502 21:26:49.852762 18857 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:26:49.907177 18722 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:26:49.972252 20147 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:26:49.999543 18590 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:26:50.071831 18451 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:26:50.131829 18989 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240502 21:26:50.181561 18546 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.134:43635, user_credentials={real_user=slave}} blocked reactor thread for 83953.7us
W20240502 21:26:50.670179 18943 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.131:42501, user_credentials={real_user=slave}} blocked reactor thread for 197172us
W20240502 21:26:51.723297 18406 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.132:43607, user_credentials={real_user=slave}} blocked reactor thread for 63693.4us
W20240502 21:26:51.939697 18543 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.131:42501, user_credentials={real_user=slave}} blocked reactor thread for 207441us
W20240502 21:26:52.019047 18543 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.131:42501, user_credentials={real_user=slave}} blocked reactor thread for 63548.5us
W20240502 21:26:52.226666 18543 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.131:42501, user_credentials={real_user=slave}} blocked reactor thread for 78233.5us
W20240502 21:26:52.329414 18406 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.135:43395, user_credentials={real_user=slave}} blocked reactor thread for 74191.8us
I20240502 21:26:52.933208 19124 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:26:52.934908 18722 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:26:52.947537 18590 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:26:52.968356 20147 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:26:52.973441 18989 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:26:52.979144 18857 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:26:52.993894 18451 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
132c427694cc4f858dd86045029e179e | 127.15.255.190:36947 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
allow_unsafe_replication_factor | true | runtime,unsafe | all 1 server(s) checked
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714685120732928-16382-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.15.255.148:40185 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+---------+----------+----------------+-----------------
2877108756304e05bd60ba1186b870da | 127.15.255.134:43635 | HEALTHY | <none> | 1 | 0
501e3c0d44fe4d70af9ff926d9622417 | 127.15.255.131:42501 | HEALTHY | <none> | 1 | 0
82f50f4b66f3450aa9f58a0b317f5cea | 127.15.255.130:45139 | HEALTHY | <none> | 3 | 0
90e329b457c24caf93a9f880345eb288 | 127.15.255.135:43395 | HEALTHY | <none> | 0 | 0
9be399440ded4e2ebe6b6e0bf486cbb1 | 127.15.255.133:34217 | HEALTHY | <none> | 2 | 0
c75831edb3b44dd59c00ef151adca49d | 127.15.255.132:43607 | HEALTHY | <none> | 3 | 0
faadabfb099b4abf90a466395be585b5 | 127.15.255.129:32983 | HEALTHY | <none> | 6 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 7
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 7 server(s) checked
local_ip_for_outbound_sockets | 127.15.255.129 | experimental | 127.15.255.129:32983
local_ip_for_outbound_sockets | 127.15.255.130 | experimental | 127.15.255.130:45139
local_ip_for_outbound_sockets | 127.15.255.131 | experimental | 127.15.255.131:42501
local_ip_for_outbound_sockets | 127.15.255.132 | experimental | 127.15.255.132:43607
local_ip_for_outbound_sockets | 127.15.255.133 | experimental | 127.15.255.133:34217
local_ip_for_outbound_sockets | 127.15.255.134 | experimental | 127.15.255.134:43635
local_ip_for_outbound_sockets | 127.15.255.135 | experimental | 127.15.255.135:43395
never_fsync | true | unsafe,advanced | all 7 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 7 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 7 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 7 server(s) checked
server_dump_info_format | pb | hidden | all 7 server(s) checked
server_dump_info_path | /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.15.255.129:32983
server_dump_info_path | /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.15.255.130:45139
server_dump_info_path | /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.15.255.131:42501
server_dump_info_path | /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.15.255.132:43607
server_dump_info_path | /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.15.255.133:34217
server_dump_info_path | /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.15.255.134:43635
server_dump_info_path | /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-6/data/info.pb | hidden | 127.15.255.135:43395
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.15.255.148:40185 | all 7 server(s) checked
time_source | builtin | all 7 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 8 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------------------+----+---------+---------------+---------+------------+------------------+-------------
TestTable | 5 | HEALTHY | 1 | 1 | 0 | 0 | 0
rebalance_test_table_0 | 5 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_1 | 5 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_2 | 5 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_3 | 5 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_4 | 5 | HEALTHY | 3 | 3 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 11
First Quartile | 11
Median | 11
Third Quartile | 12
Maximum | 12
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 7
Tables | 6
Tablets | 16
Replicas | 80
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20240502 21:26:53.075551 16382 log_verifier.cc:126] Checking tablet 1df9d3ad5ddc438798543900b6c67b46
I20240502 21:26:53.594338 16382 log_verifier.cc:177] Verified matching terms for 501 ops in tablet 1df9d3ad5ddc438798543900b6c67b46
I20240502 21:26:53.595558 16382 log_verifier.cc:126] Checking tablet 2d111096a9af46068a2476a6231d6520
I20240502 21:26:54.121145 16382 log_verifier.cc:177] Verified matching terms for 600 ops in tablet 2d111096a9af46068a2476a6231d6520
I20240502 21:26:54.122627 16382 log_verifier.cc:126] Checking tablet 32c9fc5a18c54207881124fda86f42ff
I20240502 21:26:54.592170 16382 log_verifier.cc:177] Verified matching terms for 534 ops in tablet 32c9fc5a18c54207881124fda86f42ff
I20240502 21:26:54.593410 16382 log_verifier.cc:126] Checking tablet 37e9dcc6191c42d3a5c1503dedc5511e
I20240502 21:26:55.048831 16382 log_verifier.cc:177] Verified matching terms for 531 ops in tablet 37e9dcc6191c42d3a5c1503dedc5511e
I20240502 21:26:55.050040 16382 log_verifier.cc:126] Checking tablet 40d4f61d0f6d4d31a5626c7743c88133
I20240502 21:26:55.087395 16382 log_verifier.cc:177] Verified matching terms for 1 ops in tablet 40d4f61d0f6d4d31a5626c7743c88133
I20240502 21:26:55.087486 16382 log_verifier.cc:126] Checking tablet 4aff104e3b6b424baa628bbe612b62e8
I20240502 21:26:55.453177 16382 log_verifier.cc:177] Verified matching terms for 410 ops in tablet 4aff104e3b6b424baa628bbe612b62e8
I20240502 21:26:55.454111 16382 log_verifier.cc:126] Checking tablet 51edafdf2a9348548a494119668bfd31
I20240502 21:26:55.958045 16382 log_verifier.cc:177] Verified matching terms for 584 ops in tablet 51edafdf2a9348548a494119668bfd31
I20240502 21:26:55.959371 16382 log_verifier.cc:126] Checking tablet 693a8666bbe54ae1b774f7f4da5ba67e
I20240502 21:26:56.476081 16382 log_verifier.cc:177] Verified matching terms for 583 ops in tablet 693a8666bbe54ae1b774f7f4da5ba67e
I20240502 21:26:56.477460 16382 log_verifier.cc:126] Checking tablet 6f0107b0690c4c64b0a4f2472a1c8322
I20240502 21:26:56.921643 16382 log_verifier.cc:177] Verified matching terms for 500 ops in tablet 6f0107b0690c4c64b0a4f2472a1c8322
I20240502 21:26:56.923125 16382 log_verifier.cc:126] Checking tablet 84b235fd89634168938c0dddb534462a
I20240502 21:26:57.400624 16382 log_verifier.cc:177] Verified matching terms for 571 ops in tablet 84b235fd89634168938c0dddb534462a
I20240502 21:26:57.401919 16382 log_verifier.cc:126] Checking tablet 9844e983dd0a403abe0ac90cd96690d6
I20240502 21:26:57.778208 16382 log_verifier.cc:177] Verified matching terms for 405 ops in tablet 9844e983dd0a403abe0ac90cd96690d6
I20240502 21:26:57.779163 16382 log_verifier.cc:126] Checking tablet a7a9256f2b784abcb780b190e2ff3983
I20240502 21:26:58.251787 16382 log_verifier.cc:177] Verified matching terms for 510 ops in tablet a7a9256f2b784abcb780b190e2ff3983
I20240502 21:26:58.253106 16382 log_verifier.cc:126] Checking tablet b0852f54089944ce935f8ee4cbf2bc5a
I20240502 21:26:58.636085 16382 log_verifier.cc:177] Verified matching terms for 452 ops in tablet b0852f54089944ce935f8ee4cbf2bc5a
I20240502 21:26:58.637041 16382 log_verifier.cc:126] Checking tablet e0e69b88ad014363b97d1f650df693a8
I20240502 21:26:59.110916 16382 log_verifier.cc:177] Verified matching terms for 508 ops in tablet e0e69b88ad014363b97d1f650df693a8
I20240502 21:26:59.112989 16382 log_verifier.cc:126] Checking tablet ebf7371ff0ef4de99ced8fec0468af28
I20240502 21:26:59.479624 16382 log_verifier.cc:177] Verified matching terms for 386 ops in tablet ebf7371ff0ef4de99ced8fec0468af28
I20240502 21:26:59.480494 16382 log_verifier.cc:126] Checking tablet ed78d40318fe4ab5bee4a01b182a34b6
I20240502 21:26:59.908354 16382 log_verifier.cc:177] Verified matching terms for 497 ops in tablet ed78d40318fe4ab5bee4a01b182a34b6
I20240502 21:26:59.918640 16382 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvNciQZ/build/asan/bin/kudu
/tmp/dist-test-taskvNciQZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.255.136:0
--local_ip_for_outbound_sockets=127.15.255.136
--webserver_interface=127.15.255.136
--webserver_port=0
--tserver_master_addrs=127.15.255.190:36947
--builtin_ntp_servers=127.15.255.148:40185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240502 21:27:00.265722 22551 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:27:00.266019 22551 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:27:00.266217 22551 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:27:00.274508 22551 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 21:27:00.274618 22551 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:27:00.274734 22551 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.255.136
I20240502 21:27:00.283551 22551 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.255.148:40185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.255.136:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.15.255.136
--webserver_port=0
--tserver_master_addrs=127.15.255.190:36947
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.255.136
--log_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:27:00.284972 22551 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:27:00.286697 22551 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:27:00.298190 22556 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:27:00.298907 22559 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:27:00.299373 22557 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:27:00.299976 22551 server_base.cc:1034] running on GCE node
I20240502 21:27:00.300582 22551 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:27:00.301434 22551 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:27:00.302628 22551 hybrid_clock.cc:648] HybridClock initialized: now 1714685220302585 us; error 36 us; skew 500 ppm
I20240502 21:27:00.302982 22551 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:27:00.305265 22551 webserver.cc:460] Webserver started at http://127.15.255.136:43311/ using document root <none> and password file <none>
I20240502 21:27:00.306100 22551 fs_manager.cc:359] Metadata directory not provided
I20240502 21:27:00.306202 22551 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:27:00.306564 22551 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:27:00.309402 22551 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-7/data/instance:
uuid: "bc20559563fe44efa0a2b7b084aafbce"
format_stamp: "Formatted at 2024-05-02 21:27:00 on dist-test-slave-jdfx"
I20240502 21:27:00.310155 22551 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-7/wal/instance:
uuid: "bc20559563fe44efa0a2b7b084aafbce"
format_stamp: "Formatted at 2024-05-02 21:27:00 on dist-test-slave-jdfx"
I20240502 21:27:00.315424 22551 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20240502 21:27:00.318734 22565 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:27:00.319967 22551 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240502 21:27:00.320104 22551 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "bc20559563fe44efa0a2b7b084aafbce"
format_stamp: "Formatted at 2024-05-02 21:27:00 on dist-test-slave-jdfx"
I20240502 21:27:00.320274 22551 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:27:00.337447 22551 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:27:00.338224 22551 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:27:00.338514 22551 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:27:00.339390 22551 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:27:00.341089 22551 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:27:00.341209 22551 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:27:00.341323 22551 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:27:00.341413 22551 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 21:27:00.371897 22551 rpc_server.cc:307] RPC server started. Bound to: 127.15.255.136:34965
I20240502 21:27:00.371963 22677 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.255.136:34965 every 8 connection(s)
I20240502 21:27:00.373302 22551 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240502 21:27:00.383003 16382 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvNciQZ/build/asan/bin/kudu as pid 22551
I20240502 21:27:00.383200 16382 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-7/wal/instance
I20240502 21:27:00.385830 22678 heartbeater.cc:347] Connected to a master server at 127.15.255.190:36947
I20240502 21:27:00.386085 22678 heartbeater.cc:464] Registering TS with master...
I20240502 21:27:00.386869 22678 heartbeater.cc:511] Master 127.15.255.190:36947 requested a full tablet report, sending...
I20240502 21:27:00.388137 18329 ts_manager.cc:194] Registered new tserver with Master: bc20559563fe44efa0a2b7b084aafbce (127.15.255.136:34965)
I20240502 21:27:00.389206 18329 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.255.136:53397
I20240502 21:27:00.911142 18722 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:27:00.914916 19124 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:27:00.916296 18857 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:27:00.917788 20147 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:00.919582 18590 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:27:00.931427 18989 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:27:00.936609 22613 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:00.949445 18451 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:27:01.167029 18590 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:27:01.172781 18722 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:27:01.174877 19124 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:27:01.176815 20147 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:01.187098 18451 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:27:01.197310 18857 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:27:01.204685 22613 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:01.235512 18989 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:27:01.314540 18744 consensus_queue.cc:237] T 1df9d3ad5ddc438798543900b6c67b46 P 501e3c0d44fe4d70af9ff926d9622417 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 501, Committed index: 501, Last appended: 1.501, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 502 OBSOLETE_local: false peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } attrs { replace: true } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: true } }
I20240502 21:27:01.317575 18876 raft_consensus.cc:1261] T 1df9d3ad5ddc438798543900b6c67b46 P c75831edb3b44dd59c00ef151adca49d [term 1 FOLLOWER]: Refusing update from remote peer 501e3c0d44fe4d70af9ff926d9622417: Log matching property violated. Preceding OpId in replica: term: 1 index: 501. Preceding OpId from leader: term: 1 index: 502. (index mismatch)
I20240502 21:27:01.318315 19146 raft_consensus.cc:1261] T 1df9d3ad5ddc438798543900b6c67b46 P 2877108756304e05bd60ba1186b870da [term 1 FOLLOWER]: Refusing update from remote peer 501e3c0d44fe4d70af9ff926d9622417: Log matching property violated. Preceding OpId in replica: term: 1 index: 501. Preceding OpId from leader: term: 1 index: 502. (index mismatch)
I20240502 21:27:01.319036 22467 consensus_queue.cc:1035] T 1df9d3ad5ddc438798543900b6c67b46 P 501e3c0d44fe4d70af9ff926d9622417 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 502, Last known committed idx: 501, Time since last communication: 0.000s
I20240502 21:27:01.319664 18611 raft_consensus.cc:1261] T 1df9d3ad5ddc438798543900b6c67b46 P 82f50f4b66f3450aa9f58a0b317f5cea [term 1 FOLLOWER]: Refusing update from remote peer 501e3c0d44fe4d70af9ff926d9622417: Log matching property violated. Preceding OpId in replica: term: 1 index: 501. Preceding OpId from leader: term: 1 index: 502. (index mismatch)
I20240502 21:27:01.320202 18472 raft_consensus.cc:1261] T 1df9d3ad5ddc438798543900b6c67b46 P faadabfb099b4abf90a466395be585b5 [term 1 FOLLOWER]: Refusing update from remote peer 501e3c0d44fe4d70af9ff926d9622417: Log matching property violated. Preceding OpId in replica: term: 1 index: 501. Preceding OpId from leader: term: 1 index: 502. (index mismatch)
I20240502 21:27:01.320402 22774 consensus_queue.cc:1035] T 1df9d3ad5ddc438798543900b6c67b46 P 501e3c0d44fe4d70af9ff926d9622417 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 502, Last known committed idx: 501, Time since last communication: 0.000s
I20240502 21:27:01.328545 22774 consensus_queue.cc:1035] T 1df9d3ad5ddc438798543900b6c67b46 P 501e3c0d44fe4d70af9ff926d9622417 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 502, Last known committed idx: 501, Time since last communication: 0.000s
I20240502 21:27:01.333719 22777 consensus_queue.cc:1035] T 1df9d3ad5ddc438798543900b6c67b46 P 501e3c0d44fe4d70af9ff926d9622417 [LEADER]: Connected to new peer: Peer: permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 502, Last known committed idx: 501, Time since last communication: 0.000s
W20240502 21:27:01.334134 18677 consensus_peers.cc:485] T 1df9d3ad5ddc438798543900b6c67b46 P 501e3c0d44fe4d70af9ff926d9622417 -> Peer bc20559563fe44efa0a2b7b084aafbce (127.15.255.136:34965): Couldn't send request to peer bc20559563fe44efa0a2b7b084aafbce. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1df9d3ad5ddc438798543900b6c67b46. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:01.350304 22467 raft_consensus.cc:2932] T 1df9d3ad5ddc438798543900b6c67b46 P 501e3c0d44fe4d70af9ff926d9622417 [term 1 LEADER]: Committing config change with OpId 1.502: config changed from index -1 to 502, NON_VOTER bc20559563fe44efa0a2b7b084aafbce (127.15.255.136) added. New config: { opid_index: 502 OBSOLETE_local: false peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } attrs { replace: true } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: true } } }
I20240502 21:27:01.350920 18876 raft_consensus.cc:2932] T 1df9d3ad5ddc438798543900b6c67b46 P c75831edb3b44dd59c00ef151adca49d [term 1 FOLLOWER]: Committing config change with OpId 1.502: config changed from index -1 to 502, NON_VOTER bc20559563fe44efa0a2b7b084aafbce (127.15.255.136) added. New config: { opid_index: 502 OBSOLETE_local: false peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } attrs { replace: true } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: true } } }
I20240502 21:27:01.353621 18472 raft_consensus.cc:2932] T 1df9d3ad5ddc438798543900b6c67b46 P faadabfb099b4abf90a466395be585b5 [term 1 FOLLOWER]: Committing config change with OpId 1.502: config changed from index -1 to 502, NON_VOTER bc20559563fe44efa0a2b7b084aafbce (127.15.255.136) added. New config: { opid_index: 502 OBSOLETE_local: false peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } attrs { replace: true } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: true } } }
I20240502 21:27:01.354827 19140 raft_consensus.cc:2932] T 1df9d3ad5ddc438798543900b6c67b46 P 2877108756304e05bd60ba1186b870da [term 1 FOLLOWER]: Committing config change with OpId 1.502: config changed from index -1 to 502, NON_VOTER bc20559563fe44efa0a2b7b084aafbce (127.15.255.136) added. New config: { opid_index: 502 OBSOLETE_local: false peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } attrs { replace: true } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: true } } }
I20240502 21:27:01.355404 18611 raft_consensus.cc:2932] T 1df9d3ad5ddc438798543900b6c67b46 P 82f50f4b66f3450aa9f58a0b317f5cea [term 1 FOLLOWER]: Committing config change with OpId 1.502: config changed from index -1 to 502, NON_VOTER bc20559563fe44efa0a2b7b084aafbce (127.15.255.136) added. New config: { opid_index: 502 OBSOLETE_local: false peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } attrs { replace: true } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: true } } }
I20240502 21:27:01.358723 18321 catalog_manager.cc:5484] T 1df9d3ad5ddc438798543900b6c67b46 P faadabfb099b4abf90a466395be585b5 reported cstate change: config changed from index -1 to 502, NON_VOTER bc20559563fe44efa0a2b7b084aafbce (127.15.255.136) added. New cstate: current_term: 1 leader_uuid: "501e3c0d44fe4d70af9ff926d9622417" committed_config { opid_index: 502 OBSOLETE_local: false peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } attrs { replace: true } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: true } } }
I20240502 21:27:01.381335 18876 consensus_queue.cc:237] T ebf7371ff0ef4de99ced8fec0468af28 P c75831edb3b44dd59c00ef151adca49d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 386, Committed index: 386, Last appended: 1.386, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 387 OBSOLETE_local: false peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } attrs { replace: true } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: true } }
I20240502 21:27:01.384581 20168 raft_consensus.cc:1261] T ebf7371ff0ef4de99ced8fec0468af28 P 90e329b457c24caf93a9f880345eb288 [term 1 FOLLOWER]: Refusing update from remote peer c75831edb3b44dd59c00ef151adca49d: Log matching property violated. Preceding OpId in replica: term: 1 index: 386. Preceding OpId from leader: term: 1 index: 387. (index mismatch)
I20240502 21:27:01.384583 19140 raft_consensus.cc:1261] T ebf7371ff0ef4de99ced8fec0468af28 P 2877108756304e05bd60ba1186b870da [term 1 FOLLOWER]: Refusing update from remote peer c75831edb3b44dd59c00ef151adca49d: Log matching property violated. Preceding OpId in replica: term: 1 index: 386. Preceding OpId from leader: term: 1 index: 387. (index mismatch)
I20240502 21:27:01.385200 22529 consensus_queue.cc:1035] T ebf7371ff0ef4de99ced8fec0468af28 P c75831edb3b44dd59c00ef151adca49d [LEADER]: Connected to new peer: Peer: permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 387, Last known committed idx: 386, Time since last communication: 0.000s
I20240502 21:27:01.385682 22529 consensus_queue.cc:1035] T ebf7371ff0ef4de99ced8fec0468af28 P c75831edb3b44dd59c00ef151adca49d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 387, Last known committed idx: 386, Time since last communication: 0.000s
I20240502 21:27:01.389389 18744 raft_consensus.cc:1261] T ebf7371ff0ef4de99ced8fec0468af28 P 501e3c0d44fe4d70af9ff926d9622417 [term 1 FOLLOWER]: Refusing update from remote peer c75831edb3b44dd59c00ef151adca49d: Log matching property violated. Preceding OpId in replica: term: 1 index: 386. Preceding OpId from leader: term: 1 index: 387. (index mismatch)
I20240502 21:27:01.389075 22529 raft_consensus.cc:2932] T ebf7371ff0ef4de99ced8fec0468af28 P c75831edb3b44dd59c00ef151adca49d [term 1 LEADER]: Committing config change with OpId 1.387: config changed from index 117 to 387, NON_VOTER bc20559563fe44efa0a2b7b084aafbce (127.15.255.136) added. New config: { opid_index: 387 OBSOLETE_local: false peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } attrs { replace: true } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: true } } }
I20240502 21:27:01.389572 19009 raft_consensus.cc:1261] T ebf7371ff0ef4de99ced8fec0468af28 P 9be399440ded4e2ebe6b6e0bf486cbb1 [term 1 FOLLOWER]: Refusing update from remote peer c75831edb3b44dd59c00ef151adca49d: Log matching property violated. Preceding OpId in replica: term: 1 index: 386. Preceding OpId from leader: term: 1 index: 387. (index mismatch)
I20240502 21:27:01.390334 22798 consensus_queue.cc:1035] T ebf7371ff0ef4de99ced8fec0468af28 P c75831edb3b44dd59c00ef151adca49d [LEADER]: Connected to new peer: Peer: permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 387, Last known committed idx: 386, Time since last communication: 0.000s
I20240502 21:27:01.392364 22678 heartbeater.cc:503] Master 127.15.255.190:36947 was elected leader, sending a full tablet report...
I20240502 21:27:01.396108 22793 consensus_queue.cc:1035] T ebf7371ff0ef4de99ced8fec0468af28 P c75831edb3b44dd59c00ef151adca49d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 387, Last known committed idx: 386, Time since last communication: 0.000s
I20240502 21:27:01.401674 20168 raft_consensus.cc:2932] T ebf7371ff0ef4de99ced8fec0468af28 P 90e329b457c24caf93a9f880345eb288 [term 1 FOLLOWER]: Committing config change with OpId 1.387: config changed from index 117 to 387, NON_VOTER bc20559563fe44efa0a2b7b084aafbce (127.15.255.136) added. New config: { opid_index: 387 OBSOLETE_local: false peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } attrs { replace: true } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: true } } }
I20240502 21:27:01.414701 19009 raft_consensus.cc:2932] T ebf7371ff0ef4de99ced8fec0468af28 P 9be399440ded4e2ebe6b6e0bf486cbb1 [term 1 FOLLOWER]: Committing config change with OpId 1.387: config changed from index 117 to 387, NON_VOTER bc20559563fe44efa0a2b7b084aafbce (127.15.255.136) added. New config: { opid_index: 387 OBSOLETE_local: false peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } attrs { replace: true } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: true } } }
I20240502 21:27:01.423938 18327 catalog_manager.cc:5484] T ebf7371ff0ef4de99ced8fec0468af28 P 9be399440ded4e2ebe6b6e0bf486cbb1 reported cstate change: config changed from index 117 to 387, NON_VOTER bc20559563fe44efa0a2b7b084aafbce (127.15.255.136) added. New cstate: current_term: 1 leader_uuid: "c75831edb3b44dd59c00ef151adca49d" committed_config { opid_index: 387 OBSOLETE_local: false peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } attrs { replace: true } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: true } } }
I20240502 21:27:01.431043 18744 raft_consensus.cc:2932] T ebf7371ff0ef4de99ced8fec0468af28 P 501e3c0d44fe4d70af9ff926d9622417 [term 1 FOLLOWER]: Committing config change with OpId 1.387: config changed from index 117 to 387, NON_VOTER bc20559563fe44efa0a2b7b084aafbce (127.15.255.136) added. New config: { opid_index: 387 OBSOLETE_local: false peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } attrs { replace: true } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: true } } }
W20240502 21:27:01.432518 18812 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.134:43635, user_credentials={real_user=slave}} blocked reactor thread for 43951.6us
W20240502 21:27:01.435645 18809 consensus_peers.cc:485] T ebf7371ff0ef4de99ced8fec0468af28 P c75831edb3b44dd59c00ef151adca49d -> Peer bc20559563fe44efa0a2b7b084aafbce (127.15.255.136:34965): Couldn't send request to peer bc20559563fe44efa0a2b7b084aafbce. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ebf7371ff0ef4de99ced8fec0468af28. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:01.438724 19140 raft_consensus.cc:2932] T ebf7371ff0ef4de99ced8fec0468af28 P 2877108756304e05bd60ba1186b870da [term 1 FOLLOWER]: Committing config change with OpId 1.387: config changed from index 117 to 387, NON_VOTER bc20559563fe44efa0a2b7b084aafbce (127.15.255.136) added. New config: { opid_index: 387 OBSOLETE_local: false peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } attrs { replace: true } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: true } } }
I20240502 21:27:01.460808 18611 consensus_queue.cc:237] T a7a9256f2b784abcb780b190e2ff3983 P 82f50f4b66f3450aa9f58a0b317f5cea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 510, Committed index: 510, Last appended: 1.510, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 511 OBSOLETE_local: false peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } attrs { replace: true } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: true } }
I20240502 21:27:01.463934 20168 raft_consensus.cc:1261] T a7a9256f2b784abcb780b190e2ff3983 P 90e329b457c24caf93a9f880345eb288 [term 1 FOLLOWER]: Refusing update from remote peer 82f50f4b66f3450aa9f58a0b317f5cea: Log matching property violated. Preceding OpId in replica: term: 1 index: 510. Preceding OpId from leader: term: 1 index: 511. (index mismatch)
I20240502 21:27:01.464046 18472 raft_consensus.cc:1261] T a7a9256f2b784abcb780b190e2ff3983 P faadabfb099b4abf90a466395be585b5 [term 1 FOLLOWER]: Refusing update from remote peer 82f50f4b66f3450aa9f58a0b317f5cea: Log matching property violated. Preceding OpId in replica: term: 1 index: 510. Preceding OpId from leader: term: 1 index: 511. (index mismatch)
I20240502 21:27:01.464620 18876 raft_consensus.cc:1261] T a7a9256f2b784abcb780b190e2ff3983 P c75831edb3b44dd59c00ef151adca49d [term 1 FOLLOWER]: Refusing update from remote peer 82f50f4b66f3450aa9f58a0b317f5cea: Log matching property violated. Preceding OpId in replica: term: 1 index: 510. Preceding OpId from leader: term: 1 index: 511. (index mismatch)
I20240502 21:27:01.464674 22811 consensus_queue.cc:1035] T a7a9256f2b784abcb780b190e2ff3983 P 82f50f4b66f3450aa9f58a0b317f5cea [LEADER]: Connected to new peer: Peer: permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 511, Last known committed idx: 510, Time since last communication: 0.000s
I20240502 21:27:01.465204 22532 consensus_queue.cc:1035] T a7a9256f2b784abcb780b190e2ff3983 P 82f50f4b66f3450aa9f58a0b317f5cea [LEADER]: Connected to new peer: Peer: permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 511, Last known committed idx: 510, Time since last communication: 0.000s
I20240502 21:27:01.465689 22532 consensus_queue.cc:1035] T a7a9256f2b784abcb780b190e2ff3983 P 82f50f4b66f3450aa9f58a0b317f5cea [LEADER]: Connected to new peer: Peer: permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 511, Last known committed idx: 510, Time since last communication: 0.000s
I20240502 21:27:01.467069 18744 raft_consensus.cc:1261] T a7a9256f2b784abcb780b190e2ff3983 P 501e3c0d44fe4d70af9ff926d9622417 [term 1 FOLLOWER]: Refusing update from remote peer 82f50f4b66f3450aa9f58a0b317f5cea: Log matching property violated. Preceding OpId in replica: term: 1 index: 510. Preceding OpId from leader: term: 1 index: 511. (index mismatch)
I20240502 21:27:01.467636 22812 consensus_queue.cc:1035] T a7a9256f2b784abcb780b190e2ff3983 P 82f50f4b66f3450aa9f58a0b317f5cea [LEADER]: Connected to new peer: Peer: permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 511, Last known committed idx: 510, Time since last communication: 0.000s
W20240502 21:27:01.473573 18543 consensus_peers.cc:485] T a7a9256f2b784abcb780b190e2ff3983 P 82f50f4b66f3450aa9f58a0b317f5cea -> Peer bc20559563fe44efa0a2b7b084aafbce (127.15.255.136:34965): Couldn't send request to peer bc20559563fe44efa0a2b7b084aafbce. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a7a9256f2b784abcb780b190e2ff3983. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:01.475598 22812 raft_consensus.cc:2932] T a7a9256f2b784abcb780b190e2ff3983 P 82f50f4b66f3450aa9f58a0b317f5cea [term 1 LEADER]: Committing config change with OpId 1.511: config changed from index 132 to 511, NON_VOTER bc20559563fe44efa0a2b7b084aafbce (127.15.255.136) added. New config: { opid_index: 511 OBSOLETE_local: false peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } attrs { replace: true } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: true } } }
I20240502 21:27:01.477203 18744 raft_consensus.cc:2932] T a7a9256f2b784abcb780b190e2ff3983 P 501e3c0d44fe4d70af9ff926d9622417 [term 1 FOLLOWER]: Committing config change with OpId 1.511: config changed from index 132 to 511, NON_VOTER bc20559563fe44efa0a2b7b084aafbce (127.15.255.136) added. New config: { opid_index: 511 OBSOLETE_local: false peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } attrs { replace: true } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: true } } }
I20240502 21:27:01.477290 18876 raft_consensus.cc:2932] T a7a9256f2b784abcb780b190e2ff3983 P c75831edb3b44dd59c00ef151adca49d [term 1 FOLLOWER]: Committing config change with OpId 1.511: config changed from index 132 to 511, NON_VOTER bc20559563fe44efa0a2b7b084aafbce (127.15.255.136) added. New config: { opid_index: 511 OBSOLETE_local: false peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } attrs { replace: true } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: true } } }
I20240502 21:27:01.478024 20167 raft_consensus.cc:2932] T a7a9256f2b784abcb780b190e2ff3983 P 90e329b457c24caf93a9f880345eb288 [term 1 FOLLOWER]: Committing config change with OpId 1.511: config changed from index 132 to 511, NON_VOTER bc20559563fe44efa0a2b7b084aafbce (127.15.255.136) added. New config: { opid_index: 511 OBSOLETE_local: false peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } attrs { replace: true } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: true } } }
I20240502 21:27:01.481853 18472 raft_consensus.cc:2932] T a7a9256f2b784abcb780b190e2ff3983 P faadabfb099b4abf90a466395be585b5 [term 1 FOLLOWER]: Committing config change with OpId 1.511: config changed from index 132 to 511, NON_VOTER bc20559563fe44efa0a2b7b084aafbce (127.15.255.136) added. New config: { opid_index: 511 OBSOLETE_local: false peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } attrs { replace: true } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: true } } }
I20240502 21:27:01.482743 18327 catalog_manager.cc:5484] T a7a9256f2b784abcb780b190e2ff3983 P c75831edb3b44dd59c00ef151adca49d reported cstate change: config changed from index 132 to 511, NON_VOTER bc20559563fe44efa0a2b7b084aafbce (127.15.255.136) added. New cstate: current_term: 1 leader_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" committed_config { opid_index: 511 OBSOLETE_local: false peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } attrs { replace: true } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: true } } }
I20240502 21:27:01.511766 18472 consensus_queue.cc:237] T 693a8666bbe54ae1b774f7f4da5ba67e P faadabfb099b4abf90a466395be585b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 583, Committed index: 583, Last appended: 1.583, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 584 OBSOLETE_local: false peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } attrs { replace: true } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: true } }
I20240502 21:27:01.514853 20167 raft_consensus.cc:1261] T 693a8666bbe54ae1b774f7f4da5ba67e P 90e329b457c24caf93a9f880345eb288 [term 1 FOLLOWER]: Refusing update from remote peer faadabfb099b4abf90a466395be585b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 583. Preceding OpId from leader: term: 1 index: 584. (index mismatch)
I20240502 21:27:01.515451 18744 raft_consensus.cc:1261] T 693a8666bbe54ae1b774f7f4da5ba67e P 501e3c0d44fe4d70af9ff926d9622417 [term 1 FOLLOWER]: Refusing update from remote peer faadabfb099b4abf90a466395be585b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 583. Preceding OpId from leader: term: 1 index: 584. (index mismatch)
I20240502 21:27:01.515673 22530 consensus_queue.cc:1035] T 693a8666bbe54ae1b774f7f4da5ba67e P faadabfb099b4abf90a466395be585b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 584, Last known committed idx: 583, Time since last communication: 0.000s
I20240502 21:27:01.516010 22820 consensus_queue.cc:1035] T 693a8666bbe54ae1b774f7f4da5ba67e P faadabfb099b4abf90a466395be585b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 584, Last known committed idx: 583, Time since last communication: 0.000s
W20240502 21:27:01.523149 18404 consensus_peers.cc:485] T 693a8666bbe54ae1b774f7f4da5ba67e P faadabfb099b4abf90a466395be585b5 -> Peer bc20559563fe44efa0a2b7b084aafbce (127.15.255.136:34965): Couldn't send request to peer bc20559563fe44efa0a2b7b084aafbce. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 693a8666bbe54ae1b774f7f4da5ba67e. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:01.526415 19140 raft_consensus.cc:1261] T 693a8666bbe54ae1b774f7f4da5ba67e P 2877108756304e05bd60ba1186b870da [term 1 FOLLOWER]: Refusing update from remote peer faadabfb099b4abf90a466395be585b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 583. Preceding OpId from leader: term: 1 index: 584. (index mismatch)
I20240502 21:27:01.526517 19009 raft_consensus.cc:1261] T 693a8666bbe54ae1b774f7f4da5ba67e P 9be399440ded4e2ebe6b6e0bf486cbb1 [term 1 FOLLOWER]: Refusing update from remote peer faadabfb099b4abf90a466395be585b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 583. Preceding OpId from leader: term: 1 index: 584. (index mismatch)
I20240502 21:27:01.526993 22827 consensus_queue.cc:1035] T 693a8666bbe54ae1b774f7f4da5ba67e P faadabfb099b4abf90a466395be585b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 584, Last known committed idx: 583, Time since last communication: 0.000s
I20240502 21:27:01.527400 22820 consensus_queue.cc:1035] T 693a8666bbe54ae1b774f7f4da5ba67e P faadabfb099b4abf90a466395be585b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 584, Last known committed idx: 583, Time since last communication: 0.000s
I20240502 21:27:01.530208 22820 raft_consensus.cc:2932] T 693a8666bbe54ae1b774f7f4da5ba67e P faadabfb099b4abf90a466395be585b5 [term 1 LEADER]: Committing config change with OpId 1.584: config changed from index 181 to 584, NON_VOTER bc20559563fe44efa0a2b7b084aafbce (127.15.255.136) added. New config: { opid_index: 584 OBSOLETE_local: false peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } attrs { replace: true } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: true } } }
I20240502 21:27:01.531122 20167 raft_consensus.cc:2932] T 693a8666bbe54ae1b774f7f4da5ba67e P 90e329b457c24caf93a9f880345eb288 [term 1 FOLLOWER]: Committing config change with OpId 1.584: config changed from index 181 to 584, NON_VOTER bc20559563fe44efa0a2b7b084aafbce (127.15.255.136) added. New config: { opid_index: 584 OBSOLETE_local: false peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } attrs { replace: true } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: true } } }
I20240502 21:27:01.534579 19009 raft_consensus.cc:2932] T 693a8666bbe54ae1b774f7f4da5ba67e P 9be399440ded4e2ebe6b6e0bf486cbb1 [term 1 FOLLOWER]: Committing config change with OpId 1.584: config changed from index 181 to 584, NON_VOTER bc20559563fe44efa0a2b7b084aafbce (127.15.255.136) added. New config: { opid_index: 584 OBSOLETE_local: false peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } attrs { replace: true } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: true } } }
I20240502 21:27:01.535410 19140 raft_consensus.cc:2932] T 693a8666bbe54ae1b774f7f4da5ba67e P 2877108756304e05bd60ba1186b870da [term 1 FOLLOWER]: Committing config change with OpId 1.584: config changed from index 181 to 584, NON_VOTER bc20559563fe44efa0a2b7b084aafbce (127.15.255.136) added. New config: { opid_index: 584 OBSOLETE_local: false peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } attrs { replace: true } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: true } } }
I20240502 21:27:01.536204 18324 catalog_manager.cc:5484] T 693a8666bbe54ae1b774f7f4da5ba67e P 90e329b457c24caf93a9f880345eb288 reported cstate change: config changed from index 181 to 584, NON_VOTER bc20559563fe44efa0a2b7b084aafbce (127.15.255.136) added. New cstate: current_term: 1 leader_uuid: "faadabfb099b4abf90a466395be585b5" committed_config { opid_index: 584 OBSOLETE_local: false peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } attrs { replace: true } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: true } } }
I20240502 21:27:01.541970 18739 raft_consensus.cc:2932] T 693a8666bbe54ae1b774f7f4da5ba67e P 501e3c0d44fe4d70af9ff926d9622417 [term 1 FOLLOWER]: Committing config change with OpId 1.584: config changed from index 181 to 584, NON_VOTER bc20559563fe44efa0a2b7b084aafbce (127.15.255.136) added. New config: { opid_index: 584 OBSOLETE_local: false peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } attrs { replace: true } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: true } } }
I20240502 21:27:01.570052 18472 consensus_queue.cc:237] T ed78d40318fe4ab5bee4a01b182a34b6 P faadabfb099b4abf90a466395be585b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 497, Committed index: 497, Last appended: 1.497, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 498 OBSOLETE_local: false peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } attrs { replace: true } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: true } }
I20240502 21:27:01.572715 19009 raft_consensus.cc:1261] T ed78d40318fe4ab5bee4a01b182a34b6 P 9be399440ded4e2ebe6b6e0bf486cbb1 [term 1 FOLLOWER]: Refusing update from remote peer faadabfb099b4abf90a466395be585b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 497. Preceding OpId from leader: term: 1 index: 498. (index mismatch)
I20240502 21:27:01.573292 22820 consensus_queue.cc:1035] T ed78d40318fe4ab5bee4a01b182a34b6 P faadabfb099b4abf90a466395be585b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 498, Last known committed idx: 497, Time since last communication: 0.000s
I20240502 21:27:01.573733 18739 raft_consensus.cc:1261] T ed78d40318fe4ab5bee4a01b182a34b6 P 501e3c0d44fe4d70af9ff926d9622417 [term 1 FOLLOWER]: Refusing update from remote peer faadabfb099b4abf90a466395be585b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 497. Preceding OpId from leader: term: 1 index: 498. (index mismatch)
I20240502 21:27:01.573931 18876 raft_consensus.cc:1261] T ed78d40318fe4ab5bee4a01b182a34b6 P c75831edb3b44dd59c00ef151adca49d [term 1 FOLLOWER]: Refusing update from remote peer faadabfb099b4abf90a466395be585b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 497. Preceding OpId from leader: term: 1 index: 498. (index mismatch)
W20240502 21:27:01.574347 18404 consensus_peers.cc:485] T ed78d40318fe4ab5bee4a01b182a34b6 P faadabfb099b4abf90a466395be585b5 -> Peer bc20559563fe44efa0a2b7b084aafbce (127.15.255.136:34965): Couldn't send request to peer bc20559563fe44efa0a2b7b084aafbce. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ed78d40318fe4ab5bee4a01b182a34b6. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:01.574599 22820 consensus_queue.cc:1035] T ed78d40318fe4ab5bee4a01b182a34b6 P faadabfb099b4abf90a466395be585b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 498, Last known committed idx: 497, Time since last communication: 0.000s
I20240502 21:27:01.575042 19140 raft_consensus.cc:1261] T ed78d40318fe4ab5bee4a01b182a34b6 P 2877108756304e05bd60ba1186b870da [term 1 FOLLOWER]: Refusing update from remote peer faadabfb099b4abf90a466395be585b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 497. Preceding OpId from leader: term: 1 index: 498. (index mismatch)
I20240502 21:27:01.576275 22820 consensus_queue.cc:1035] T ed78d40318fe4ab5bee4a01b182a34b6 P faadabfb099b4abf90a466395be585b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 498, Last known committed idx: 497, Time since last communication: 0.000s
I20240502 21:27:01.576568 22827 consensus_queue.cc:1035] T ed78d40318fe4ab5bee4a01b182a34b6 P faadabfb099b4abf90a466395be585b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 498, Last known committed idx: 497, Time since last communication: 0.000s
I20240502 21:27:01.589972 22820 raft_consensus.cc:2932] T ed78d40318fe4ab5bee4a01b182a34b6 P faadabfb099b4abf90a466395be585b5 [term 1 LEADER]: Committing config change with OpId 1.498: config changed from index -1 to 498, NON_VOTER bc20559563fe44efa0a2b7b084aafbce (127.15.255.136) added. New config: { opid_index: 498 OBSOLETE_local: false peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } attrs { replace: true } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: true } } }
I20240502 21:27:01.591609 19009 raft_consensus.cc:2932] T ed78d40318fe4ab5bee4a01b182a34b6 P 9be399440ded4e2ebe6b6e0bf486cbb1 [term 1 FOLLOWER]: Committing config change with OpId 1.498: config changed from index -1 to 498, NON_VOTER bc20559563fe44efa0a2b7b084aafbce (127.15.255.136) added. New config: { opid_index: 498 OBSOLETE_local: false peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } attrs { replace: true } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: true } } }
I20240502 21:27:01.593771 19140 raft_consensus.cc:2932] T ed78d40318fe4ab5bee4a01b182a34b6 P 2877108756304e05bd60ba1186b870da [term 1 FOLLOWER]: Committing config change with OpId 1.498: config changed from index -1 to 498, NON_VOTER bc20559563fe44efa0a2b7b084aafbce (127.15.255.136) added. New config: { opid_index: 498 OBSOLETE_local: false peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } attrs { replace: true } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: true } } }
I20240502 21:27:01.596402 18739 raft_consensus.cc:2932] T ed78d40318fe4ab5bee4a01b182a34b6 P 501e3c0d44fe4d70af9ff926d9622417 [term 1 FOLLOWER]: Committing config change with OpId 1.498: config changed from index -1 to 498, NON_VOTER bc20559563fe44efa0a2b7b084aafbce (127.15.255.136) added. New config: { opid_index: 498 OBSOLETE_local: false peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } attrs { replace: true } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: true } } }
I20240502 21:27:01.597746 18327 catalog_manager.cc:5484] T ed78d40318fe4ab5bee4a01b182a34b6 P faadabfb099b4abf90a466395be585b5 reported cstate change: config changed from index -1 to 498, NON_VOTER bc20559563fe44efa0a2b7b084aafbce (127.15.255.136) added. New cstate: current_term: 1 leader_uuid: "faadabfb099b4abf90a466395be585b5" committed_config { opid_index: 498 OBSOLETE_local: false peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:01.605587 18876 raft_consensus.cc:2932] T ed78d40318fe4ab5bee4a01b182a34b6 P c75831edb3b44dd59c00ef151adca49d [term 1 FOLLOWER]: Committing config change with OpId 1.498: config changed from index -1 to 498, NON_VOTER bc20559563fe44efa0a2b7b084aafbce (127.15.255.136) added. New config: { opid_index: 498 OBSOLETE_local: false peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } attrs { replace: true } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: true } } }
I20240502 21:27:01.750607 22870 ts_tablet_manager.cc:902] T 1df9d3ad5ddc438798543900b6c67b46 P bc20559563fe44efa0a2b7b084aafbce: Initiating tablet copy from peer 501e3c0d44fe4d70af9ff926d9622417 (127.15.255.131:42501)
I20240502 21:27:01.760883 22870 tablet_copy_client.cc:308] T 1df9d3ad5ddc438798543900b6c67b46 P bc20559563fe44efa0a2b7b084aafbce: tablet copy: Beginning tablet copy session from remote peer at address 127.15.255.131:42501
I20240502 21:27:01.769855 18765 tablet_copy_service.cc:140] P 501e3c0d44fe4d70af9ff926d9622417: Received BeginTabletCopySession request for tablet 1df9d3ad5ddc438798543900b6c67b46 from peer bc20559563fe44efa0a2b7b084aafbce ({username='slave'} at 127.15.255.136:36233)
I20240502 21:27:01.770087 18765 tablet_copy_service.cc:161] P 501e3c0d44fe4d70af9ff926d9622417: Beginning new tablet copy session on tablet 1df9d3ad5ddc438798543900b6c67b46 from peer bc20559563fe44efa0a2b7b084aafbce at {username='slave'} at 127.15.255.136:36233: session id = bc20559563fe44efa0a2b7b084aafbce-1df9d3ad5ddc438798543900b6c67b46
I20240502 21:27:01.773962 18765 tablet_copy_source_session.cc:205] T 1df9d3ad5ddc438798543900b6c67b46 P 501e3c0d44fe4d70af9ff926d9622417: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:27:01.780412 22870 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1df9d3ad5ddc438798543900b6c67b46. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:01.791915 22870 tablet_copy_client.cc:783] T 1df9d3ad5ddc438798543900b6c67b46 P bc20559563fe44efa0a2b7b084aafbce: tablet copy: Starting download of 0 data blocks...
I20240502 21:27:01.792279 22870 tablet_copy_client.cc:647] T 1df9d3ad5ddc438798543900b6c67b46 P bc20559563fe44efa0a2b7b084aafbce: tablet copy: Starting download of 1 WAL segments...
I20240502 21:27:01.796856 22870 tablet_copy_client.cc:523] T 1df9d3ad5ddc438798543900b6c67b46 P bc20559563fe44efa0a2b7b084aafbce: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:27:01.800652 22870 tablet_bootstrap.cc:492] T 1df9d3ad5ddc438798543900b6c67b46 P bc20559563fe44efa0a2b7b084aafbce: Bootstrap starting.
I20240502 21:27:01.825536 22876 ts_tablet_manager.cc:902] T ebf7371ff0ef4de99ced8fec0468af28 P bc20559563fe44efa0a2b7b084aafbce: Initiating tablet copy from peer c75831edb3b44dd59c00ef151adca49d (127.15.255.132:43607)
I20240502 21:27:01.828370 22876 tablet_copy_client.cc:308] T ebf7371ff0ef4de99ced8fec0468af28 P bc20559563fe44efa0a2b7b084aafbce: tablet copy: Beginning tablet copy session from remote peer at address 127.15.255.132:43607
I20240502 21:27:01.837663 18898 tablet_copy_service.cc:140] P c75831edb3b44dd59c00ef151adca49d: Received BeginTabletCopySession request for tablet ebf7371ff0ef4de99ced8fec0468af28 from peer bc20559563fe44efa0a2b7b084aafbce ({username='slave'} at 127.15.255.136:39477)
I20240502 21:27:01.837839 18898 tablet_copy_service.cc:161] P c75831edb3b44dd59c00ef151adca49d: Beginning new tablet copy session on tablet ebf7371ff0ef4de99ced8fec0468af28 from peer bc20559563fe44efa0a2b7b084aafbce at {username='slave'} at 127.15.255.136:39477: session id = bc20559563fe44efa0a2b7b084aafbce-ebf7371ff0ef4de99ced8fec0468af28
I20240502 21:27:01.840939 18898 tablet_copy_source_session.cc:205] T ebf7371ff0ef4de99ced8fec0468af28 P c75831edb3b44dd59c00ef151adca49d: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:27:01.843802 22876 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ebf7371ff0ef4de99ced8fec0468af28. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:01.852372 22876 tablet_copy_client.cc:783] T ebf7371ff0ef4de99ced8fec0468af28 P bc20559563fe44efa0a2b7b084aafbce: tablet copy: Starting download of 0 data blocks...
I20240502 21:27:01.852648 22876 tablet_copy_client.cc:647] T ebf7371ff0ef4de99ced8fec0468af28 P bc20559563fe44efa0a2b7b084aafbce: tablet copy: Starting download of 1 WAL segments...
I20240502 21:27:01.855911 22876 tablet_copy_client.cc:523] T ebf7371ff0ef4de99ced8fec0468af28 P bc20559563fe44efa0a2b7b084aafbce: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:27:01.860023 22876 tablet_bootstrap.cc:492] T ebf7371ff0ef4de99ced8fec0468af28 P bc20559563fe44efa0a2b7b084aafbce: Bootstrap starting.
I20240502 21:27:01.879105 22881 ts_tablet_manager.cc:902] T a7a9256f2b784abcb780b190e2ff3983 P bc20559563fe44efa0a2b7b084aafbce: Initiating tablet copy from peer 82f50f4b66f3450aa9f58a0b317f5cea (127.15.255.130:45139)
I20240502 21:27:01.884588 22870 log.cc:824] T 1df9d3ad5ddc438798543900b6c67b46 P bc20559563fe44efa0a2b7b084aafbce: Log is configured to *not* fsync() on all Append() calls
I20240502 21:27:01.913026 22881 tablet_copy_client.cc:308] T a7a9256f2b784abcb780b190e2ff3983 P bc20559563fe44efa0a2b7b084aafbce: tablet copy: Beginning tablet copy session from remote peer at address 127.15.255.130:45139
I20240502 21:27:01.923022 18632 tablet_copy_service.cc:140] P 82f50f4b66f3450aa9f58a0b317f5cea: Received BeginTabletCopySession request for tablet a7a9256f2b784abcb780b190e2ff3983 from peer bc20559563fe44efa0a2b7b084aafbce ({username='slave'} at 127.15.255.136:48763)
I20240502 21:27:01.923173 18632 tablet_copy_service.cc:161] P 82f50f4b66f3450aa9f58a0b317f5cea: Beginning new tablet copy session on tablet a7a9256f2b784abcb780b190e2ff3983 from peer bc20559563fe44efa0a2b7b084aafbce at {username='slave'} at 127.15.255.136:48763: session id = bc20559563fe44efa0a2b7b084aafbce-a7a9256f2b784abcb780b190e2ff3983
I20240502 21:27:01.926115 18632 tablet_copy_source_session.cc:205] T a7a9256f2b784abcb780b190e2ff3983 P 82f50f4b66f3450aa9f58a0b317f5cea: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:27:01.927632 22881 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7a9256f2b784abcb780b190e2ff3983. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:01.936758 22881 tablet_copy_client.cc:783] T a7a9256f2b784abcb780b190e2ff3983 P bc20559563fe44efa0a2b7b084aafbce: tablet copy: Starting download of 0 data blocks...
I20240502 21:27:01.937014 22881 tablet_copy_client.cc:647] T a7a9256f2b784abcb780b190e2ff3983 P bc20559563fe44efa0a2b7b084aafbce: tablet copy: Starting download of 1 WAL segments...
I20240502 21:27:01.940564 22881 tablet_copy_client.cc:523] T a7a9256f2b784abcb780b190e2ff3983 P bc20559563fe44efa0a2b7b084aafbce: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:27:01.944278 22881 tablet_bootstrap.cc:492] T a7a9256f2b784abcb780b190e2ff3983 P bc20559563fe44efa0a2b7b084aafbce: Bootstrap starting.
I20240502 21:27:02.037405 22897 ts_tablet_manager.cc:902] T 693a8666bbe54ae1b774f7f4da5ba67e P bc20559563fe44efa0a2b7b084aafbce: Initiating tablet copy from peer faadabfb099b4abf90a466395be585b5 (127.15.255.129:32983)
I20240502 21:27:02.068784 22897 tablet_copy_client.cc:308] T 693a8666bbe54ae1b774f7f4da5ba67e P bc20559563fe44efa0a2b7b084aafbce: tablet copy: Beginning tablet copy session from remote peer at address 127.15.255.129:32983
I20240502 21:27:02.076973 18496 tablet_copy_service.cc:140] P faadabfb099b4abf90a466395be585b5: Received BeginTabletCopySession request for tablet 693a8666bbe54ae1b774f7f4da5ba67e from peer bc20559563fe44efa0a2b7b084aafbce ({username='slave'} at 127.15.255.136:50439)
I20240502 21:27:02.077152 18496 tablet_copy_service.cc:161] P faadabfb099b4abf90a466395be585b5: Beginning new tablet copy session on tablet 693a8666bbe54ae1b774f7f4da5ba67e from peer bc20559563fe44efa0a2b7b084aafbce at {username='slave'} at 127.15.255.136:50439: session id = bc20559563fe44efa0a2b7b084aafbce-693a8666bbe54ae1b774f7f4da5ba67e
I20240502 21:27:02.080026 18496 tablet_copy_source_session.cc:205] T 693a8666bbe54ae1b774f7f4da5ba67e P faadabfb099b4abf90a466395be585b5: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:27:02.082650 22897 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 693a8666bbe54ae1b774f7f4da5ba67e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:02.092912 22897 tablet_copy_client.cc:783] T 693a8666bbe54ae1b774f7f4da5ba67e P bc20559563fe44efa0a2b7b084aafbce: tablet copy: Starting download of 0 data blocks...
I20240502 21:27:02.093173 22897 tablet_copy_client.cc:647] T 693a8666bbe54ae1b774f7f4da5ba67e P bc20559563fe44efa0a2b7b084aafbce: tablet copy: Starting download of 1 WAL segments...
I20240502 21:27:02.098554 22897 tablet_copy_client.cc:523] T 693a8666bbe54ae1b774f7f4da5ba67e P bc20559563fe44efa0a2b7b084aafbce: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:27:02.102234 22897 tablet_bootstrap.cc:492] T 693a8666bbe54ae1b774f7f4da5ba67e P bc20559563fe44efa0a2b7b084aafbce: Bootstrap starting.
I20240502 21:27:02.145794 22908 ts_tablet_manager.cc:902] T ed78d40318fe4ab5bee4a01b182a34b6 P bc20559563fe44efa0a2b7b084aafbce: Initiating tablet copy from peer faadabfb099b4abf90a466395be585b5 (127.15.255.129:32983)
I20240502 21:27:02.172617 22908 tablet_copy_client.cc:308] T ed78d40318fe4ab5bee4a01b182a34b6 P bc20559563fe44efa0a2b7b084aafbce: tablet copy: Beginning tablet copy session from remote peer at address 127.15.255.129:32983
I20240502 21:27:02.173411 18496 tablet_copy_service.cc:140] P faadabfb099b4abf90a466395be585b5: Received BeginTabletCopySession request for tablet ed78d40318fe4ab5bee4a01b182a34b6 from peer bc20559563fe44efa0a2b7b084aafbce ({username='slave'} at 127.15.255.136:50439)
I20240502 21:27:02.173580 18496 tablet_copy_service.cc:161] P faadabfb099b4abf90a466395be585b5: Beginning new tablet copy session on tablet ed78d40318fe4ab5bee4a01b182a34b6 from peer bc20559563fe44efa0a2b7b084aafbce at {username='slave'} at 127.15.255.136:50439: session id = bc20559563fe44efa0a2b7b084aafbce-ed78d40318fe4ab5bee4a01b182a34b6
I20240502 21:27:02.176522 18496 tablet_copy_source_session.cc:205] T ed78d40318fe4ab5bee4a01b182a34b6 P faadabfb099b4abf90a466395be585b5: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:27:02.177855 22908 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed78d40318fe4ab5bee4a01b182a34b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:02.184139 22908 tablet_copy_client.cc:783] T ed78d40318fe4ab5bee4a01b182a34b6 P bc20559563fe44efa0a2b7b084aafbce: tablet copy: Starting download of 0 data blocks...
I20240502 21:27:02.184382 22908 tablet_copy_client.cc:647] T ed78d40318fe4ab5bee4a01b182a34b6 P bc20559563fe44efa0a2b7b084aafbce: tablet copy: Starting download of 1 WAL segments...
I20240502 21:27:02.188473 22908 tablet_copy_client.cc:523] T ed78d40318fe4ab5bee4a01b182a34b6 P bc20559563fe44efa0a2b7b084aafbce: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:27:02.191793 22908 tablet_bootstrap.cc:492] T ed78d40318fe4ab5bee4a01b182a34b6 P bc20559563fe44efa0a2b7b084aafbce: Bootstrap starting.
I20240502 21:27:02.681910 22876 tablet_bootstrap.cc:492] T ebf7371ff0ef4de99ced8fec0468af28 P bc20559563fe44efa0a2b7b084aafbce: Bootstrap replayed 1/1 log segments. Stats: ops{read=387 overwritten=0 applied=387 ignored=0} inserts{seen=1069 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:27:02.682556 22876 tablet_bootstrap.cc:492] T ebf7371ff0ef4de99ced8fec0468af28 P bc20559563fe44efa0a2b7b084aafbce: Bootstrap complete.
I20240502 21:27:02.685335 22876 ts_tablet_manager.cc:1367] T ebf7371ff0ef4de99ced8fec0468af28 P bc20559563fe44efa0a2b7b084aafbce: Time spent bootstrapping tablet: real 0.825s user 0.461s sys 0.047s
I20240502 21:27:02.694828 22876 raft_consensus.cc:348] T ebf7371ff0ef4de99ced8fec0468af28 P bc20559563fe44efa0a2b7b084aafbce [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 387 OBSOLETE_local: false peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } attrs { replace: true } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: true } }
I20240502 21:27:02.704157 22876 raft_consensus.cc:729] T ebf7371ff0ef4de99ced8fec0468af28 P bc20559563fe44efa0a2b7b084aafbce [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: bc20559563fe44efa0a2b7b084aafbce, State: Initialized, Role: LEARNER
I20240502 21:27:02.704860 22876 consensus_queue.cc:260] T ebf7371ff0ef4de99ced8fec0468af28 P bc20559563fe44efa0a2b7b084aafbce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 387, Last appended: 1.387, Last appended by leader: 387, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 387 OBSOLETE_local: false peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } attrs { replace: true } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: true } }
I20240502 21:27:02.724494 22876 ts_tablet_manager.cc:1398] T ebf7371ff0ef4de99ced8fec0468af28 P bc20559563fe44efa0a2b7b084aafbce: Time spent starting tablet: real 0.037s user 0.003s sys 0.012s
I20240502 21:27:02.725517 18898 tablet_copy_service.cc:342] P c75831edb3b44dd59c00ef151adca49d: Request end of tablet copy session bc20559563fe44efa0a2b7b084aafbce-ebf7371ff0ef4de99ced8fec0468af28 received from {username='slave'} at 127.15.255.136:39477
I20240502 21:27:02.725641 18898 tablet_copy_service.cc:433] P c75831edb3b44dd59c00ef151adca49d: ending tablet copy session bc20559563fe44efa0a2b7b084aafbce-ebf7371ff0ef4de99ced8fec0468af28 on tablet ebf7371ff0ef4de99ced8fec0468af28 with peer bc20559563fe44efa0a2b7b084aafbce
I20240502 21:27:02.735376 22870 tablet_bootstrap.cc:492] T 1df9d3ad5ddc438798543900b6c67b46 P bc20559563fe44efa0a2b7b084aafbce: Bootstrap replayed 1/1 log segments. Stats: ops{read=502 overwritten=0 applied=502 ignored=0} inserts{seen=1187 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:27:02.735918 22870 tablet_bootstrap.cc:492] T 1df9d3ad5ddc438798543900b6c67b46 P bc20559563fe44efa0a2b7b084aafbce: Bootstrap complete.
I20240502 21:27:02.736270 22870 ts_tablet_manager.cc:1367] T 1df9d3ad5ddc438798543900b6c67b46 P bc20559563fe44efa0a2b7b084aafbce: Time spent bootstrapping tablet: real 0.936s user 0.602s sys 0.043s
I20240502 21:27:02.737807 22870 raft_consensus.cc:348] T 1df9d3ad5ddc438798543900b6c67b46 P bc20559563fe44efa0a2b7b084aafbce [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 502 OBSOLETE_local: false peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } attrs { replace: true } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: true } }
I20240502 21:27:02.738129 22870 raft_consensus.cc:729] T 1df9d3ad5ddc438798543900b6c67b46 P bc20559563fe44efa0a2b7b084aafbce [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: bc20559563fe44efa0a2b7b084aafbce, State: Initialized, Role: LEARNER
I20240502 21:27:02.738412 22870 consensus_queue.cc:260] T 1df9d3ad5ddc438798543900b6c67b46 P bc20559563fe44efa0a2b7b084aafbce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 502, Last appended: 1.502, Last appended by leader: 502, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 502 OBSOLETE_local: false peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } attrs { replace: true } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: true } }
I20240502 21:27:02.739110 22870 ts_tablet_manager.cc:1398] T 1df9d3ad5ddc438798543900b6c67b46 P bc20559563fe44efa0a2b7b084aafbce: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20240502 21:27:02.739851 18765 tablet_copy_service.cc:342] P 501e3c0d44fe4d70af9ff926d9622417: Request end of tablet copy session bc20559563fe44efa0a2b7b084aafbce-1df9d3ad5ddc438798543900b6c67b46 received from {username='slave'} at 127.15.255.136:36233
I20240502 21:27:02.740011 18765 tablet_copy_service.cc:433] P 501e3c0d44fe4d70af9ff926d9622417: ending tablet copy session bc20559563fe44efa0a2b7b084aafbce-1df9d3ad5ddc438798543900b6c67b46 on tablet 1df9d3ad5ddc438798543900b6c67b46 with peer bc20559563fe44efa0a2b7b084aafbce
I20240502 21:27:02.929946 22633 raft_consensus.cc:1203] T 1df9d3ad5ddc438798543900b6c67b46 P bc20559563fe44efa0a2b7b084aafbce [term 1 LEARNER]: Deduplicated request from leader. Original: 1.501->[1.502-1.502] Dedup: 1.502->[]
I20240502 21:27:02.934638 22881 tablet_bootstrap.cc:492] T a7a9256f2b784abcb780b190e2ff3983 P bc20559563fe44efa0a2b7b084aafbce: Bootstrap replayed 1/1 log segments. Stats: ops{read=511 overwritten=0 applied=511 ignored=0} inserts{seen=1180 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:27:02.935103 22881 tablet_bootstrap.cc:492] T a7a9256f2b784abcb780b190e2ff3983 P bc20559563fe44efa0a2b7b084aafbce: Bootstrap complete.
I20240502 21:27:02.935460 22881 ts_tablet_manager.cc:1367] T a7a9256f2b784abcb780b190e2ff3983 P bc20559563fe44efa0a2b7b084aafbce: Time spent bootstrapping tablet: real 0.991s user 0.581s sys 0.048s
I20240502 21:27:02.936334 22881 raft_consensus.cc:348] T a7a9256f2b784abcb780b190e2ff3983 P bc20559563fe44efa0a2b7b084aafbce [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 511 OBSOLETE_local: false peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } attrs { replace: true } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: true } }
I20240502 21:27:02.936666 22881 raft_consensus.cc:729] T a7a9256f2b784abcb780b190e2ff3983 P bc20559563fe44efa0a2b7b084aafbce [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: bc20559563fe44efa0a2b7b084aafbce, State: Initialized, Role: LEARNER
I20240502 21:27:02.936893 22881 consensus_queue.cc:260] T a7a9256f2b784abcb780b190e2ff3983 P bc20559563fe44efa0a2b7b084aafbce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 511, Last appended: 1.511, Last appended by leader: 511, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 511 OBSOLETE_local: false peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } attrs { replace: true } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: true } }
I20240502 21:27:02.937649 22881 ts_tablet_manager.cc:1398] T a7a9256f2b784abcb780b190e2ff3983 P bc20559563fe44efa0a2b7b084aafbce: Time spent starting tablet: real 0.002s user 0.005s sys 0.000s
I20240502 21:27:02.938350 18632 tablet_copy_service.cc:342] P 82f50f4b66f3450aa9f58a0b317f5cea: Request end of tablet copy session bc20559563fe44efa0a2b7b084aafbce-a7a9256f2b784abcb780b190e2ff3983 received from {username='slave'} at 127.15.255.136:48763
I20240502 21:27:02.938483 18632 tablet_copy_service.cc:433] P 82f50f4b66f3450aa9f58a0b317f5cea: ending tablet copy session bc20559563fe44efa0a2b7b084aafbce-a7a9256f2b784abcb780b190e2ff3983 on tablet a7a9256f2b784abcb780b190e2ff3983 with peer bc20559563fe44efa0a2b7b084aafbce
I20240502 21:27:02.956120 22633 raft_consensus.cc:1203] T a7a9256f2b784abcb780b190e2ff3983 P bc20559563fe44efa0a2b7b084aafbce [term 1 LEARNER]: Deduplicated request from leader. Original: 1.510->[1.511-1.511] Dedup: 1.511->[]
I20240502 21:27:03.096334 22908 tablet_bootstrap.cc:492] T ed78d40318fe4ab5bee4a01b182a34b6 P bc20559563fe44efa0a2b7b084aafbce: Bootstrap replayed 1/1 log segments. Stats: ops{read=498 overwritten=0 applied=498 ignored=0} inserts{seen=1170 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:27:03.096837 22908 tablet_bootstrap.cc:492] T ed78d40318fe4ab5bee4a01b182a34b6 P bc20559563fe44efa0a2b7b084aafbce: Bootstrap complete.
I20240502 21:27:03.097190 22908 ts_tablet_manager.cc:1367] T ed78d40318fe4ab5bee4a01b182a34b6 P bc20559563fe44efa0a2b7b084aafbce: Time spent bootstrapping tablet: real 0.905s user 0.551s sys 0.080s
I20240502 21:27:03.097923 22897 tablet_bootstrap.cc:492] T 693a8666bbe54ae1b774f7f4da5ba67e P bc20559563fe44efa0a2b7b084aafbce: Bootstrap replayed 1/1 log segments. Stats: ops{read=584 overwritten=0 applied=584 ignored=0} inserts{seen=1212 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:27:03.097998 22908 raft_consensus.cc:348] T ed78d40318fe4ab5bee4a01b182a34b6 P bc20559563fe44efa0a2b7b084aafbce [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 498 OBSOLETE_local: false peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } attrs { replace: true } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: true } }
I20240502 21:27:03.098287 22908 raft_consensus.cc:729] T ed78d40318fe4ab5bee4a01b182a34b6 P bc20559563fe44efa0a2b7b084aafbce [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: bc20559563fe44efa0a2b7b084aafbce, State: Initialized, Role: LEARNER
I20240502 21:27:03.098335 22897 tablet_bootstrap.cc:492] T 693a8666bbe54ae1b774f7f4da5ba67e P bc20559563fe44efa0a2b7b084aafbce: Bootstrap complete.
I20240502 21:27:03.098657 22897 ts_tablet_manager.cc:1367] T 693a8666bbe54ae1b774f7f4da5ba67e P bc20559563fe44efa0a2b7b084aafbce: Time spent bootstrapping tablet: real 0.997s user 0.645s sys 0.064s
I20240502 21:27:03.098570 22908 consensus_queue.cc:260] T ed78d40318fe4ab5bee4a01b182a34b6 P bc20559563fe44efa0a2b7b084aafbce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 498, Last appended: 1.498, Last appended by leader: 498, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 498 OBSOLETE_local: false peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } attrs { replace: true } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: true } }
I20240502 21:27:03.099192 22908 ts_tablet_manager.cc:1398] T ed78d40318fe4ab5bee4a01b182a34b6 P bc20559563fe44efa0a2b7b084aafbce: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:27:03.099342 22897 raft_consensus.cc:348] T 693a8666bbe54ae1b774f7f4da5ba67e P bc20559563fe44efa0a2b7b084aafbce [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 584 OBSOLETE_local: false peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } attrs { replace: true } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: true } }
I20240502 21:27:03.099704 22897 raft_consensus.cc:729] T 693a8666bbe54ae1b774f7f4da5ba67e P bc20559563fe44efa0a2b7b084aafbce [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: bc20559563fe44efa0a2b7b084aafbce, State: Initialized, Role: LEARNER
I20240502 21:27:03.099931 22897 consensus_queue.cc:260] T 693a8666bbe54ae1b774f7f4da5ba67e P bc20559563fe44efa0a2b7b084aafbce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 584, Last appended: 1.584, Last appended by leader: 584, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 584 OBSOLETE_local: false peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } attrs { replace: true } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: true } }
I20240502 21:27:03.100292 18496 tablet_copy_service.cc:342] P faadabfb099b4abf90a466395be585b5: Request end of tablet copy session bc20559563fe44efa0a2b7b084aafbce-ed78d40318fe4ab5bee4a01b182a34b6 received from {username='slave'} at 127.15.255.136:50439
I20240502 21:27:03.100412 18496 tablet_copy_service.cc:433] P faadabfb099b4abf90a466395be585b5: ending tablet copy session bc20559563fe44efa0a2b7b084aafbce-ed78d40318fe4ab5bee4a01b182a34b6 on tablet ed78d40318fe4ab5bee4a01b182a34b6 with peer bc20559563fe44efa0a2b7b084aafbce
I20240502 21:27:03.101004 22897 ts_tablet_manager.cc:1398] T 693a8666bbe54ae1b774f7f4da5ba67e P bc20559563fe44efa0a2b7b084aafbce: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240502 21:27:03.101838 18496 tablet_copy_service.cc:342] P faadabfb099b4abf90a466395be585b5: Request end of tablet copy session bc20559563fe44efa0a2b7b084aafbce-693a8666bbe54ae1b774f7f4da5ba67e received from {username='slave'} at 127.15.255.136:50439
I20240502 21:27:03.101956 18496 tablet_copy_service.cc:433] P faadabfb099b4abf90a466395be585b5: ending tablet copy session bc20559563fe44efa0a2b7b084aafbce-693a8666bbe54ae1b774f7f4da5ba67e on tablet 693a8666bbe54ae1b774f7f4da5ba67e with peer bc20559563fe44efa0a2b7b084aafbce
I20240502 21:27:03.108006 22633 raft_consensus.cc:1203] T ebf7371ff0ef4de99ced8fec0468af28 P bc20559563fe44efa0a2b7b084aafbce [term 1 LEARNER]: Deduplicated request from leader. Original: 1.386->[1.387-1.387] Dedup: 1.387->[]
I20240502 21:27:03.297194 22633 raft_consensus.cc:1203] T ed78d40318fe4ab5bee4a01b182a34b6 P bc20559563fe44efa0a2b7b084aafbce [term 1 LEARNER]: Deduplicated request from leader. Original: 1.497->[1.498-1.498] Dedup: 1.498->[]
I20240502 21:27:03.392200 22963 raft_consensus.cc:1050] T 1df9d3ad5ddc438798543900b6c67b46 P 501e3c0d44fe4d70af9ff926d9622417: attempting to promote NON_VOTER bc20559563fe44efa0a2b7b084aafbce to VOTER
I20240502 21:27:03.393440 22963 consensus_queue.cc:237] T 1df9d3ad5ddc438798543900b6c67b46 P 501e3c0d44fe4d70af9ff926d9622417 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 502, Committed index: 502, Last appended: 1.502, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 503 OBSOLETE_local: false peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } attrs { replace: true } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: false } }
I20240502 21:27:03.395670 18876 raft_consensus.cc:1261] T 1df9d3ad5ddc438798543900b6c67b46 P c75831edb3b44dd59c00ef151adca49d [term 1 FOLLOWER]: Refusing update from remote peer 501e3c0d44fe4d70af9ff926d9622417: Log matching property violated. Preceding OpId in replica: term: 1 index: 502. Preceding OpId from leader: term: 1 index: 503. (index mismatch)
I20240502 21:27:03.396142 18472 raft_consensus.cc:1261] T 1df9d3ad5ddc438798543900b6c67b46 P faadabfb099b4abf90a466395be585b5 [term 1 FOLLOWER]: Refusing update from remote peer 501e3c0d44fe4d70af9ff926d9622417: Log matching property violated. Preceding OpId in replica: term: 1 index: 502. Preceding OpId from leader: term: 1 index: 503. (index mismatch)
I20240502 21:27:03.396224 18611 raft_consensus.cc:1261] T 1df9d3ad5ddc438798543900b6c67b46 P 82f50f4b66f3450aa9f58a0b317f5cea [term 1 FOLLOWER]: Refusing update from remote peer 501e3c0d44fe4d70af9ff926d9622417: Log matching property violated. Preceding OpId in replica: term: 1 index: 502. Preceding OpId from leader: term: 1 index: 503. (index mismatch)
I20240502 21:27:03.396657 19140 raft_consensus.cc:1261] T 1df9d3ad5ddc438798543900b6c67b46 P 2877108756304e05bd60ba1186b870da [term 1 FOLLOWER]: Refusing update from remote peer 501e3c0d44fe4d70af9ff926d9622417: Log matching property violated. Preceding OpId in replica: term: 1 index: 502. Preceding OpId from leader: term: 1 index: 503. (index mismatch)
I20240502 21:27:03.396857 22962 consensus_queue.cc:1035] T 1df9d3ad5ddc438798543900b6c67b46 P 501e3c0d44fe4d70af9ff926d9622417 [LEADER]: Connected to new peer: Peer: permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 503, Last known committed idx: 502, Time since last communication: 0.000s
I20240502 21:27:03.397236 22633 raft_consensus.cc:1261] T 1df9d3ad5ddc438798543900b6c67b46 P bc20559563fe44efa0a2b7b084aafbce [term 1 LEARNER]: Refusing update from remote peer 501e3c0d44fe4d70af9ff926d9622417: Log matching property violated. Preceding OpId in replica: term: 1 index: 502. Preceding OpId from leader: term: 1 index: 503. (index mismatch)
I20240502 21:27:03.397348 22962 consensus_queue.cc:1035] T 1df9d3ad5ddc438798543900b6c67b46 P 501e3c0d44fe4d70af9ff926d9622417 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 503, Last known committed idx: 502, Time since last communication: 0.000s
I20240502 21:27:03.397773 23019 consensus_queue.cc:1035] T 1df9d3ad5ddc438798543900b6c67b46 P 501e3c0d44fe4d70af9ff926d9622417 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 503, Last known committed idx: 502, Time since last communication: 0.000s
I20240502 21:27:03.398051 22924 consensus_queue.cc:1035] T 1df9d3ad5ddc438798543900b6c67b46 P 501e3c0d44fe4d70af9ff926d9622417 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 503, Last known committed idx: 502, Time since last communication: 0.000s
I20240502 21:27:03.398496 22963 consensus_queue.cc:1035] T 1df9d3ad5ddc438798543900b6c67b46 P 501e3c0d44fe4d70af9ff926d9622417 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 503, Last known committed idx: 502, Time since last communication: 0.000s
I20240502 21:27:03.405740 22963 raft_consensus.cc:2932] T 1df9d3ad5ddc438798543900b6c67b46 P 501e3c0d44fe4d70af9ff926d9622417 [term 1 LEADER]: Committing config change with OpId 1.503: config changed from index 502 to 503, bc20559563fe44efa0a2b7b084aafbce (127.15.255.136) changed from NON_VOTER to VOTER. New config: { opid_index: 503 OBSOLETE_local: false peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } attrs { replace: true } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: false } } }
I20240502 21:27:03.406404 18472 raft_consensus.cc:2932] T 1df9d3ad5ddc438798543900b6c67b46 P faadabfb099b4abf90a466395be585b5 [term 1 FOLLOWER]: Committing config change with OpId 1.503: config changed from index 502 to 503, bc20559563fe44efa0a2b7b084aafbce (127.15.255.136) changed from NON_VOTER to VOTER. New config: { opid_index: 503 OBSOLETE_local: false peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } attrs { replace: true } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: false } } }
I20240502 21:27:03.409571 18876 raft_consensus.cc:2932] T 1df9d3ad5ddc438798543900b6c67b46 P c75831edb3b44dd59c00ef151adca49d [term 1 FOLLOWER]: Committing config change with OpId 1.503: config changed from index 502 to 503, bc20559563fe44efa0a2b7b084aafbce (127.15.255.136) changed from NON_VOTER to VOTER. New config: { opid_index: 503 OBSOLETE_local: false peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } attrs { replace: true } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: false } } }
I20240502 21:27:03.409466 22633 raft_consensus.cc:2932] T 1df9d3ad5ddc438798543900b6c67b46 P bc20559563fe44efa0a2b7b084aafbce [term 1 FOLLOWER]: Committing config change with OpId 1.503: config changed from index 502 to 503, bc20559563fe44efa0a2b7b084aafbce (127.15.255.136) changed from NON_VOTER to VOTER. New config: { opid_index: 503 OBSOLETE_local: false peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } attrs { replace: true } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: false } } }
I20240502 21:27:03.410670 18611 raft_consensus.cc:2932] T 1df9d3ad5ddc438798543900b6c67b46 P 82f50f4b66f3450aa9f58a0b317f5cea [term 1 FOLLOWER]: Committing config change with OpId 1.503: config changed from index 502 to 503, bc20559563fe44efa0a2b7b084aafbce (127.15.255.136) changed from NON_VOTER to VOTER. New config: { opid_index: 503 OBSOLETE_local: false peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } attrs { replace: true } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: false } } }
I20240502 21:27:03.413465 18322 catalog_manager.cc:5484] T 1df9d3ad5ddc438798543900b6c67b46 P 501e3c0d44fe4d70af9ff926d9622417 reported cstate change: config changed from index 502 to 503, bc20559563fe44efa0a2b7b084aafbce (127.15.255.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "501e3c0d44fe4d70af9ff926d9622417" committed_config { opid_index: 503 OBSOLETE_local: false peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:27:03.419178 22925 raft_consensus.cc:1050] T a7a9256f2b784abcb780b190e2ff3983 P 82f50f4b66f3450aa9f58a0b317f5cea: attempting to promote NON_VOTER bc20559563fe44efa0a2b7b084aafbce to VOTER
I20240502 21:27:03.419495 19140 raft_consensus.cc:2932] T 1df9d3ad5ddc438798543900b6c67b46 P 2877108756304e05bd60ba1186b870da [term 1 FOLLOWER]: Committing config change with OpId 1.503: config changed from index 502 to 503, bc20559563fe44efa0a2b7b084aafbce (127.15.255.136) changed from NON_VOTER to VOTER. New config: { opid_index: 503 OBSOLETE_local: false peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } attrs { replace: true } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: false } } }
I20240502 21:27:03.420161 22925 consensus_queue.cc:237] T a7a9256f2b784abcb780b190e2ff3983 P 82f50f4b66f3450aa9f58a0b317f5cea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 511, Committed index: 511, Last appended: 1.511, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 512 OBSOLETE_local: false peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } attrs { replace: true } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: false } }
I20240502 21:27:03.423008 18876 raft_consensus.cc:1261] T a7a9256f2b784abcb780b190e2ff3983 P c75831edb3b44dd59c00ef151adca49d [term 1 FOLLOWER]: Refusing update from remote peer 82f50f4b66f3450aa9f58a0b317f5cea: Log matching property violated. Preceding OpId in replica: term: 1 index: 511. Preceding OpId from leader: term: 1 index: 512. (index mismatch)
I20240502 21:27:03.423141 20167 raft_consensus.cc:1261] T a7a9256f2b784abcb780b190e2ff3983 P 90e329b457c24caf93a9f880345eb288 [term 1 FOLLOWER]: Refusing update from remote peer 82f50f4b66f3450aa9f58a0b317f5cea: Log matching property violated. Preceding OpId in replica: term: 1 index: 511. Preceding OpId from leader: term: 1 index: 512. (index mismatch)
I20240502 21:27:03.423470 22961 consensus_queue.cc:1035] T a7a9256f2b784abcb780b190e2ff3983 P 82f50f4b66f3450aa9f58a0b317f5cea [LEADER]: Connected to new peer: Peer: permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 512, Last known committed idx: 511, Time since last communication: 0.000s
I20240502 21:27:03.424021 22961 consensus_queue.cc:1035] T a7a9256f2b784abcb780b190e2ff3983 P 82f50f4b66f3450aa9f58a0b317f5cea [LEADER]: Connected to new peer: Peer: permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 512, Last known committed idx: 511, Time since last communication: 0.000s
I20240502 21:27:03.424243 18472 raft_consensus.cc:1261] T a7a9256f2b784abcb780b190e2ff3983 P faadabfb099b4abf90a466395be585b5 [term 1 FOLLOWER]: Refusing update from remote peer 82f50f4b66f3450aa9f58a0b317f5cea: Log matching property violated. Preceding OpId in replica: term: 1 index: 511. Preceding OpId from leader: term: 1 index: 512. (index mismatch)
I20240502 21:27:03.424840 22961 consensus_queue.cc:1035] T a7a9256f2b784abcb780b190e2ff3983 P 82f50f4b66f3450aa9f58a0b317f5cea [LEADER]: Connected to new peer: Peer: permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 512, Last known committed idx: 511, Time since last communication: 0.000s
I20240502 21:27:03.436710 18739 raft_consensus.cc:1261] T a7a9256f2b784abcb780b190e2ff3983 P 501e3c0d44fe4d70af9ff926d9622417 [term 1 FOLLOWER]: Refusing update from remote peer 82f50f4b66f3450aa9f58a0b317f5cea: Log matching property violated. Preceding OpId in replica: term: 1 index: 511. Preceding OpId from leader: term: 1 index: 512. (index mismatch)
I20240502 21:27:03.436844 22633 raft_consensus.cc:1261] T a7a9256f2b784abcb780b190e2ff3983 P bc20559563fe44efa0a2b7b084aafbce [term 1 LEARNER]: Refusing update from remote peer 82f50f4b66f3450aa9f58a0b317f5cea: Log matching property violated. Preceding OpId in replica: term: 1 index: 511. Preceding OpId from leader: term: 1 index: 512. (index mismatch)
I20240502 21:27:03.438014 22961 consensus_queue.cc:1035] T a7a9256f2b784abcb780b190e2ff3983 P 82f50f4b66f3450aa9f58a0b317f5cea [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 512, Last known committed idx: 511, Time since last communication: 0.000s
I20240502 21:27:03.438277 23028 consensus_queue.cc:1035] T a7a9256f2b784abcb780b190e2ff3983 P 82f50f4b66f3450aa9f58a0b317f5cea [LEADER]: Connected to new peer: Peer: permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 512, Last known committed idx: 511, Time since last communication: 0.000s
I20240502 21:27:03.440920 22961 raft_consensus.cc:2932] T a7a9256f2b784abcb780b190e2ff3983 P 82f50f4b66f3450aa9f58a0b317f5cea [term 1 LEADER]: Committing config change with OpId 1.512: config changed from index 511 to 512, bc20559563fe44efa0a2b7b084aafbce (127.15.255.136) changed from NON_VOTER to VOTER. New config: { opid_index: 512 OBSOLETE_local: false peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } attrs { replace: true } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: false } } }
I20240502 21:27:03.441519 18739 raft_consensus.cc:2932] T a7a9256f2b784abcb780b190e2ff3983 P 501e3c0d44fe4d70af9ff926d9622417 [term 1 FOLLOWER]: Committing config change with OpId 1.512: config changed from index 511 to 512, bc20559563fe44efa0a2b7b084aafbce (127.15.255.136) changed from NON_VOTER to VOTER. New config: { opid_index: 512 OBSOLETE_local: false peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } attrs { replace: true } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: false } } }
I20240502 21:27:03.441812 22633 raft_consensus.cc:2932] T a7a9256f2b784abcb780b190e2ff3983 P bc20559563fe44efa0a2b7b084aafbce [term 1 FOLLOWER]: Committing config change with OpId 1.512: config changed from index 511 to 512, bc20559563fe44efa0a2b7b084aafbce (127.15.255.136) changed from NON_VOTER to VOTER. New config: { opid_index: 512 OBSOLETE_local: false peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } attrs { replace: true } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: false } } }
I20240502 21:27:03.444455 18876 raft_consensus.cc:2932] T a7a9256f2b784abcb780b190e2ff3983 P c75831edb3b44dd59c00ef151adca49d [term 1 FOLLOWER]: Committing config change with OpId 1.512: config changed from index 511 to 512, bc20559563fe44efa0a2b7b084aafbce (127.15.255.136) changed from NON_VOTER to VOTER. New config: { opid_index: 512 OBSOLETE_local: false peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } attrs { replace: true } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: false } } }
I20240502 21:27:03.445756 20167 raft_consensus.cc:2932] T a7a9256f2b784abcb780b190e2ff3983 P 90e329b457c24caf93a9f880345eb288 [term 1 FOLLOWER]: Committing config change with OpId 1.512: config changed from index 511 to 512, bc20559563fe44efa0a2b7b084aafbce (127.15.255.136) changed from NON_VOTER to VOTER. New config: { opid_index: 512 OBSOLETE_local: false peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } attrs { replace: true } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: false } } }
I20240502 21:27:03.449681 18473 raft_consensus.cc:2932] T a7a9256f2b784abcb780b190e2ff3983 P faadabfb099b4abf90a466395be585b5 [term 1 FOLLOWER]: Committing config change with OpId 1.512: config changed from index 511 to 512, bc20559563fe44efa0a2b7b084aafbce (127.15.255.136) changed from NON_VOTER to VOTER. New config: { opid_index: 512 OBSOLETE_local: false peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } attrs { replace: true } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: false } } }
I20240502 21:27:03.451584 18320 catalog_manager.cc:5484] T a7a9256f2b784abcb780b190e2ff3983 P 90e329b457c24caf93a9f880345eb288 reported cstate change: config changed from index 511 to 512, bc20559563fe44efa0a2b7b084aafbce (127.15.255.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" committed_config { opid_index: 512 OBSOLETE_local: false peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } attrs { replace: true } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: false } } }
I20240502 21:27:03.455247 18739 consensus_queue.cc:237] T 1df9d3ad5ddc438798543900b6c67b46 P 501e3c0d44fe4d70af9ff926d9622417 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 503, Committed index: 503, Last appended: 1.503, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 504 OBSOLETE_local: false peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: false } }
I20240502 21:27:03.457885 18873 raft_consensus.cc:1261] T 1df9d3ad5ddc438798543900b6c67b46 P c75831edb3b44dd59c00ef151adca49d [term 1 FOLLOWER]: Refusing update from remote peer 501e3c0d44fe4d70af9ff926d9622417: Log matching property violated. Preceding OpId in replica: term: 1 index: 503. Preceding OpId from leader: term: 1 index: 504. (index mismatch)
I20240502 21:27:03.458434 22962 consensus_queue.cc:1035] T 1df9d3ad5ddc438798543900b6c67b46 P 501e3c0d44fe4d70af9ff926d9622417 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 504, Last known committed idx: 503, Time since last communication: 0.000s
I20240502 21:27:03.458863 19140 raft_consensus.cc:1261] T 1df9d3ad5ddc438798543900b6c67b46 P 2877108756304e05bd60ba1186b870da [term 1 FOLLOWER]: Refusing update from remote peer 501e3c0d44fe4d70af9ff926d9622417: Log matching property violated. Preceding OpId in replica: term: 1 index: 503. Preceding OpId from leader: term: 1 index: 504. (index mismatch)
I20240502 21:27:03.459477 22633 raft_consensus.cc:1261] T 1df9d3ad5ddc438798543900b6c67b46 P bc20559563fe44efa0a2b7b084aafbce [term 1 FOLLOWER]: Refusing update from remote peer 501e3c0d44fe4d70af9ff926d9622417: Log matching property violated. Preceding OpId in replica: term: 1 index: 503. Preceding OpId from leader: term: 1 index: 504. (index mismatch)
I20240502 21:27:03.459755 18473 raft_consensus.cc:1261] T 1df9d3ad5ddc438798543900b6c67b46 P faadabfb099b4abf90a466395be585b5 [term 1 FOLLOWER]: Refusing update from remote peer 501e3c0d44fe4d70af9ff926d9622417: Log matching property violated. Preceding OpId in replica: term: 1 index: 503. Preceding OpId from leader: term: 1 index: 504. (index mismatch)
I20240502 21:27:03.460505 23019 consensus_queue.cc:1035] T 1df9d3ad5ddc438798543900b6c67b46 P 501e3c0d44fe4d70af9ff926d9622417 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 504, Last known committed idx: 503, Time since last communication: 0.000s
I20240502 21:27:03.461025 22963 consensus_queue.cc:1035] T 1df9d3ad5ddc438798543900b6c67b46 P 501e3c0d44fe4d70af9ff926d9622417 [LEADER]: Connected to new peer: Peer: permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 504, Last known committed idx: 503, Time since last communication: 0.000s
I20240502 21:27:03.461402 23019 consensus_queue.cc:1035] T 1df9d3ad5ddc438798543900b6c67b46 P 501e3c0d44fe4d70af9ff926d9622417 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 504, Last known committed idx: 503, Time since last communication: 0.000s
I20240502 21:27:03.463724 18611 consensus_queue.cc:237] T a7a9256f2b784abcb780b190e2ff3983 P 82f50f4b66f3450aa9f58a0b317f5cea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 512, Committed index: 512, Last appended: 1.512, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 513 OBSOLETE_local: false peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: false } }
I20240502 21:27:03.464906 22632 raft_consensus.cc:1203] T 693a8666bbe54ae1b774f7f4da5ba67e P bc20559563fe44efa0a2b7b084aafbce [term 1 LEARNER]: Deduplicated request from leader. Original: 1.583->[1.584-1.584] Dedup: 1.584->[]
I20240502 21:27:03.465266 22963 raft_consensus.cc:2932] T 1df9d3ad5ddc438798543900b6c67b46 P 501e3c0d44fe4d70af9ff926d9622417 [term 1 LEADER]: Committing config change with OpId 1.504: config changed from index 503 to 504, VOTER 82f50f4b66f3450aa9f58a0b317f5cea (127.15.255.130) evicted. New config: { opid_index: 504 OBSOLETE_local: false peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: false } } }
I20240502 21:27:03.466277 22631 raft_consensus.cc:1261] T a7a9256f2b784abcb780b190e2ff3983 P bc20559563fe44efa0a2b7b084aafbce [term 1 FOLLOWER]: Refusing update from remote peer 82f50f4b66f3450aa9f58a0b317f5cea: Log matching property violated. Preceding OpId in replica: term: 1 index: 512. Preceding OpId from leader: term: 1 index: 513. (index mismatch)
I20240502 21:27:03.466593 18472 raft_consensus.cc:1261] T a7a9256f2b784abcb780b190e2ff3983 P faadabfb099b4abf90a466395be585b5 [term 1 FOLLOWER]: Refusing update from remote peer 82f50f4b66f3450aa9f58a0b317f5cea: Log matching property violated. Preceding OpId in replica: term: 1 index: 512. Preceding OpId from leader: term: 1 index: 513. (index mismatch)
I20240502 21:27:03.466809 23028 consensus_queue.cc:1035] T a7a9256f2b784abcb780b190e2ff3983 P 82f50f4b66f3450aa9f58a0b317f5cea [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 513, Last known committed idx: 512, Time since last communication: 0.000s
I20240502 21:27:03.467337 23028 consensus_queue.cc:1035] T a7a9256f2b784abcb780b190e2ff3983 P 82f50f4b66f3450aa9f58a0b317f5cea [LEADER]: Connected to new peer: Peer: permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 513, Last known committed idx: 512, Time since last communication: 0.000s
I20240502 21:27:03.467501 22632 raft_consensus.cc:2932] T 1df9d3ad5ddc438798543900b6c67b46 P bc20559563fe44efa0a2b7b084aafbce [term 1 FOLLOWER]: Committing config change with OpId 1.504: config changed from index 503 to 504, VOTER 82f50f4b66f3450aa9f58a0b317f5cea (127.15.255.130) evicted. New config: { opid_index: 504 OBSOLETE_local: false peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: false } } }
I20240502 21:27:03.468763 20167 raft_consensus.cc:1261] T a7a9256f2b784abcb780b190e2ff3983 P 90e329b457c24caf93a9f880345eb288 [term 1 FOLLOWER]: Refusing update from remote peer 82f50f4b66f3450aa9f58a0b317f5cea: Log matching property violated. Preceding OpId in replica: term: 1 index: 512. Preceding OpId from leader: term: 1 index: 513. (index mismatch)
I20240502 21:27:03.469605 23028 consensus_queue.cc:1035] T a7a9256f2b784abcb780b190e2ff3983 P 82f50f4b66f3450aa9f58a0b317f5cea [LEADER]: Connected to new peer: Peer: permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 513, Last known committed idx: 512, Time since last communication: 0.000s
I20240502 21:27:03.469852 19140 raft_consensus.cc:2932] T 1df9d3ad5ddc438798543900b6c67b46 P 2877108756304e05bd60ba1186b870da [term 1 FOLLOWER]: Committing config change with OpId 1.504: config changed from index 503 to 504, VOTER 82f50f4b66f3450aa9f58a0b317f5cea (127.15.255.130) evicted. New config: { opid_index: 504 OBSOLETE_local: false peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: false } } }
I20240502 21:27:03.472048 18739 raft_consensus.cc:1261] T a7a9256f2b784abcb780b190e2ff3983 P 501e3c0d44fe4d70af9ff926d9622417 [term 1 FOLLOWER]: Refusing update from remote peer 82f50f4b66f3450aa9f58a0b317f5cea: Log matching property violated. Preceding OpId in replica: term: 1 index: 512. Preceding OpId from leader: term: 1 index: 513. (index mismatch)
I20240502 21:27:03.472572 22964 consensus_queue.cc:1035] T a7a9256f2b784abcb780b190e2ff3983 P 82f50f4b66f3450aa9f58a0b317f5cea [LEADER]: Connected to new peer: Peer: permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 513, Last known committed idx: 512, Time since last communication: 0.000s
I20240502 21:27:03.474907 20167 raft_consensus.cc:2932] T a7a9256f2b784abcb780b190e2ff3983 P 90e329b457c24caf93a9f880345eb288 [term 1 FOLLOWER]: Committing config change with OpId 1.513: config changed from index 512 to 513, VOTER c75831edb3b44dd59c00ef151adca49d (127.15.255.132) evicted. New config: { opid_index: 513 OBSOLETE_local: false peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: false } } }
I20240502 21:27:03.475575 18312 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 1df9d3ad5ddc438798543900b6c67b46 with cas_config_opid_index 503: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:27:03.475294 22964 raft_consensus.cc:2932] T a7a9256f2b784abcb780b190e2ff3983 P 82f50f4b66f3450aa9f58a0b317f5cea [term 1 LEADER]: Committing config change with OpId 1.513: config changed from index 512 to 513, VOTER c75831edb3b44dd59c00ef151adca49d (127.15.255.132) evicted. New config: { opid_index: 513 OBSOLETE_local: false peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: false } } }
I20240502 21:27:03.477401 18327 catalog_manager.cc:5484] T 1df9d3ad5ddc438798543900b6c67b46 P bc20559563fe44efa0a2b7b084aafbce reported cstate change: config changed from index 503 to 504, VOTER 82f50f4b66f3450aa9f58a0b317f5cea (127.15.255.130) evicted. New cstate: current_term: 1 leader_uuid: "501e3c0d44fe4d70af9ff926d9622417" committed_config { opid_index: 504 OBSOLETE_local: false peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: false } } }
I20240502 21:27:03.478020 18873 raft_consensus.cc:2932] T 1df9d3ad5ddc438798543900b6c67b46 P c75831edb3b44dd59c00ef151adca49d [term 1 FOLLOWER]: Committing config change with OpId 1.504: config changed from index 503 to 504, VOTER 82f50f4b66f3450aa9f58a0b317f5cea (127.15.255.130) evicted. New config: { opid_index: 504 OBSOLETE_local: false peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: false } } }
I20240502 21:27:03.480794 22631 raft_consensus.cc:2932] T a7a9256f2b784abcb780b190e2ff3983 P bc20559563fe44efa0a2b7b084aafbce [term 1 FOLLOWER]: Committing config change with OpId 1.513: config changed from index 512 to 513, VOTER c75831edb3b44dd59c00ef151adca49d (127.15.255.132) evicted. New config: { opid_index: 513 OBSOLETE_local: false peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: false } } }
I20240502 21:27:03.478996 18739 raft_consensus.cc:2932] T a7a9256f2b784abcb780b190e2ff3983 P 501e3c0d44fe4d70af9ff926d9622417 [term 1 FOLLOWER]: Committing config change with OpId 1.513: config changed from index 512 to 513, VOTER c75831edb3b44dd59c00ef151adca49d (127.15.255.132) evicted. New config: { opid_index: 513 OBSOLETE_local: false peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: false } } }
I20240502 21:27:03.481541 18319 catalog_manager.cc:5484] T a7a9256f2b784abcb780b190e2ff3983 P 90e329b457c24caf93a9f880345eb288 reported cstate change: config changed from index 512 to 513, VOTER c75831edb3b44dd59c00ef151adca49d (127.15.255.132) evicted. New cstate: current_term: 1 leader_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" committed_config { opid_index: 513 OBSOLETE_local: false peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: false } } }
I20240502 21:27:03.485491 18473 raft_consensus.cc:2932] T 1df9d3ad5ddc438798543900b6c67b46 P faadabfb099b4abf90a466395be585b5 [term 1 FOLLOWER]: Committing config change with OpId 1.504: config changed from index 503 to 504, VOTER 82f50f4b66f3450aa9f58a0b317f5cea (127.15.255.130) evicted. New config: { opid_index: 504 OBSOLETE_local: false peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: false } } }
I20240502 21:27:03.486133 18590 tablet_service.cc:1514] Processing DeleteTablet for tablet 1df9d3ad5ddc438798543900b6c67b46 with delete_type TABLET_DATA_TOMBSTONED (TS 82f50f4b66f3450aa9f58a0b317f5cea not found in new config with opid_index 504) from {username='slave'} at 127.0.0.1:33750
I20240502 21:27:03.487437 18315 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet a7a9256f2b784abcb780b190e2ff3983 with cas_config_opid_index 512: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:27:03.488884 18857 tablet_service.cc:1514] Processing DeleteTablet for tablet a7a9256f2b784abcb780b190e2ff3983 with delete_type TABLET_DATA_TOMBSTONED (TS c75831edb3b44dd59c00ef151adca49d not found in new config with opid_index 513) from {username='slave'} at 127.0.0.1:38486
I20240502 21:27:03.488613 18472 raft_consensus.cc:2932] T a7a9256f2b784abcb780b190e2ff3983 P faadabfb099b4abf90a466395be585b5 [term 1 FOLLOWER]: Committing config change with OpId 1.513: config changed from index 512 to 513, VOTER c75831edb3b44dd59c00ef151adca49d (127.15.255.132) evicted. New config: { opid_index: 513 OBSOLETE_local: false peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: false } } }
I20240502 21:27:03.490861 23039 tablet_replica.cc:330] T 1df9d3ad5ddc438798543900b6c67b46 P 82f50f4b66f3450aa9f58a0b317f5cea: stopping tablet replica
I20240502 21:27:03.491056 23039 raft_consensus.cc:2229] T 1df9d3ad5ddc438798543900b6c67b46 P 82f50f4b66f3450aa9f58a0b317f5cea [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:03.491333 23039 raft_consensus.cc:2258] T 1df9d3ad5ddc438798543900b6c67b46 P 82f50f4b66f3450aa9f58a0b317f5cea [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:03.495280 23039 ts_tablet_manager.cc:1875] T 1df9d3ad5ddc438798543900b6c67b46 P 82f50f4b66f3450aa9f58a0b317f5cea: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:27:03.496706 22990 raft_consensus.cc:1050] T ebf7371ff0ef4de99ced8fec0468af28 P c75831edb3b44dd59c00ef151adca49d: attempting to promote NON_VOTER bc20559563fe44efa0a2b7b084aafbce to VOTER
I20240502 21:27:03.497728 22990 consensus_queue.cc:237] T ebf7371ff0ef4de99ced8fec0468af28 P c75831edb3b44dd59c00ef151adca49d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 387, Committed index: 387, Last appended: 1.387, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 388 OBSOLETE_local: false peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } attrs { replace: true } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: false } }
I20240502 21:27:03.498018 23040 tablet_replica.cc:330] T a7a9256f2b784abcb780b190e2ff3983 P c75831edb3b44dd59c00ef151adca49d: stopping tablet replica
I20240502 21:27:03.498191 23040 raft_consensus.cc:2229] T a7a9256f2b784abcb780b190e2ff3983 P c75831edb3b44dd59c00ef151adca49d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:03.498414 23040 raft_consensus.cc:2258] T a7a9256f2b784abcb780b190e2ff3983 P c75831edb3b44dd59c00ef151adca49d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:03.501379 23039 ts_tablet_manager.cc:1888] T 1df9d3ad5ddc438798543900b6c67b46 P 82f50f4b66f3450aa9f58a0b317f5cea: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.503
I20240502 21:27:03.501567 23039 log.cc:1196] T 1df9d3ad5ddc438798543900b6c67b46 P 82f50f4b66f3450aa9f58a0b317f5cea: Deleting WAL directory at /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-1/wal/wals/1df9d3ad5ddc438798543900b6c67b46
I20240502 21:27:03.502388 18315 catalog_manager.cc:4830] TS 82f50f4b66f3450aa9f58a0b317f5cea (127.15.255.130:45139): tablet 1df9d3ad5ddc438798543900b6c67b46 (table rebalance_test_table_4 [id=c79c94b660074ffc94fcea415b4caae8]) successfully deleted
I20240502 21:27:03.503006 23040 ts_tablet_manager.cc:1875] T a7a9256f2b784abcb780b190e2ff3983 P c75831edb3b44dd59c00ef151adca49d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:27:03.505625 20167 raft_consensus.cc:1261] T ebf7371ff0ef4de99ced8fec0468af28 P 90e329b457c24caf93a9f880345eb288 [term 1 FOLLOWER]: Refusing update from remote peer c75831edb3b44dd59c00ef151adca49d: Log matching property violated. Preceding OpId in replica: term: 1 index: 387. Preceding OpId from leader: term: 1 index: 388. (index mismatch)
I20240502 21:27:03.506057 19140 raft_consensus.cc:1261] T ebf7371ff0ef4de99ced8fec0468af28 P 2877108756304e05bd60ba1186b870da [term 1 FOLLOWER]: Refusing update from remote peer c75831edb3b44dd59c00ef151adca49d: Log matching property violated. Preceding OpId in replica: term: 1 index: 387. Preceding OpId from leader: term: 1 index: 388. (index mismatch)
I20240502 21:27:03.506261 22631 raft_consensus.cc:1261] T ebf7371ff0ef4de99ced8fec0468af28 P bc20559563fe44efa0a2b7b084aafbce [term 1 LEARNER]: Refusing update from remote peer c75831edb3b44dd59c00ef151adca49d: Log matching property violated. Preceding OpId in replica: term: 1 index: 387. Preceding OpId from leader: term: 1 index: 388. (index mismatch)
I20240502 21:27:03.506538 22798 consensus_queue.cc:1035] T ebf7371ff0ef4de99ced8fec0468af28 P c75831edb3b44dd59c00ef151adca49d [LEADER]: Connected to new peer: Peer: permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 388, Last known committed idx: 387, Time since last communication: 0.000s
I20240502 21:27:03.507084 22798 consensus_queue.cc:1035] T ebf7371ff0ef4de99ced8fec0468af28 P c75831edb3b44dd59c00ef151adca49d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 388, Last known committed idx: 387, Time since last communication: 0.000s
I20240502 21:27:03.507879 22798 consensus_queue.cc:1035] T ebf7371ff0ef4de99ced8fec0468af28 P c75831edb3b44dd59c00ef151adca49d [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 388, Last known committed idx: 387, Time since last communication: 0.000s
I20240502 21:27:03.509209 23040 ts_tablet_manager.cc:1888] T a7a9256f2b784abcb780b190e2ff3983 P c75831edb3b44dd59c00ef151adca49d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.512
I20240502 21:27:03.509375 23040 log.cc:1196] T a7a9256f2b784abcb780b190e2ff3983 P c75831edb3b44dd59c00ef151adca49d: Deleting WAL directory at /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-3/wal/wals/a7a9256f2b784abcb780b190e2ff3983
I20240502 21:27:03.510272 18314 catalog_manager.cc:4830] TS c75831edb3b44dd59c00ef151adca49d (127.15.255.132:43607): tablet a7a9256f2b784abcb780b190e2ff3983 (table rebalance_test_table_1 [id=76deee1e4597471ebedfc30f6d635eaa]) successfully deleted
I20240502 21:27:03.511317 19005 raft_consensus.cc:1261] T ebf7371ff0ef4de99ced8fec0468af28 P 9be399440ded4e2ebe6b6e0bf486cbb1 [term 1 FOLLOWER]: Refusing update from remote peer c75831edb3b44dd59c00ef151adca49d: Log matching property violated. Preceding OpId in replica: term: 1 index: 387. Preceding OpId from leader: term: 1 index: 388. (index mismatch)
I20240502 21:27:03.511868 22990 consensus_queue.cc:1035] T ebf7371ff0ef4de99ced8fec0468af28 P c75831edb3b44dd59c00ef151adca49d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 388, Last known committed idx: 387, Time since last communication: 0.000s
I20240502 21:27:03.514632 18739 raft_consensus.cc:1261] T ebf7371ff0ef4de99ced8fec0468af28 P 501e3c0d44fe4d70af9ff926d9622417 [term 1 FOLLOWER]: Refusing update from remote peer c75831edb3b44dd59c00ef151adca49d: Log matching property violated. Preceding OpId in replica: term: 1 index: 387. Preceding OpId from leader: term: 1 index: 388. (index mismatch)
I20240502 21:27:03.515163 22990 consensus_queue.cc:1035] T ebf7371ff0ef4de99ced8fec0468af28 P c75831edb3b44dd59c00ef151adca49d [LEADER]: Connected to new peer: Peer: permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 388, Last known committed idx: 387, Time since last communication: 0.000s
I20240502 21:27:03.518373 22990 raft_consensus.cc:2932] T ebf7371ff0ef4de99ced8fec0468af28 P c75831edb3b44dd59c00ef151adca49d [term 1 LEADER]: Committing config change with OpId 1.388: config changed from index 387 to 388, bc20559563fe44efa0a2b7b084aafbce (127.15.255.136) changed from NON_VOTER to VOTER. New config: { opid_index: 388 OBSOLETE_local: false peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } attrs { replace: true } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: false } } }
I20240502 21:27:03.518824 18739 raft_consensus.cc:2932] T ebf7371ff0ef4de99ced8fec0468af28 P 501e3c0d44fe4d70af9ff926d9622417 [term 1 FOLLOWER]: Committing config change with OpId 1.388: config changed from index 387 to 388, bc20559563fe44efa0a2b7b084aafbce (127.15.255.136) changed from NON_VOTER to VOTER. New config: { opid_index: 388 OBSOLETE_local: false peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } attrs { replace: true } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: false } } }
I20240502 21:27:03.523172 19140 raft_consensus.cc:2932] T ebf7371ff0ef4de99ced8fec0468af28 P 2877108756304e05bd60ba1186b870da [term 1 FOLLOWER]: Committing config change with OpId 1.388: config changed from index 387 to 388, bc20559563fe44efa0a2b7b084aafbce (127.15.255.136) changed from NON_VOTER to VOTER. New config: { opid_index: 388 OBSOLETE_local: false peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } attrs { replace: true } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: false } } }
I20240502 21:27:03.523589 19005 raft_consensus.cc:2932] T ebf7371ff0ef4de99ced8fec0468af28 P 9be399440ded4e2ebe6b6e0bf486cbb1 [term 1 FOLLOWER]: Committing config change with OpId 1.388: config changed from index 387 to 388, bc20559563fe44efa0a2b7b084aafbce (127.15.255.136) changed from NON_VOTER to VOTER. New config: { opid_index: 388 OBSOLETE_local: false peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } attrs { replace: true } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: false } } }
I20240502 21:27:03.524773 18327 catalog_manager.cc:5484] T ebf7371ff0ef4de99ced8fec0468af28 P 501e3c0d44fe4d70af9ff926d9622417 reported cstate change: config changed from index 387 to 388, bc20559563fe44efa0a2b7b084aafbce (127.15.255.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c75831edb3b44dd59c00ef151adca49d" committed_config { opid_index: 388 OBSOLETE_local: false peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } attrs { replace: true } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: false } } }
I20240502 21:27:03.537808 22631 raft_consensus.cc:2932] T ebf7371ff0ef4de99ced8fec0468af28 P bc20559563fe44efa0a2b7b084aafbce [term 1 FOLLOWER]: Committing config change with OpId 1.388: config changed from index 387 to 388, bc20559563fe44efa0a2b7b084aafbce (127.15.255.136) changed from NON_VOTER to VOTER. New config: { opid_index: 388 OBSOLETE_local: false peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } attrs { replace: true } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: false } } }
I20240502 21:27:03.537955 20167 raft_consensus.cc:2932] T ebf7371ff0ef4de99ced8fec0468af28 P 90e329b457c24caf93a9f880345eb288 [term 1 FOLLOWER]: Committing config change with OpId 1.388: config changed from index 387 to 388, bc20559563fe44efa0a2b7b084aafbce (127.15.255.136) changed from NON_VOTER to VOTER. New config: { opid_index: 388 OBSOLETE_local: false peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } attrs { replace: true } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: false } } }
I20240502 21:27:03.541229 18873 consensus_queue.cc:237] T ebf7371ff0ef4de99ced8fec0468af28 P c75831edb3b44dd59c00ef151adca49d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 388, Committed index: 388, Last appended: 1.388, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 389 OBSOLETE_local: false peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: false } }
I20240502 21:27:03.543376 20167 raft_consensus.cc:1261] T ebf7371ff0ef4de99ced8fec0468af28 P 90e329b457c24caf93a9f880345eb288 [term 1 FOLLOWER]: Refusing update from remote peer c75831edb3b44dd59c00ef151adca49d: Log matching property violated. Preceding OpId in replica: term: 1 index: 388. Preceding OpId from leader: term: 1 index: 389. (index mismatch)
I20240502 21:27:03.543769 19140 raft_consensus.cc:1261] T ebf7371ff0ef4de99ced8fec0468af28 P 2877108756304e05bd60ba1186b870da [term 1 FOLLOWER]: Refusing update from remote peer c75831edb3b44dd59c00ef151adca49d: Log matching property violated. Preceding OpId in replica: term: 1 index: 388. Preceding OpId from leader: term: 1 index: 389. (index mismatch)
I20240502 21:27:03.544057 23047 consensus_queue.cc:1035] T ebf7371ff0ef4de99ced8fec0468af28 P c75831edb3b44dd59c00ef151adca49d [LEADER]: Connected to new peer: Peer: permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 389, Last known committed idx: 388, Time since last communication: 0.000s
I20240502 21:27:03.545410 23047 consensus_queue.cc:1035] T ebf7371ff0ef4de99ced8fec0468af28 P c75831edb3b44dd59c00ef151adca49d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 389, Last known committed idx: 388, Time since last communication: 0.000s
I20240502 21:27:03.547017 22631 raft_consensus.cc:1261] T ebf7371ff0ef4de99ced8fec0468af28 P bc20559563fe44efa0a2b7b084aafbce [term 1 FOLLOWER]: Refusing update from remote peer c75831edb3b44dd59c00ef151adca49d: Log matching property violated. Preceding OpId in replica: term: 1 index: 388. Preceding OpId from leader: term: 1 index: 389. (index mismatch)
I20240502 21:27:03.547130 18739 raft_consensus.cc:1261] T ebf7371ff0ef4de99ced8fec0468af28 P 501e3c0d44fe4d70af9ff926d9622417 [term 1 FOLLOWER]: Refusing update from remote peer c75831edb3b44dd59c00ef151adca49d: Log matching property violated. Preceding OpId in replica: term: 1 index: 388. Preceding OpId from leader: term: 1 index: 389. (index mismatch)
I20240502 21:27:03.547468 23047 consensus_queue.cc:1035] T ebf7371ff0ef4de99ced8fec0468af28 P c75831edb3b44dd59c00ef151adca49d [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 389, Last known committed idx: 388, Time since last communication: 0.000s
I20240502 21:27:03.547778 22990 consensus_queue.cc:1035] T ebf7371ff0ef4de99ced8fec0468af28 P c75831edb3b44dd59c00ef151adca49d [LEADER]: Connected to new peer: Peer: permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 389, Last known committed idx: 388, Time since last communication: 0.000s
I20240502 21:27:03.547792 22798 raft_consensus.cc:2932] T ebf7371ff0ef4de99ced8fec0468af28 P c75831edb3b44dd59c00ef151adca49d [term 1 LEADER]: Committing config change with OpId 1.389: config changed from index 388 to 389, VOTER 9be399440ded4e2ebe6b6e0bf486cbb1 (127.15.255.133) evicted. New config: { opid_index: 389 OBSOLETE_local: false peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: false } } }
I20240502 21:27:03.548898 19140 raft_consensus.cc:2932] T ebf7371ff0ef4de99ced8fec0468af28 P 2877108756304e05bd60ba1186b870da [term 1 FOLLOWER]: Committing config change with OpId 1.389: config changed from index 388 to 389, VOTER 9be399440ded4e2ebe6b6e0bf486cbb1 (127.15.255.133) evicted. New config: { opid_index: 389 OBSOLETE_local: false peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: false } } }
I20240502 21:27:03.549423 18739 raft_consensus.cc:2932] T ebf7371ff0ef4de99ced8fec0468af28 P 501e3c0d44fe4d70af9ff926d9622417 [term 1 FOLLOWER]: Committing config change with OpId 1.389: config changed from index 388 to 389, VOTER 9be399440ded4e2ebe6b6e0bf486cbb1 (127.15.255.133) evicted. New config: { opid_index: 389 OBSOLETE_local: false peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: false } } }
I20240502 21:27:03.549361 22631 raft_consensus.cc:2932] T ebf7371ff0ef4de99ced8fec0468af28 P bc20559563fe44efa0a2b7b084aafbce [term 1 FOLLOWER]: Committing config change with OpId 1.389: config changed from index 388 to 389, VOTER 9be399440ded4e2ebe6b6e0bf486cbb1 (127.15.255.133) evicted. New config: { opid_index: 389 OBSOLETE_local: false peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: false } } }
I20240502 21:27:03.551512 18314 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet ebf7371ff0ef4de99ced8fec0468af28 with cas_config_opid_index 388: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:27:03.552199 20167 raft_consensus.cc:2932] T ebf7371ff0ef4de99ced8fec0468af28 P 90e329b457c24caf93a9f880345eb288 [term 1 FOLLOWER]: Committing config change with OpId 1.389: config changed from index 388 to 389, VOTER 9be399440ded4e2ebe6b6e0bf486cbb1 (127.15.255.133) evicted. New config: { opid_index: 389 OBSOLETE_local: false peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: false } } }
I20240502 21:27:03.553303 18319 catalog_manager.cc:5484] T ebf7371ff0ef4de99ced8fec0468af28 P c75831edb3b44dd59c00ef151adca49d reported cstate change: config changed from index 388 to 389, VOTER 9be399440ded4e2ebe6b6e0bf486cbb1 (127.15.255.133) evicted. New cstate: current_term: 1 leader_uuid: "c75831edb3b44dd59c00ef151adca49d" committed_config { opid_index: 389 OBSOLETE_local: false peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:03.557587 18989 tablet_service.cc:1514] Processing DeleteTablet for tablet ebf7371ff0ef4de99ced8fec0468af28 with delete_type TABLET_DATA_TOMBSTONED (TS 9be399440ded4e2ebe6b6e0bf486cbb1 not found in new config with opid_index 389) from {username='slave'} at 127.0.0.1:40858
I20240502 21:27:03.565315 23048 tablet_replica.cc:330] T ebf7371ff0ef4de99ced8fec0468af28 P 9be399440ded4e2ebe6b6e0bf486cbb1: stopping tablet replica
I20240502 21:27:03.565501 23048 raft_consensus.cc:2229] T ebf7371ff0ef4de99ced8fec0468af28 P 9be399440ded4e2ebe6b6e0bf486cbb1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:03.565753 23048 raft_consensus.cc:2258] T ebf7371ff0ef4de99ced8fec0468af28 P 9be399440ded4e2ebe6b6e0bf486cbb1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:03.569037 23048 ts_tablet_manager.cc:1875] T ebf7371ff0ef4de99ced8fec0468af28 P 9be399440ded4e2ebe6b6e0bf486cbb1: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:27:03.575042 23048 ts_tablet_manager.cc:1888] T ebf7371ff0ef4de99ced8fec0468af28 P 9be399440ded4e2ebe6b6e0bf486cbb1: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.388
I20240502 21:27:03.575150 23048 log.cc:1196] T ebf7371ff0ef4de99ced8fec0468af28 P 9be399440ded4e2ebe6b6e0bf486cbb1: Deleting WAL directory at /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-4/wal/wals/ebf7371ff0ef4de99ced8fec0468af28
I20240502 21:27:03.575767 18313 catalog_manager.cc:4830] TS 9be399440ded4e2ebe6b6e0bf486cbb1 (127.15.255.133:34217): tablet ebf7371ff0ef4de99ced8fec0468af28 (table rebalance_test_table_3 [id=5af23c390f054d07b5a7985c2611ca30]) successfully deleted
I20240502 21:27:03.633311 18473 consensus_queue.cc:237] T 40d4f61d0f6d4d31a5626c7743c88133 P faadabfb099b4abf90a466395be585b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } attrs { replace: true } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: true } }
I20240502 21:27:03.635687 20167 raft_consensus.cc:1261] T 40d4f61d0f6d4d31a5626c7743c88133 P 90e329b457c24caf93a9f880345eb288 [term 1 FOLLOWER]: Refusing update from remote peer faadabfb099b4abf90a466395be585b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:27:03.635947 18739 raft_consensus.cc:1261] T 40d4f61d0f6d4d31a5626c7743c88133 P 501e3c0d44fe4d70af9ff926d9622417 [term 1 FOLLOWER]: Refusing update from remote peer faadabfb099b4abf90a466395be585b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 21:27:03.636554 18404 consensus_peers.cc:485] T 40d4f61d0f6d4d31a5626c7743c88133 P faadabfb099b4abf90a466395be585b5 -> Peer bc20559563fe44efa0a2b7b084aafbce (127.15.255.136:34965): Couldn't send request to peer bc20559563fe44efa0a2b7b084aafbce. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 40d4f61d0f6d4d31a5626c7743c88133. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:03.636557 22992 consensus_queue.cc:1035] T 40d4f61d0f6d4d31a5626c7743c88133 P faadabfb099b4abf90a466395be585b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:27:03.636893 23080 consensus_queue.cc:1035] T 40d4f61d0f6d4d31a5626c7743c88133 P faadabfb099b4abf90a466395be585b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:27:03.637053 19005 raft_consensus.cc:1261] T 40d4f61d0f6d4d31a5626c7743c88133 P 9be399440ded4e2ebe6b6e0bf486cbb1 [term 1 FOLLOWER]: Refusing update from remote peer faadabfb099b4abf90a466395be585b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:27:03.637549 23080 consensus_queue.cc:1035] T 40d4f61d0f6d4d31a5626c7743c88133 P faadabfb099b4abf90a466395be585b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:27:03.638072 18611 raft_consensus.cc:1261] T 40d4f61d0f6d4d31a5626c7743c88133 P 82f50f4b66f3450aa9f58a0b317f5cea [term 1 FOLLOWER]: Refusing update from remote peer faadabfb099b4abf90a466395be585b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:27:03.638562 22992 consensus_queue.cc:1035] T 40d4f61d0f6d4d31a5626c7743c88133 P faadabfb099b4abf90a466395be585b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 21:27:03.641112 23080 raft_consensus.cc:2932] T 40d4f61d0f6d4d31a5626c7743c88133 P faadabfb099b4abf90a466395be585b5 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER bc20559563fe44efa0a2b7b084aafbce (127.15.255.136) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } attrs { replace: true } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: true } } }
I20240502 21:27:03.641677 18611 raft_consensus.cc:2932] T 40d4f61d0f6d4d31a5626c7743c88133 P 82f50f4b66f3450aa9f58a0b317f5cea [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER bc20559563fe44efa0a2b7b084aafbce (127.15.255.136) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } attrs { replace: true } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: true } } }
I20240502 21:27:03.646335 18319 catalog_manager.cc:5484] T 40d4f61d0f6d4d31a5626c7743c88133 P 82f50f4b66f3450aa9f58a0b317f5cea reported cstate change: config changed from index -1 to 2, NON_VOTER bc20559563fe44efa0a2b7b084aafbce (127.15.255.136) added. New cstate: current_term: 1 leader_uuid: "faadabfb099b4abf90a466395be585b5" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } attrs { replace: true } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: true } } }
I20240502 21:27:03.646972 19005 raft_consensus.cc:2932] T 40d4f61d0f6d4d31a5626c7743c88133 P 9be399440ded4e2ebe6b6e0bf486cbb1 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER bc20559563fe44efa0a2b7b084aafbce (127.15.255.136) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } attrs { replace: true } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: true } } }
I20240502 21:27:03.647686 18739 raft_consensus.cc:2932] T 40d4f61d0f6d4d31a5626c7743c88133 P 501e3c0d44fe4d70af9ff926d9622417 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER bc20559563fe44efa0a2b7b084aafbce (127.15.255.136) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } attrs { replace: true } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: true } } }
I20240502 21:27:03.666440 20167 raft_consensus.cc:2932] T 40d4f61d0f6d4d31a5626c7743c88133 P 90e329b457c24caf93a9f880345eb288 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER bc20559563fe44efa0a2b7b084aafbce (127.15.255.136) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } attrs { replace: true } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: true } } }
I20240502 21:27:03.681149 19140 consensus_queue.cc:237] T b0852f54089944ce935f8ee4cbf2bc5a P 2877108756304e05bd60ba1186b870da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 452, Committed index: 452, Last appended: 1.452, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 453 OBSOLETE_local: false peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false replace: true } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: true } }
I20240502 21:27:03.683981 20167 raft_consensus.cc:1261] T b0852f54089944ce935f8ee4cbf2bc5a P 90e329b457c24caf93a9f880345eb288 [term 1 FOLLOWER]: Refusing update from remote peer 2877108756304e05bd60ba1186b870da: Log matching property violated. Preceding OpId in replica: term: 1 index: 452. Preceding OpId from leader: term: 1 index: 453. (index mismatch)
I20240502 21:27:03.684125 18611 raft_consensus.cc:1261] T b0852f54089944ce935f8ee4cbf2bc5a P 82f50f4b66f3450aa9f58a0b317f5cea [term 1 FOLLOWER]: Refusing update from remote peer 2877108756304e05bd60ba1186b870da: Log matching property violated. Preceding OpId in replica: term: 1 index: 452. Preceding OpId from leader: term: 1 index: 453. (index mismatch)
I20240502 21:27:03.685029 22419 consensus_queue.cc:1035] T b0852f54089944ce935f8ee4cbf2bc5a P 2877108756304e05bd60ba1186b870da [LEADER]: Connected to new peer: Peer: permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 453, Last known committed idx: 452, Time since last communication: 0.000s
I20240502 21:27:03.686398 22419 consensus_queue.cc:1035] T b0852f54089944ce935f8ee4cbf2bc5a P 2877108756304e05bd60ba1186b870da [LEADER]: Connected to new peer: Peer: permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 453, Last known committed idx: 452, Time since last communication: 0.000s
I20240502 21:27:03.692276 22992 raft_consensus.cc:1050] T ed78d40318fe4ab5bee4a01b182a34b6 P faadabfb099b4abf90a466395be585b5: attempting to promote NON_VOTER bc20559563fe44efa0a2b7b084aafbce to VOTER
I20240502 21:27:03.693322 22992 consensus_queue.cc:237] T ed78d40318fe4ab5bee4a01b182a34b6 P faadabfb099b4abf90a466395be585b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 498, Committed index: 498, Last appended: 1.498, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 499 OBSOLETE_local: false peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } attrs { replace: true } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: false } }
I20240502 21:27:03.696575 18739 raft_consensus.cc:1261] T b0852f54089944ce935f8ee4cbf2bc5a P 501e3c0d44fe4d70af9ff926d9622417 [term 1 FOLLOWER]: Refusing update from remote peer 2877108756304e05bd60ba1186b870da: Log matching property violated. Preceding OpId in replica: term: 1 index: 452. Preceding OpId from leader: term: 1 index: 453. (index mismatch)
I20240502 21:27:03.697161 22419 consensus_queue.cc:1035] T b0852f54089944ce935f8ee4cbf2bc5a P 2877108756304e05bd60ba1186b870da [LEADER]: Connected to new peer: Peer: permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 453, Last known committed idx: 452, Time since last communication: 0.000s
I20240502 21:27:03.701719 19005 raft_consensus.cc:1261] T ed78d40318fe4ab5bee4a01b182a34b6 P 9be399440ded4e2ebe6b6e0bf486cbb1 [term 1 FOLLOWER]: Refusing update from remote peer faadabfb099b4abf90a466395be585b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 498. Preceding OpId from leader: term: 1 index: 499. (index mismatch)
I20240502 21:27:03.702239 23080 consensus_queue.cc:1035] T ed78d40318fe4ab5bee4a01b182a34b6 P faadabfb099b4abf90a466395be585b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 499, Last known committed idx: 498, Time since last communication: 0.000s
I20240502 21:27:03.706683 18744 raft_consensus.cc:1261] T ed78d40318fe4ab5bee4a01b182a34b6 P 501e3c0d44fe4d70af9ff926d9622417 [term 1 FOLLOWER]: Refusing update from remote peer faadabfb099b4abf90a466395be585b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 498. Preceding OpId from leader: term: 1 index: 499. (index mismatch)
I20240502 21:27:03.706991 22631 raft_consensus.cc:1261] T ed78d40318fe4ab5bee4a01b182a34b6 P bc20559563fe44efa0a2b7b084aafbce [term 1 LEARNER]: Refusing update from remote peer faadabfb099b4abf90a466395be585b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 498. Preceding OpId from leader: term: 1 index: 499. (index mismatch)
I20240502 21:27:03.707680 22833 consensus_queue.cc:1035] T ed78d40318fe4ab5bee4a01b182a34b6 P faadabfb099b4abf90a466395be585b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 499, Last known committed idx: 498, Time since last communication: 0.000s
I20240502 21:27:03.708274 22833 consensus_queue.cc:1035] T ed78d40318fe4ab5bee4a01b182a34b6 P faadabfb099b4abf90a466395be585b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 499, Last known committed idx: 498, Time since last communication: 0.000s
I20240502 21:27:03.714232 18873 raft_consensus.cc:1261] T ed78d40318fe4ab5bee4a01b182a34b6 P c75831edb3b44dd59c00ef151adca49d [term 1 FOLLOWER]: Refusing update from remote peer faadabfb099b4abf90a466395be585b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 498. Preceding OpId from leader: term: 1 index: 499. (index mismatch)
I20240502 21:27:03.714964 22833 consensus_queue.cc:1035] T ed78d40318fe4ab5bee4a01b182a34b6 P faadabfb099b4abf90a466395be585b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 499, Last known committed idx: 498, Time since last communication: 0.000s
I20240502 21:27:03.717397 19146 raft_consensus.cc:1261] T ed78d40318fe4ab5bee4a01b182a34b6 P 2877108756304e05bd60ba1186b870da [term 1 FOLLOWER]: Refusing update from remote peer faadabfb099b4abf90a466395be585b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 498. Preceding OpId from leader: term: 1 index: 499. (index mismatch)
I20240502 21:27:03.717981 22833 consensus_queue.cc:1035] T ed78d40318fe4ab5bee4a01b182a34b6 P faadabfb099b4abf90a466395be585b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 499, Last known committed idx: 498, Time since last communication: 0.000s
I20240502 21:27:03.749868 18739 raft_consensus.cc:2932] T b0852f54089944ce935f8ee4cbf2bc5a P 501e3c0d44fe4d70af9ff926d9622417 [term 1 FOLLOWER]: Committing config change with OpId 1.453: config changed from index 206 to 453, NON_VOTER bc20559563fe44efa0a2b7b084aafbce (127.15.255.136) added. New config: { opid_index: 453 OBSOLETE_local: false peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false replace: true } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: true } } }
I20240502 21:27:03.748883 22419 raft_consensus.cc:2932] T b0852f54089944ce935f8ee4cbf2bc5a P 2877108756304e05bd60ba1186b870da [term 1 LEADER]: Committing config change with OpId 1.453: config changed from index 206 to 453, NON_VOTER bc20559563fe44efa0a2b7b084aafbce (127.15.255.136) added. New config: { opid_index: 453 OBSOLETE_local: false peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false replace: true } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: true } } }
I20240502 21:27:03.759223 18319 catalog_manager.cc:5484] T b0852f54089944ce935f8ee4cbf2bc5a P 501e3c0d44fe4d70af9ff926d9622417 reported cstate change: config changed from index 206 to 453, NON_VOTER bc20559563fe44efa0a2b7b084aafbce (127.15.255.136) added. New cstate: current_term: 1 leader_uuid: "2877108756304e05bd60ba1186b870da" committed_config { opid_index: 453 OBSOLETE_local: false peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false replace: true } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: true } } }
I20240502 21:27:03.765148 18610 raft_consensus.cc:2932] T b0852f54089944ce935f8ee4cbf2bc5a P 82f50f4b66f3450aa9f58a0b317f5cea [term 1 FOLLOWER]: Committing config change with OpId 1.453: config changed from index 206 to 453, NON_VOTER bc20559563fe44efa0a2b7b084aafbce (127.15.255.136) added. New config: { opid_index: 453 OBSOLETE_local: false peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false replace: true } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: true } } }
I20240502 21:27:03.776515 22992 raft_consensus.cc:2932] T ed78d40318fe4ab5bee4a01b182a34b6 P faadabfb099b4abf90a466395be585b5 [term 1 LEADER]: Committing config change with OpId 1.499: config changed from index 498 to 499, bc20559563fe44efa0a2b7b084aafbce (127.15.255.136) changed from NON_VOTER to VOTER. New config: { opid_index: 499 OBSOLETE_local: false peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } attrs { replace: true } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: false } } }
I20240502 21:27:03.780264 19005 raft_consensus.cc:1261] T b0852f54089944ce935f8ee4cbf2bc5a P 9be399440ded4e2ebe6b6e0bf486cbb1 [term 1 FOLLOWER]: Refusing update from remote peer 2877108756304e05bd60ba1186b870da: Log matching property violated. Preceding OpId in replica: term: 1 index: 452. Preceding OpId from leader: term: 1 index: 453. (index mismatch)
I20240502 21:27:03.780282 20167 raft_consensus.cc:2932] T b0852f54089944ce935f8ee4cbf2bc5a P 90e329b457c24caf93a9f880345eb288 [term 1 FOLLOWER]: Committing config change with OpId 1.453: config changed from index 206 to 453, NON_VOTER bc20559563fe44efa0a2b7b084aafbce (127.15.255.136) added. New config: { opid_index: 453 OBSOLETE_local: false peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false replace: true } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: true } } }
I20240502 21:27:03.780845 23092 consensus_queue.cc:1035] T b0852f54089944ce935f8ee4cbf2bc5a P 2877108756304e05bd60ba1186b870da [LEADER]: Connected to new peer: Peer: permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 453, Last known committed idx: 452, Time since last communication: 0.000s
I20240502 21:27:03.781921 19009 raft_consensus.cc:2932] T ed78d40318fe4ab5bee4a01b182a34b6 P 9be399440ded4e2ebe6b6e0bf486cbb1 [term 1 FOLLOWER]: Committing config change with OpId 1.499: config changed from index 498 to 499, bc20559563fe44efa0a2b7b084aafbce (127.15.255.136) changed from NON_VOTER to VOTER. New config: { opid_index: 499 OBSOLETE_local: false peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } attrs { replace: true } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: false } } }
I20240502 21:27:03.787598 22631 raft_consensus.cc:2932] T ed78d40318fe4ab5bee4a01b182a34b6 P bc20559563fe44efa0a2b7b084aafbce [term 1 FOLLOWER]: Committing config change with OpId 1.499: config changed from index 498 to 499, bc20559563fe44efa0a2b7b084aafbce (127.15.255.136) changed from NON_VOTER to VOTER. New config: { opid_index: 499 OBSOLETE_local: false peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } attrs { replace: true } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: false } } }
I20240502 21:27:03.791013 19005 raft_consensus.cc:2932] T b0852f54089944ce935f8ee4cbf2bc5a P 9be399440ded4e2ebe6b6e0bf486cbb1 [term 1 FOLLOWER]: Committing config change with OpId 1.453: config changed from index 206 to 453, NON_VOTER bc20559563fe44efa0a2b7b084aafbce (127.15.255.136) added. New config: { opid_index: 453 OBSOLETE_local: false peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false replace: true } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: true } } }
I20240502 21:27:03.791569 18744 raft_consensus.cc:2932] T ed78d40318fe4ab5bee4a01b182a34b6 P 501e3c0d44fe4d70af9ff926d9622417 [term 1 FOLLOWER]: Committing config change with OpId 1.499: config changed from index 498 to 499, bc20559563fe44efa0a2b7b084aafbce (127.15.255.136) changed from NON_VOTER to VOTER. New config: { opid_index: 499 OBSOLETE_local: false peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } attrs { replace: true } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: false } } }
I20240502 21:27:03.800184 18324 catalog_manager.cc:5484] T ed78d40318fe4ab5bee4a01b182a34b6 P bc20559563fe44efa0a2b7b084aafbce reported cstate change: config changed from index 498 to 499, bc20559563fe44efa0a2b7b084aafbce (127.15.255.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "faadabfb099b4abf90a466395be585b5" committed_config { opid_index: 499 OBSOLETE_local: false peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } attrs { replace: true } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: false } } }
I20240502 21:27:03.802805 19140 raft_consensus.cc:2932] T ed78d40318fe4ab5bee4a01b182a34b6 P 2877108756304e05bd60ba1186b870da [term 1 FOLLOWER]: Committing config change with OpId 1.499: config changed from index 498 to 499, bc20559563fe44efa0a2b7b084aafbce (127.15.255.136) changed from NON_VOTER to VOTER. New config: { opid_index: 499 OBSOLETE_local: false peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } attrs { replace: true } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: false } } }
I20240502 21:27:03.803470 18873 raft_consensus.cc:2932] T ed78d40318fe4ab5bee4a01b182a34b6 P c75831edb3b44dd59c00ef151adca49d [term 1 FOLLOWER]: Committing config change with OpId 1.499: config changed from index 498 to 499, bc20559563fe44efa0a2b7b084aafbce (127.15.255.136) changed from NON_VOTER to VOTER. New config: { opid_index: 499 OBSOLETE_local: false peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } attrs { replace: true } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: false } } }
W20240502 21:27:03.814198 19076 consensus_peers.cc:485] T b0852f54089944ce935f8ee4cbf2bc5a P 2877108756304e05bd60ba1186b870da -> Peer bc20559563fe44efa0a2b7b084aafbce (127.15.255.136:34965): Couldn't send request to peer bc20559563fe44efa0a2b7b084aafbce. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b0852f54089944ce935f8ee4cbf2bc5a. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:03.822312 18873 consensus_queue.cc:237] T 37e9dcc6191c42d3a5c1503dedc5511e P c75831edb3b44dd59c00ef151adca49d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 531, Committed index: 531, Last appended: 1.531, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 532 OBSOLETE_local: false peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } attrs { replace: true } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } attrs { promote: false } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: true } }
I20240502 21:27:03.825268 18473 raft_consensus.cc:1261] T 37e9dcc6191c42d3a5c1503dedc5511e P faadabfb099b4abf90a466395be585b5 [term 1 FOLLOWER]: Refusing update from remote peer c75831edb3b44dd59c00ef151adca49d: Log matching property violated. Preceding OpId in replica: term: 1 index: 531. Preceding OpId from leader: term: 1 index: 532. (index mismatch)
I20240502 21:27:03.825335 19146 raft_consensus.cc:1261] T 37e9dcc6191c42d3a5c1503dedc5511e P 2877108756304e05bd60ba1186b870da [term 1 FOLLOWER]: Refusing update from remote peer c75831edb3b44dd59c00ef151adca49d: Log matching property violated. Preceding OpId in replica: term: 1 index: 531. Preceding OpId from leader: term: 1 index: 532. (index mismatch)
W20240502 21:27:03.825771 18809 consensus_peers.cc:485] T 37e9dcc6191c42d3a5c1503dedc5511e P c75831edb3b44dd59c00ef151adca49d -> Peer bc20559563fe44efa0a2b7b084aafbce (127.15.255.136:34965): Couldn't send request to peer bc20559563fe44efa0a2b7b084aafbce. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 37e9dcc6191c42d3a5c1503dedc5511e. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:03.826238 19005 raft_consensus.cc:1261] T 37e9dcc6191c42d3a5c1503dedc5511e P 9be399440ded4e2ebe6b6e0bf486cbb1 [term 1 FOLLOWER]: Refusing update from remote peer c75831edb3b44dd59c00ef151adca49d: Log matching property violated. Preceding OpId in replica: term: 1 index: 531. Preceding OpId from leader: term: 1 index: 532. (index mismatch)
I20240502 21:27:03.826283 22991 consensus_queue.cc:1035] T 37e9dcc6191c42d3a5c1503dedc5511e P c75831edb3b44dd59c00ef151adca49d [LEADER]: Connected to new peer: Peer: permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 532, Last known committed idx: 531, Time since last communication: 0.000s
I20240502 21:27:03.826550 18611 raft_consensus.cc:1261] T 37e9dcc6191c42d3a5c1503dedc5511e P 82f50f4b66f3450aa9f58a0b317f5cea [term 1 FOLLOWER]: Refusing update from remote peer c75831edb3b44dd59c00ef151adca49d: Log matching property violated. Preceding OpId in replica: term: 1 index: 531. Preceding OpId from leader: term: 1 index: 532. (index mismatch)
I20240502 21:27:03.826575 23047 consensus_queue.cc:1035] T 37e9dcc6191c42d3a5c1503dedc5511e P c75831edb3b44dd59c00ef151adca49d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 532, Last known committed idx: 531, Time since last communication: 0.000s
I20240502 21:27:03.827047 22990 consensus_queue.cc:1035] T 37e9dcc6191c42d3a5c1503dedc5511e P c75831edb3b44dd59c00ef151adca49d [LEADER]: Connected to new peer: Peer: permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 532, Last known committed idx: 531, Time since last communication: 0.000s
I20240502 21:27:03.827422 22798 consensus_queue.cc:1035] T 37e9dcc6191c42d3a5c1503dedc5511e P c75831edb3b44dd59c00ef151adca49d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 532, Last known committed idx: 531, Time since last communication: 0.000s
I20240502 21:27:03.830144 22798 raft_consensus.cc:2932] T 37e9dcc6191c42d3a5c1503dedc5511e P c75831edb3b44dd59c00ef151adca49d [term 1 LEADER]: Committing config change with OpId 1.532: config changed from index 137 to 532, NON_VOTER bc20559563fe44efa0a2b7b084aafbce (127.15.255.136) added. New config: { opid_index: 532 OBSOLETE_local: false peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } attrs { replace: true } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } attrs { promote: false } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: true } } }
I20240502 21:27:03.832929 18611 raft_consensus.cc:2932] T 37e9dcc6191c42d3a5c1503dedc5511e P 82f50f4b66f3450aa9f58a0b317f5cea [term 1 FOLLOWER]: Committing config change with OpId 1.532: config changed from index 137 to 532, NON_VOTER bc20559563fe44efa0a2b7b084aafbce (127.15.255.136) added. New config: { opid_index: 532 OBSOLETE_local: false peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } attrs { replace: true } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } attrs { promote: false } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: true } } }
I20240502 21:27:03.836297 19140 raft_consensus.cc:2932] T 37e9dcc6191c42d3a5c1503dedc5511e P 2877108756304e05bd60ba1186b870da [term 1 FOLLOWER]: Committing config change with OpId 1.532: config changed from index 137 to 532, NON_VOTER bc20559563fe44efa0a2b7b084aafbce (127.15.255.136) added. New config: { opid_index: 532 OBSOLETE_local: false peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } attrs { replace: true } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } attrs { promote: false } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: true } } }
I20240502 21:27:03.836752 18473 raft_consensus.cc:2932] T 37e9dcc6191c42d3a5c1503dedc5511e P faadabfb099b4abf90a466395be585b5 [term 1 FOLLOWER]: Committing config change with OpId 1.532: config changed from index 137 to 532, NON_VOTER bc20559563fe44efa0a2b7b084aafbce (127.15.255.136) added. New config: { opid_index: 532 OBSOLETE_local: false peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } attrs { replace: true } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } attrs { promote: false } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: true } } }
I20240502 21:27:03.838587 19009 raft_consensus.cc:2932] T 37e9dcc6191c42d3a5c1503dedc5511e P 9be399440ded4e2ebe6b6e0bf486cbb1 [term 1 FOLLOWER]: Committing config change with OpId 1.532: config changed from index 137 to 532, NON_VOTER bc20559563fe44efa0a2b7b084aafbce (127.15.255.136) added. New config: { opid_index: 532 OBSOLETE_local: false peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } attrs { replace: true } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } attrs { promote: false } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: true } } }
I20240502 21:27:03.841780 18329 catalog_manager.cc:5484] T 37e9dcc6191c42d3a5c1503dedc5511e P 2877108756304e05bd60ba1186b870da reported cstate change: config changed from index 137 to 532, NON_VOTER bc20559563fe44efa0a2b7b084aafbce (127.15.255.136) added. New cstate: current_term: 1 leader_uuid: "c75831edb3b44dd59c00ef151adca49d" committed_config { opid_index: 532 OBSOLETE_local: false peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } attrs { replace: true } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } attrs { promote: false } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: true } } }
I20240502 21:27:03.895507 18473 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ed78d40318fe4ab5bee4a01b182a34b6"
dest_uuid: "faadabfb099b4abf90a466395be585b5"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:58174
I20240502 21:27:03.895740 18473 raft_consensus.cc:595] T ed78d40318fe4ab5bee4a01b182a34b6 P faadabfb099b4abf90a466395be585b5 [term 1 LEADER]: Received request to transfer leadership
I20240502 21:27:03.971024 23080 raft_consensus.cc:1050] T 693a8666bbe54ae1b774f7f4da5ba67e P faadabfb099b4abf90a466395be585b5: attempting to promote NON_VOTER bc20559563fe44efa0a2b7b084aafbce to VOTER
I20240502 21:27:03.971925 23080 consensus_queue.cc:237] T 693a8666bbe54ae1b774f7f4da5ba67e P faadabfb099b4abf90a466395be585b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 584, Committed index: 584, Last appended: 1.584, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 585 OBSOLETE_local: false peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } attrs { replace: true } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: false } }
I20240502 21:27:03.974730 22631 raft_consensus.cc:1261] T 693a8666bbe54ae1b774f7f4da5ba67e P bc20559563fe44efa0a2b7b084aafbce [term 1 LEARNER]: Refusing update from remote peer faadabfb099b4abf90a466395be585b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 584. Preceding OpId from leader: term: 1 index: 585. (index mismatch)
I20240502 21:27:03.974880 20167 raft_consensus.cc:1261] T 693a8666bbe54ae1b774f7f4da5ba67e P 90e329b457c24caf93a9f880345eb288 [term 1 FOLLOWER]: Refusing update from remote peer faadabfb099b4abf90a466395be585b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 584. Preceding OpId from leader: term: 1 index: 585. (index mismatch)
I20240502 21:27:03.975098 19009 raft_consensus.cc:1261] T 693a8666bbe54ae1b774f7f4da5ba67e P 9be399440ded4e2ebe6b6e0bf486cbb1 [term 1 FOLLOWER]: Refusing update from remote peer faadabfb099b4abf90a466395be585b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 584. Preceding OpId from leader: term: 1 index: 585. (index mismatch)
I20240502 21:27:03.975248 22820 consensus_queue.cc:1035] T 693a8666bbe54ae1b774f7f4da5ba67e P faadabfb099b4abf90a466395be585b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 585, Last known committed idx: 584, Time since last communication: 0.000s
I20240502 21:27:03.975337 19140 raft_consensus.cc:1261] T 693a8666bbe54ae1b774f7f4da5ba67e P 2877108756304e05bd60ba1186b870da [term 1 FOLLOWER]: Refusing update from remote peer faadabfb099b4abf90a466395be585b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 584. Preceding OpId from leader: term: 1 index: 585. (index mismatch)
I20240502 21:27:03.975616 22833 consensus_queue.cc:1035] T 693a8666bbe54ae1b774f7f4da5ba67e P faadabfb099b4abf90a466395be585b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 585, Last known committed idx: 584, Time since last communication: 0.000s
I20240502 21:27:03.975826 18744 raft_consensus.cc:1261] T 693a8666bbe54ae1b774f7f4da5ba67e P 501e3c0d44fe4d70af9ff926d9622417 [term 1 FOLLOWER]: Refusing update from remote peer faadabfb099b4abf90a466395be585b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 584. Preceding OpId from leader: term: 1 index: 585. (index mismatch)
I20240502 21:27:03.975881 22992 consensus_queue.cc:1035] T 693a8666bbe54ae1b774f7f4da5ba67e P faadabfb099b4abf90a466395be585b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 585, Last known committed idx: 584, Time since last communication: 0.000s
I20240502 21:27:03.976174 22820 consensus_queue.cc:1035] T 693a8666bbe54ae1b774f7f4da5ba67e P faadabfb099b4abf90a466395be585b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 585, Last known committed idx: 584, Time since last communication: 0.000s
I20240502 21:27:03.977895 23080 consensus_queue.cc:1035] T 693a8666bbe54ae1b774f7f4da5ba67e P faadabfb099b4abf90a466395be585b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 585, Last known committed idx: 584, Time since last communication: 0.000s
I20240502 21:27:03.988879 22820 raft_consensus.cc:2932] T 693a8666bbe54ae1b774f7f4da5ba67e P faadabfb099b4abf90a466395be585b5 [term 1 LEADER]: Committing config change with OpId 1.585: config changed from index 584 to 585, bc20559563fe44efa0a2b7b084aafbce (127.15.255.136) changed from NON_VOTER to VOTER. New config: { opid_index: 585 OBSOLETE_local: false peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } attrs { replace: true } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: false } } }
I20240502 21:27:03.989591 19009 raft_consensus.cc:2932] T 693a8666bbe54ae1b774f7f4da5ba67e P 9be399440ded4e2ebe6b6e0bf486cbb1 [term 1 FOLLOWER]: Committing config change with OpId 1.585: config changed from index 584 to 585, bc20559563fe44efa0a2b7b084aafbce (127.15.255.136) changed from NON_VOTER to VOTER. New config: { opid_index: 585 OBSOLETE_local: false peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } attrs { replace: true } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: false } } }
I20240502 21:27:03.990036 19140 raft_consensus.cc:2932] T 693a8666bbe54ae1b774f7f4da5ba67e P 2877108756304e05bd60ba1186b870da [term 1 FOLLOWER]: Committing config change with OpId 1.585: config changed from index 584 to 585, bc20559563fe44efa0a2b7b084aafbce (127.15.255.136) changed from NON_VOTER to VOTER. New config: { opid_index: 585 OBSOLETE_local: false peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } attrs { replace: true } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: false } } }
I20240502 21:27:03.994036 18321 catalog_manager.cc:5484] T 693a8666bbe54ae1b774f7f4da5ba67e P 9be399440ded4e2ebe6b6e0bf486cbb1 reported cstate change: config changed from index 584 to 585, bc20559563fe44efa0a2b7b084aafbce (127.15.255.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "faadabfb099b4abf90a466395be585b5" committed_config { opid_index: 585 OBSOLETE_local: false peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } attrs { replace: true } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: false } } }
I20240502 21:27:03.995680 22631 raft_consensus.cc:2932] T 693a8666bbe54ae1b774f7f4da5ba67e P bc20559563fe44efa0a2b7b084aafbce [term 1 FOLLOWER]: Committing config change with OpId 1.585: config changed from index 584 to 585, bc20559563fe44efa0a2b7b084aafbce (127.15.255.136) changed from NON_VOTER to VOTER. New config: { opid_index: 585 OBSOLETE_local: false peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } attrs { replace: true } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: false } } }
I20240502 21:27:03.996699 18744 raft_consensus.cc:2932] T 693a8666bbe54ae1b774f7f4da5ba67e P 501e3c0d44fe4d70af9ff926d9622417 [term 1 FOLLOWER]: Committing config change with OpId 1.585: config changed from index 584 to 585, bc20559563fe44efa0a2b7b084aafbce (127.15.255.136) changed from NON_VOTER to VOTER. New config: { opid_index: 585 OBSOLETE_local: false peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } attrs { replace: true } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: false } } }
I20240502 21:27:03.997776 20167 raft_consensus.cc:2932] T 693a8666bbe54ae1b774f7f4da5ba67e P 90e329b457c24caf93a9f880345eb288 [term 1 FOLLOWER]: Committing config change with OpId 1.585: config changed from index 584 to 585, bc20559563fe44efa0a2b7b084aafbce (127.15.255.136) changed from NON_VOTER to VOTER. New config: { opid_index: 585 OBSOLETE_local: false peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } attrs { replace: true } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: false } } }
I20240502 21:27:04.006114 18473 consensus_queue.cc:237] T 693a8666bbe54ae1b774f7f4da5ba67e P faadabfb099b4abf90a466395be585b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 585, Committed index: 585, Last appended: 1.585, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 586 OBSOLETE_local: false peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: false } }
I20240502 21:27:04.009296 18744 raft_consensus.cc:1261] T 693a8666bbe54ae1b774f7f4da5ba67e P 501e3c0d44fe4d70af9ff926d9622417 [term 1 FOLLOWER]: Refusing update from remote peer faadabfb099b4abf90a466395be585b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 585. Preceding OpId from leader: term: 1 index: 586. (index mismatch)
I20240502 21:27:04.009773 22833 consensus_queue.cc:1035] T 693a8666bbe54ae1b774f7f4da5ba67e P faadabfb099b4abf90a466395be585b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 586, Last known committed idx: 585, Time since last communication: 0.000s
I20240502 21:27:04.010067 20167 raft_consensus.cc:1261] T 693a8666bbe54ae1b774f7f4da5ba67e P 90e329b457c24caf93a9f880345eb288 [term 1 FOLLOWER]: Refusing update from remote peer faadabfb099b4abf90a466395be585b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 585. Preceding OpId from leader: term: 1 index: 586. (index mismatch)
I20240502 21:27:04.010602 19009 raft_consensus.cc:1261] T 693a8666bbe54ae1b774f7f4da5ba67e P 9be399440ded4e2ebe6b6e0bf486cbb1 [term 1 FOLLOWER]: Refusing update from remote peer faadabfb099b4abf90a466395be585b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 585. Preceding OpId from leader: term: 1 index: 586. (index mismatch)
I20240502 21:27:04.011101 22820 consensus_queue.cc:1035] T 693a8666bbe54ae1b774f7f4da5ba67e P faadabfb099b4abf90a466395be585b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 586, Last known committed idx: 585, Time since last communication: 0.000s
I20240502 21:27:04.011775 22833 consensus_queue.cc:1035] T 693a8666bbe54ae1b774f7f4da5ba67e P faadabfb099b4abf90a466395be585b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 586, Last known committed idx: 585, Time since last communication: 0.000s
I20240502 21:27:04.012216 22631 raft_consensus.cc:1261] T 693a8666bbe54ae1b774f7f4da5ba67e P bc20559563fe44efa0a2b7b084aafbce [term 1 FOLLOWER]: Refusing update from remote peer faadabfb099b4abf90a466395be585b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 585. Preceding OpId from leader: term: 1 index: 586. (index mismatch)
I20240502 21:27:04.013469 22833 consensus_queue.cc:1035] T 693a8666bbe54ae1b774f7f4da5ba67e P faadabfb099b4abf90a466395be585b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 586, Last known committed idx: 585, Time since last communication: 0.000s
I20240502 21:27:04.014734 23080 raft_consensus.cc:2932] T 693a8666bbe54ae1b774f7f4da5ba67e P faadabfb099b4abf90a466395be585b5 [term 1 LEADER]: Committing config change with OpId 1.586: config changed from index 585 to 586, VOTER 2877108756304e05bd60ba1186b870da (127.15.255.134) evicted. New config: { opid_index: 586 OBSOLETE_local: false peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: false } } }
I20240502 21:27:04.015388 20167 raft_consensus.cc:2932] T 693a8666bbe54ae1b774f7f4da5ba67e P 90e329b457c24caf93a9f880345eb288 [term 1 FOLLOWER]: Committing config change with OpId 1.586: config changed from index 585 to 586, VOTER 2877108756304e05bd60ba1186b870da (127.15.255.134) evicted. New config: { opid_index: 586 OBSOLETE_local: false peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: false } } }
I20240502 21:27:04.017522 18739 raft_consensus.cc:2932] T 693a8666bbe54ae1b774f7f4da5ba67e P 501e3c0d44fe4d70af9ff926d9622417 [term 1 FOLLOWER]: Committing config change with OpId 1.586: config changed from index 585 to 586, VOTER 2877108756304e05bd60ba1186b870da (127.15.255.134) evicted. New config: { opid_index: 586 OBSOLETE_local: false peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: false } } }
I20240502 21:27:04.018868 18313 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 693a8666bbe54ae1b774f7f4da5ba67e with cas_config_opid_index 585: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:27:04.020926 18327 catalog_manager.cc:5484] T 693a8666bbe54ae1b774f7f4da5ba67e P 90e329b457c24caf93a9f880345eb288 reported cstate change: config changed from index 585 to 586, VOTER 2877108756304e05bd60ba1186b870da (127.15.255.134) evicted. New cstate: current_term: 1 leader_uuid: "faadabfb099b4abf90a466395be585b5" committed_config { opid_index: 586 OBSOLETE_local: false peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: false } } }
I20240502 21:27:04.020783 19005 raft_consensus.cc:2932] T 693a8666bbe54ae1b774f7f4da5ba67e P 9be399440ded4e2ebe6b6e0bf486cbb1 [term 1 FOLLOWER]: Committing config change with OpId 1.586: config changed from index 585 to 586, VOTER 2877108756304e05bd60ba1186b870da (127.15.255.134) evicted. New config: { opid_index: 586 OBSOLETE_local: false peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: false } } }
I20240502 21:27:04.020928 22632 raft_consensus.cc:2932] T 693a8666bbe54ae1b774f7f4da5ba67e P bc20559563fe44efa0a2b7b084aafbce [term 1 FOLLOWER]: Committing config change with OpId 1.586: config changed from index 585 to 586, VOTER 2877108756304e05bd60ba1186b870da (127.15.255.134) evicted. New config: { opid_index: 586 OBSOLETE_local: false peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: false } } }
I20240502 21:27:04.029493 19124 tablet_service.cc:1514] Processing DeleteTablet for tablet 693a8666bbe54ae1b774f7f4da5ba67e with delete_type TABLET_DATA_TOMBSTONED (TS 2877108756304e05bd60ba1186b870da not found in new config with opid_index 586) from {username='slave'} at 127.0.0.1:57946
I20240502 21:27:04.030081 23155 tablet_replica.cc:330] T 693a8666bbe54ae1b774f7f4da5ba67e P 2877108756304e05bd60ba1186b870da: stopping tablet replica
I20240502 21:27:04.030261 23155 raft_consensus.cc:2229] T 693a8666bbe54ae1b774f7f4da5ba67e P 2877108756304e05bd60ba1186b870da [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:04.031733 23155 raft_consensus.cc:2258] T 693a8666bbe54ae1b774f7f4da5ba67e P 2877108756304e05bd60ba1186b870da [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:04.033300 23155 ts_tablet_manager.cc:1875] T 693a8666bbe54ae1b774f7f4da5ba67e P 2877108756304e05bd60ba1186b870da: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:27:04.037616 23155 ts_tablet_manager.cc:1888] T 693a8666bbe54ae1b774f7f4da5ba67e P 2877108756304e05bd60ba1186b870da: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.585
I20240502 21:27:04.037725 23155 log.cc:1196] T 693a8666bbe54ae1b774f7f4da5ba67e P 2877108756304e05bd60ba1186b870da: Deleting WAL directory at /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-5/wal/wals/693a8666bbe54ae1b774f7f4da5ba67e
I20240502 21:27:04.038378 18315 catalog_manager.cc:4830] TS 2877108756304e05bd60ba1186b870da (127.15.255.134:43635): tablet 693a8666bbe54ae1b774f7f4da5ba67e (table rebalance_test_table_2 [id=59b7e3f7d6ce4a008110576d81d3cf39]) successfully deleted
I20240502 21:27:04.154876 18473 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ed78d40318fe4ab5bee4a01b182a34b6"
dest_uuid: "faadabfb099b4abf90a466395be585b5"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:58198
I20240502 21:27:04.155063 18473 raft_consensus.cc:595] T ed78d40318fe4ab5bee4a01b182a34b6 P faadabfb099b4abf90a466395be585b5 [term 1 LEADER]: Received request to transfer leadership
I20240502 21:27:04.174301 23193 ts_tablet_manager.cc:902] T 40d4f61d0f6d4d31a5626c7743c88133 P bc20559563fe44efa0a2b7b084aafbce: Initiating tablet copy from peer faadabfb099b4abf90a466395be585b5 (127.15.255.129:32983)
I20240502 21:27:04.175961 18611 consensus_queue.cc:237] T 9844e983dd0a403abe0ac90cd96690d6 P 82f50f4b66f3450aa9f58a0b317f5cea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 405, Committed index: 405, Last appended: 1.405, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 406 OBSOLETE_local: false peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } attrs { replace: true } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: true } }
I20240502 21:27:04.178855 19005 raft_consensus.cc:1261] T 9844e983dd0a403abe0ac90cd96690d6 P 9be399440ded4e2ebe6b6e0bf486cbb1 [term 1 FOLLOWER]: Refusing update from remote peer 82f50f4b66f3450aa9f58a0b317f5cea: Log matching property violated. Preceding OpId in replica: term: 1 index: 405. Preceding OpId from leader: term: 1 index: 406. (index mismatch)
I20240502 21:27:04.178890 18873 raft_consensus.cc:1261] T 9844e983dd0a403abe0ac90cd96690d6 P c75831edb3b44dd59c00ef151adca49d [term 1 FOLLOWER]: Refusing update from remote peer 82f50f4b66f3450aa9f58a0b317f5cea: Log matching property violated. Preceding OpId in replica: term: 1 index: 405. Preceding OpId from leader: term: 1 index: 406. (index mismatch)
I20240502 21:27:04.179514 22925 consensus_queue.cc:1035] T 9844e983dd0a403abe0ac90cd96690d6 P 82f50f4b66f3450aa9f58a0b317f5cea [LEADER]: Connected to new peer: Peer: permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 406, Last known committed idx: 405, Time since last communication: 0.000s
I20240502 21:27:04.180124 22925 consensus_queue.cc:1035] T 9844e983dd0a403abe0ac90cd96690d6 P 82f50f4b66f3450aa9f58a0b317f5cea [LEADER]: Connected to new peer: Peer: permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 406, Last known committed idx: 405, Time since last communication: 0.000s
I20240502 21:27:04.186785 22992 raft_consensus.cc:979] T ed78d40318fe4ab5bee4a01b182a34b6 P faadabfb099b4abf90a466395be585b5: : Instructing follower bc20559563fe44efa0a2b7b084aafbce to start an election
I20240502 21:27:04.186959 22992 raft_consensus.cc:1067] T ed78d40318fe4ab5bee4a01b182a34b6 P faadabfb099b4abf90a466395be585b5 [term 1 LEADER]: Signalling peer bc20559563fe44efa0a2b7b084aafbce to start an election
I20240502 21:27:04.187974 22632 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ed78d40318fe4ab5bee4a01b182a34b6"
dest_uuid: "bc20559563fe44efa0a2b7b084aafbce"
from {username='slave'} at 127.15.255.129:50721
I20240502 21:27:04.188308 22632 raft_consensus.cc:482] T ed78d40318fe4ab5bee4a01b182a34b6 P bc20559563fe44efa0a2b7b084aafbce [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:04.188521 22632 raft_consensus.cc:3037] T ed78d40318fe4ab5bee4a01b182a34b6 P bc20559563fe44efa0a2b7b084aafbce [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:04.192199 22632 raft_consensus.cc:504] T ed78d40318fe4ab5bee4a01b182a34b6 P bc20559563fe44efa0a2b7b084aafbce [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 499 OBSOLETE_local: false peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } attrs { replace: true } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: false } }
I20240502 21:27:04.194904 18876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed78d40318fe4ab5bee4a01b182a34b6" candidate_uuid: "bc20559563fe44efa0a2b7b084aafbce" candidate_term: 2 candidate_status { last_received { term: 1 index: 499 } } ignore_live_leader: true dest_uuid: "c75831edb3b44dd59c00ef151adca49d"
I20240502 21:27:04.195150 18876 raft_consensus.cc:3037] T ed78d40318fe4ab5bee4a01b182a34b6 P c75831edb3b44dd59c00ef151adca49d [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:04.197968 18876 raft_consensus.cc:2454] T ed78d40318fe4ab5bee4a01b182a34b6 P c75831edb3b44dd59c00ef151adca49d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bc20559563fe44efa0a2b7b084aafbce in term 2.
I20240502 21:27:04.199061 23202 ts_tablet_manager.cc:902] T b0852f54089944ce935f8ee4cbf2bc5a P bc20559563fe44efa0a2b7b084aafbce: Initiating tablet copy from peer 2877108756304e05bd60ba1186b870da (127.15.255.134:43635)
I20240502 21:27:04.199699 18739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed78d40318fe4ab5bee4a01b182a34b6" candidate_uuid: "bc20559563fe44efa0a2b7b084aafbce" candidate_term: 2 candidate_status { last_received { term: 1 index: 499 } } ignore_live_leader: true dest_uuid: "501e3c0d44fe4d70af9ff926d9622417"
I20240502 21:27:04.199946 18739 raft_consensus.cc:3037] T ed78d40318fe4ab5bee4a01b182a34b6 P 501e3c0d44fe4d70af9ff926d9622417 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:04.199939 22925 raft_consensus.cc:2932] T 9844e983dd0a403abe0ac90cd96690d6 P 82f50f4b66f3450aa9f58a0b317f5cea [term 1 LEADER]: Committing config change with OpId 1.406: config changed from index -1 to 406, NON_VOTER bc20559563fe44efa0a2b7b084aafbce (127.15.255.136) added. New config: { opid_index: 406 OBSOLETE_local: false peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } attrs { replace: true } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: true } } }
I20240502 21:27:04.202737 18739 raft_consensus.cc:2454] T ed78d40318fe4ab5bee4a01b182a34b6 P 501e3c0d44fe4d70af9ff926d9622417 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bc20559563fe44efa0a2b7b084aafbce in term 2.
I20240502 21:27:04.204955 19005 raft_consensus.cc:2932] T 9844e983dd0a403abe0ac90cd96690d6 P 9be399440ded4e2ebe6b6e0bf486cbb1 [term 1 FOLLOWER]: Committing config change with OpId 1.406: config changed from index -1 to 406, NON_VOTER bc20559563fe44efa0a2b7b084aafbce (127.15.255.136) added. New config: { opid_index: 406 OBSOLETE_local: false peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } attrs { replace: true } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: true } } }
I20240502 21:27:04.211094 18876 raft_consensus.cc:1226] T ed78d40318fe4ab5bee4a01b182a34b6 P c75831edb3b44dd59c00ef151adca49d [term 2 FOLLOWER]: Rejecting Update request from peer faadabfb099b4abf90a466395be585b5 for earlier term 1. Current term is 2. Ops: []
I20240502 21:27:04.211303 18319 catalog_manager.cc:5484] T 9844e983dd0a403abe0ac90cd96690d6 P 9be399440ded4e2ebe6b6e0bf486cbb1 reported cstate change: config changed from index -1 to 406, NON_VOTER bc20559563fe44efa0a2b7b084aafbce (127.15.255.136) added. New cstate: current_term: 1 leader_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" committed_config { opid_index: 406 OBSOLETE_local: false peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } attrs { replace: true } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: true } } }
I20240502 21:27:04.211687 22820 consensus_queue.cc:1046] T ed78d40318fe4ab5bee4a01b182a34b6 P faadabfb099b4abf90a466395be585b5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 }, Status: INVALID_TERM, Last received: 1.499, Next index: 500, Last known committed idx: 499, Time since last communication: 0.000s
I20240502 21:27:04.212081 22820 raft_consensus.cc:3032] T ed78d40318fe4ab5bee4a01b182a34b6 P faadabfb099b4abf90a466395be585b5 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 21:27:04.212180 22820 raft_consensus.cc:729] T ed78d40318fe4ab5bee4a01b182a34b6 P faadabfb099b4abf90a466395be585b5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: faadabfb099b4abf90a466395be585b5, State: Running, Role: LEADER
I20240502 21:27:04.212570 22820 consensus_queue.cc:260] T ed78d40318fe4ab5bee4a01b182a34b6 P faadabfb099b4abf90a466395be585b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 499, Committed index: 499, Last appended: 1.499, Last appended by leader: 499, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 499 OBSOLETE_local: false peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } attrs { replace: true } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: false } }
I20240502 21:27:04.213176 22820 raft_consensus.cc:3037] T ed78d40318fe4ab5bee4a01b182a34b6 P faadabfb099b4abf90a466395be585b5 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:04.234593 22632 leader_election.cc:288] T ed78d40318fe4ab5bee4a01b182a34b6 P bc20559563fe44efa0a2b7b084aafbce [CANDIDATE]: Term 2 election: Requested vote from peers c75831edb3b44dd59c00ef151adca49d (127.15.255.132:43607), 501e3c0d44fe4d70af9ff926d9622417 (127.15.255.131:42501), 2877108756304e05bd60ba1186b870da (127.15.255.134:43635), 9be399440ded4e2ebe6b6e0bf486cbb1 (127.15.255.133:34217), faadabfb099b4abf90a466395be585b5 (127.15.255.129:32983)
I20240502 21:27:04.260950 23214 ts_tablet_manager.cc:902] T 37e9dcc6191c42d3a5c1503dedc5511e P bc20559563fe44efa0a2b7b084aafbce: Initiating tablet copy from peer c75831edb3b44dd59c00ef151adca49d (127.15.255.132:43607)
I20240502 21:27:04.262501 18473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed78d40318fe4ab5bee4a01b182a34b6" candidate_uuid: "bc20559563fe44efa0a2b7b084aafbce" candidate_term: 2 candidate_status { last_received { term: 1 index: 499 } } ignore_live_leader: true dest_uuid: "faadabfb099b4abf90a466395be585b5"
W20240502 21:27:04.274219 18544 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.133:34217, user_credentials={real_user=slave}} blocked reactor thread for 64415.5us
I20240502 21:27:04.276157 19140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed78d40318fe4ab5bee4a01b182a34b6" candidate_uuid: "bc20559563fe44efa0a2b7b084aafbce" candidate_term: 2 candidate_status { last_received { term: 1 index: 499 } } ignore_live_leader: true dest_uuid: "2877108756304e05bd60ba1186b870da"
I20240502 21:27:04.276446 19140 raft_consensus.cc:3037] T ed78d40318fe4ab5bee4a01b182a34b6 P 2877108756304e05bd60ba1186b870da [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:04.279485 19140 raft_consensus.cc:2454] T ed78d40318fe4ab5bee4a01b182a34b6 P 2877108756304e05bd60ba1186b870da [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bc20559563fe44efa0a2b7b084aafbce in term 2.
I20240502 21:27:04.279853 19005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed78d40318fe4ab5bee4a01b182a34b6" candidate_uuid: "bc20559563fe44efa0a2b7b084aafbce" candidate_term: 2 candidate_status { last_received { term: 1 index: 499 } } ignore_live_leader: true dest_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1"
I20240502 21:27:04.280073 19005 raft_consensus.cc:3037] T ed78d40318fe4ab5bee4a01b182a34b6 P 9be399440ded4e2ebe6b6e0bf486cbb1 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:04.280359 22569 leader_election.cc:302] T ed78d40318fe4ab5bee4a01b182a34b6 P bc20559563fe44efa0a2b7b084aafbce [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 6 voters: 4 yes votes; 0 no votes. yes voters: 2877108756304e05bd60ba1186b870da, 501e3c0d44fe4d70af9ff926d9622417, bc20559563fe44efa0a2b7b084aafbce, c75831edb3b44dd59c00ef151adca49d; no voters:
I20240502 21:27:04.280983 22948 raft_consensus.cc:2781] T ed78d40318fe4ab5bee4a01b182a34b6 P bc20559563fe44efa0a2b7b084aafbce [term 2 FOLLOWER]: Leader election won for term 2
I20240502 21:27:04.282759 19005 raft_consensus.cc:2454] T ed78d40318fe4ab5bee4a01b182a34b6 P 9be399440ded4e2ebe6b6e0bf486cbb1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bc20559563fe44efa0a2b7b084aafbce in term 2.
I20240502 21:27:04.284555 18739 raft_consensus.cc:1261] T 9844e983dd0a403abe0ac90cd96690d6 P 501e3c0d44fe4d70af9ff926d9622417 [term 1 FOLLOWER]: Refusing update from remote peer 82f50f4b66f3450aa9f58a0b317f5cea: Log matching property violated. Preceding OpId in replica: term: 1 index: 405. Preceding OpId from leader: term: 1 index: 406. (index mismatch)
W20240502 21:27:04.284708 18543 consensus_peers.cc:485] T 9844e983dd0a403abe0ac90cd96690d6 P 82f50f4b66f3450aa9f58a0b317f5cea -> Peer bc20559563fe44efa0a2b7b084aafbce (127.15.255.136:34965): Couldn't send request to peer bc20559563fe44efa0a2b7b084aafbce. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9844e983dd0a403abe0ac90cd96690d6. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:04.285162 23209 consensus_queue.cc:1035] T 9844e983dd0a403abe0ac90cd96690d6 P 82f50f4b66f3450aa9f58a0b317f5cea [LEADER]: Connected to new peer: Peer: permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 406, Last known committed idx: 405, Time since last communication: 0.000s
I20240502 21:27:04.285363 18472 raft_consensus.cc:1261] T 9844e983dd0a403abe0ac90cd96690d6 P faadabfb099b4abf90a466395be585b5 [term 1 FOLLOWER]: Refusing update from remote peer 82f50f4b66f3450aa9f58a0b317f5cea: Log matching property violated. Preceding OpId in replica: term: 1 index: 405. Preceding OpId from leader: term: 1 index: 406. (index mismatch)
I20240502 21:27:04.287997 23217 consensus_queue.cc:1035] T 9844e983dd0a403abe0ac90cd96690d6 P 82f50f4b66f3450aa9f58a0b317f5cea [LEADER]: Connected to new peer: Peer: permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 406, Last known committed idx: 405, Time since last communication: 0.000s
I20240502 21:27:04.289033 18876 raft_consensus.cc:2932] T 9844e983dd0a403abe0ac90cd96690d6 P c75831edb3b44dd59c00ef151adca49d [term 1 FOLLOWER]: Committing config change with OpId 1.406: config changed from index -1 to 406, NON_VOTER bc20559563fe44efa0a2b7b084aafbce (127.15.255.136) added. New config: { opid_index: 406 OBSOLETE_local: false peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } attrs { replace: true } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: true } } }
I20240502 21:27:04.290150 23193 tablet_copy_client.cc:308] T 40d4f61d0f6d4d31a5626c7743c88133 P bc20559563fe44efa0a2b7b084aafbce: tablet copy: Beginning tablet copy session from remote peer at address 127.15.255.129:32983
I20240502 21:27:04.290820 18496 tablet_copy_service.cc:140] P faadabfb099b4abf90a466395be585b5: Received BeginTabletCopySession request for tablet 40d4f61d0f6d4d31a5626c7743c88133 from peer bc20559563fe44efa0a2b7b084aafbce ({username='slave'} at 127.15.255.136:50439)
I20240502 21:27:04.290983 18496 tablet_copy_service.cc:161] P faadabfb099b4abf90a466395be585b5: Beginning new tablet copy session on tablet 40d4f61d0f6d4d31a5626c7743c88133 from peer bc20559563fe44efa0a2b7b084aafbce at {username='slave'} at 127.15.255.136:50439: session id = bc20559563fe44efa0a2b7b084aafbce-40d4f61d0f6d4d31a5626c7743c88133
I20240502 21:27:04.291934 18473 raft_consensus.cc:2454] T ed78d40318fe4ab5bee4a01b182a34b6 P faadabfb099b4abf90a466395be585b5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bc20559563fe44efa0a2b7b084aafbce in term 2.
I20240502 21:27:04.293649 18496 tablet_copy_source_session.cc:205] T 40d4f61d0f6d4d31a5626c7743c88133 P faadabfb099b4abf90a466395be585b5: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:27:04.295128 23193 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 40d4f61d0f6d4d31a5626c7743c88133. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:04.300846 22948 raft_consensus.cc:686] T ed78d40318fe4ab5bee4a01b182a34b6 P bc20559563fe44efa0a2b7b084aafbce [term 2 LEADER]: Becoming Leader. State: Replica: bc20559563fe44efa0a2b7b084aafbce, State: Running, Role: LEADER
I20240502 21:27:04.300550 18744 raft_consensus.cc:2932] T 9844e983dd0a403abe0ac90cd96690d6 P 501e3c0d44fe4d70af9ff926d9622417 [term 1 FOLLOWER]: Committing config change with OpId 1.406: config changed from index -1 to 406, NON_VOTER bc20559563fe44efa0a2b7b084aafbce (127.15.255.136) added. New config: { opid_index: 406 OBSOLETE_local: false peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } attrs { replace: true } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: true } } }
I20240502 21:27:04.302616 22948 consensus_queue.cc:237] T ed78d40318fe4ab5bee4a01b182a34b6 P bc20559563fe44efa0a2b7b084aafbce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 499, Committed index: 499, Last appended: 1.499, Last appended by leader: 499, Current term: 2, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 499 OBSOLETE_local: false peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } attrs { replace: true } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: false } }
I20240502 21:27:04.304471 23193 tablet_copy_client.cc:783] T 40d4f61d0f6d4d31a5626c7743c88133 P bc20559563fe44efa0a2b7b084aafbce: tablet copy: Starting download of 0 data blocks...
I20240502 21:27:04.305145 23193 tablet_copy_client.cc:647] T 40d4f61d0f6d4d31a5626c7743c88133 P bc20559563fe44efa0a2b7b084aafbce: tablet copy: Starting download of 1 WAL segments...
I20240502 21:27:04.307121 23193 tablet_copy_client.cc:523] T 40d4f61d0f6d4d31a5626c7743c88133 P bc20559563fe44efa0a2b7b084aafbce: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:27:04.309329 23214 tablet_copy_client.cc:308] T 37e9dcc6191c42d3a5c1503dedc5511e P bc20559563fe44efa0a2b7b084aafbce: tablet copy: Beginning tablet copy session from remote peer at address 127.15.255.132:43607
I20240502 21:27:04.310096 18898 tablet_copy_service.cc:140] P c75831edb3b44dd59c00ef151adca49d: Received BeginTabletCopySession request for tablet 37e9dcc6191c42d3a5c1503dedc5511e from peer bc20559563fe44efa0a2b7b084aafbce ({username='slave'} at 127.15.255.136:39477)
I20240502 21:27:04.310272 18898 tablet_copy_service.cc:161] P c75831edb3b44dd59c00ef151adca49d: Beginning new tablet copy session on tablet 37e9dcc6191c42d3a5c1503dedc5511e from peer bc20559563fe44efa0a2b7b084aafbce at {username='slave'} at 127.15.255.136:39477: session id = bc20559563fe44efa0a2b7b084aafbce-37e9dcc6191c42d3a5c1503dedc5511e
I20240502 21:27:04.313068 18898 tablet_copy_source_session.cc:205] T 37e9dcc6191c42d3a5c1503dedc5511e P c75831edb3b44dd59c00ef151adca49d: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:27:04.319725 23214 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37e9dcc6191c42d3a5c1503dedc5511e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:04.319990 23202 tablet_copy_client.cc:308] T b0852f54089944ce935f8ee4cbf2bc5a P bc20559563fe44efa0a2b7b084aafbce: tablet copy: Beginning tablet copy session from remote peer at address 127.15.255.134:43635
I20240502 21:27:04.320688 19166 tablet_copy_service.cc:140] P 2877108756304e05bd60ba1186b870da: Received BeginTabletCopySession request for tablet b0852f54089944ce935f8ee4cbf2bc5a from peer bc20559563fe44efa0a2b7b084aafbce ({username='slave'} at 127.15.255.136:57045)
I20240502 21:27:04.320852 19166 tablet_copy_service.cc:161] P 2877108756304e05bd60ba1186b870da: Beginning new tablet copy session on tablet b0852f54089944ce935f8ee4cbf2bc5a from peer bc20559563fe44efa0a2b7b084aafbce at {username='slave'} at 127.15.255.136:57045: session id = bc20559563fe44efa0a2b7b084aafbce-b0852f54089944ce935f8ee4cbf2bc5a
I20240502 21:27:04.320997 23193 tablet_bootstrap.cc:492] T 40d4f61d0f6d4d31a5626c7743c88133 P bc20559563fe44efa0a2b7b084aafbce: Bootstrap starting.
I20240502 21:27:04.320575 18327 catalog_manager.cc:5484] T ed78d40318fe4ab5bee4a01b182a34b6 P bc20559563fe44efa0a2b7b084aafbce reported cstate change: term changed from 1 to 2, leader changed from faadabfb099b4abf90a466395be585b5 (127.15.255.129) to bc20559563fe44efa0a2b7b084aafbce (127.15.255.136). New cstate: current_term: 2 leader_uuid: "bc20559563fe44efa0a2b7b084aafbce" committed_config { opid_index: 499 OBSOLETE_local: false peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:27:04.323926 19166 tablet_copy_source_session.cc:205] T b0852f54089944ce935f8ee4cbf2bc5a P 2877108756304e05bd60ba1186b870da: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:27:04.325685 23202 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0852f54089944ce935f8ee4cbf2bc5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:04.332772 23202 tablet_copy_client.cc:783] T b0852f54089944ce935f8ee4cbf2bc5a P bc20559563fe44efa0a2b7b084aafbce: tablet copy: Starting download of 0 data blocks...
I20240502 21:27:04.333041 23202 tablet_copy_client.cc:647] T b0852f54089944ce935f8ee4cbf2bc5a P bc20559563fe44efa0a2b7b084aafbce: tablet copy: Starting download of 1 WAL segments...
I20240502 21:27:04.334373 23193 tablet_bootstrap.cc:492] T 40d4f61d0f6d4d31a5626c7743c88133 P bc20559563fe44efa0a2b7b084aafbce: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:27:04.334915 23193 tablet_bootstrap.cc:492] T 40d4f61d0f6d4d31a5626c7743c88133 P bc20559563fe44efa0a2b7b084aafbce: Bootstrap complete.
I20240502 21:27:04.335258 23193 ts_tablet_manager.cc:1367] T 40d4f61d0f6d4d31a5626c7743c88133 P bc20559563fe44efa0a2b7b084aafbce: Time spent bootstrapping tablet: real 0.014s user 0.012s sys 0.001s
I20240502 21:27:04.336634 23202 tablet_copy_client.cc:523] T b0852f54089944ce935f8ee4cbf2bc5a P bc20559563fe44efa0a2b7b084aafbce: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:27:04.338618 23193 raft_consensus.cc:348] T 40d4f61d0f6d4d31a5626c7743c88133 P bc20559563fe44efa0a2b7b084aafbce [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } attrs { replace: true } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: true } }
I20240502 21:27:04.338900 23193 raft_consensus.cc:729] T 40d4f61d0f6d4d31a5626c7743c88133 P bc20559563fe44efa0a2b7b084aafbce [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: bc20559563fe44efa0a2b7b084aafbce, State: Initialized, Role: LEARNER
I20240502 21:27:04.339133 23193 consensus_queue.cc:260] T 40d4f61d0f6d4d31a5626c7743c88133 P bc20559563fe44efa0a2b7b084aafbce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } attrs { replace: true } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: true } }
I20240502 21:27:04.341282 18473 raft_consensus.cc:2932] T 9844e983dd0a403abe0ac90cd96690d6 P faadabfb099b4abf90a466395be585b5 [term 1 FOLLOWER]: Committing config change with OpId 1.406: config changed from index -1 to 406, NON_VOTER bc20559563fe44efa0a2b7b084aafbce (127.15.255.136) added. New config: { opid_index: 406 OBSOLETE_local: false peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } attrs { replace: true } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: true } } }
I20240502 21:27:04.340502 23214 tablet_copy_client.cc:783] T 37e9dcc6191c42d3a5c1503dedc5511e P bc20559563fe44efa0a2b7b084aafbce: tablet copy: Starting download of 0 data blocks...
I20240502 21:27:04.341962 23193 ts_tablet_manager.cc:1398] T 40d4f61d0f6d4d31a5626c7743c88133 P bc20559563fe44efa0a2b7b084aafbce: Time spent starting tablet: real 0.007s user 0.001s sys 0.000s
I20240502 21:27:04.342010 23214 tablet_copy_client.cc:647] T 37e9dcc6191c42d3a5c1503dedc5511e P bc20559563fe44efa0a2b7b084aafbce: tablet copy: Starting download of 1 WAL segments...
I20240502 21:27:04.343075 18496 tablet_copy_service.cc:342] P faadabfb099b4abf90a466395be585b5: Request end of tablet copy session bc20559563fe44efa0a2b7b084aafbce-40d4f61d0f6d4d31a5626c7743c88133 received from {username='slave'} at 127.15.255.136:50439
I20240502 21:27:04.343196 18496 tablet_copy_service.cc:433] P faadabfb099b4abf90a466395be585b5: ending tablet copy session bc20559563fe44efa0a2b7b084aafbce-40d4f61d0f6d4d31a5626c7743c88133 on tablet 40d4f61d0f6d4d31a5626c7743c88133 with peer bc20559563fe44efa0a2b7b084aafbce
I20240502 21:27:04.345129 23202 tablet_bootstrap.cc:492] T b0852f54089944ce935f8ee4cbf2bc5a P bc20559563fe44efa0a2b7b084aafbce: Bootstrap starting.
I20240502 21:27:04.346352 23214 tablet_copy_client.cc:523] T 37e9dcc6191c42d3a5c1503dedc5511e P bc20559563fe44efa0a2b7b084aafbce: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:27:04.350317 23214 tablet_bootstrap.cc:492] T 37e9dcc6191c42d3a5c1503dedc5511e P bc20559563fe44efa0a2b7b084aafbce: Bootstrap starting.
I20240502 21:27:04.566001 22631 raft_consensus.cc:1203] T 40d4f61d0f6d4d31a5626c7743c88133 P bc20559563fe44efa0a2b7b084aafbce [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 21:27:04.698760 23193 ts_tablet_manager.cc:902] T 9844e983dd0a403abe0ac90cd96690d6 P bc20559563fe44efa0a2b7b084aafbce: Initiating tablet copy from peer 82f50f4b66f3450aa9f58a0b317f5cea (127.15.255.130:45139)
I20240502 21:27:04.724797 18744 raft_consensus.cc:1261] T ed78d40318fe4ab5bee4a01b182a34b6 P 501e3c0d44fe4d70af9ff926d9622417 [term 2 FOLLOWER]: Refusing update from remote peer bc20559563fe44efa0a2b7b084aafbce: Log matching property violated. Preceding OpId in replica: term: 1 index: 499. Preceding OpId from leader: term: 2 index: 500. (index mismatch)
I20240502 21:27:04.728942 22948 consensus_queue.cc:1035] T ed78d40318fe4ab5bee4a01b182a34b6 P bc20559563fe44efa0a2b7b084aafbce [LEADER]: Connected to new peer: Peer: permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 500, Last known committed idx: 499, Time since last communication: 0.000s
I20240502 21:27:04.735769 18876 raft_consensus.cc:1261] T ed78d40318fe4ab5bee4a01b182a34b6 P c75831edb3b44dd59c00ef151adca49d [term 2 FOLLOWER]: Refusing update from remote peer bc20559563fe44efa0a2b7b084aafbce: Log matching property violated. Preceding OpId in replica: term: 1 index: 499. Preceding OpId from leader: term: 2 index: 500. (index mismatch)
I20240502 21:27:04.736788 22948 consensus_queue.cc:1035] T ed78d40318fe4ab5bee4a01b182a34b6 P bc20559563fe44efa0a2b7b084aafbce [LEADER]: Connected to new peer: Peer: permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 500, Last known committed idx: 499, Time since last communication: 0.000s
I20240502 21:27:04.756650 23193 tablet_copy_client.cc:308] T 9844e983dd0a403abe0ac90cd96690d6 P bc20559563fe44efa0a2b7b084aafbce: tablet copy: Beginning tablet copy session from remote peer at address 127.15.255.130:45139
I20240502 21:27:04.757416 18632 tablet_copy_service.cc:140] P 82f50f4b66f3450aa9f58a0b317f5cea: Received BeginTabletCopySession request for tablet 9844e983dd0a403abe0ac90cd96690d6 from peer bc20559563fe44efa0a2b7b084aafbce ({username='slave'} at 127.15.255.136:48763)
I20240502 21:27:04.757630 18632 tablet_copy_service.cc:161] P 82f50f4b66f3450aa9f58a0b317f5cea: Beginning new tablet copy session on tablet 9844e983dd0a403abe0ac90cd96690d6 from peer bc20559563fe44efa0a2b7b084aafbce at {username='slave'} at 127.15.255.136:48763: session id = bc20559563fe44efa0a2b7b084aafbce-9844e983dd0a403abe0ac90cd96690d6
I20240502 21:27:04.760758 18632 tablet_copy_source_session.cc:205] T 9844e983dd0a403abe0ac90cd96690d6 P 82f50f4b66f3450aa9f58a0b317f5cea: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:27:04.762229 23193 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9844e983dd0a403abe0ac90cd96690d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:04.770756 23193 tablet_copy_client.cc:783] T 9844e983dd0a403abe0ac90cd96690d6 P bc20559563fe44efa0a2b7b084aafbce: tablet copy: Starting download of 0 data blocks...
I20240502 21:27:04.771049 23193 tablet_copy_client.cc:647] T 9844e983dd0a403abe0ac90cd96690d6 P bc20559563fe44efa0a2b7b084aafbce: tablet copy: Starting download of 1 WAL segments...
I20240502 21:27:04.775884 23193 tablet_copy_client.cc:523] T 9844e983dd0a403abe0ac90cd96690d6 P bc20559563fe44efa0a2b7b084aafbce: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:27:04.779623 23193 tablet_bootstrap.cc:492] T 9844e983dd0a403abe0ac90cd96690d6 P bc20559563fe44efa0a2b7b084aafbce: Bootstrap starting.
I20240502 21:27:04.811949 19140 raft_consensus.cc:1261] T ed78d40318fe4ab5bee4a01b182a34b6 P 2877108756304e05bd60ba1186b870da [term 2 FOLLOWER]: Refusing update from remote peer bc20559563fe44efa0a2b7b084aafbce: Log matching property violated. Preceding OpId in replica: term: 1 index: 499. Preceding OpId from leader: term: 2 index: 500. (index mismatch)
I20240502 21:27:04.812649 23220 consensus_queue.cc:1035] T ed78d40318fe4ab5bee4a01b182a34b6 P bc20559563fe44efa0a2b7b084aafbce [LEADER]: Connected to new peer: Peer: permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 500, Last known committed idx: 499, Time since last communication: 0.000s
I20240502 21:27:04.843361 19005 raft_consensus.cc:1261] T ed78d40318fe4ab5bee4a01b182a34b6 P 9be399440ded4e2ebe6b6e0bf486cbb1 [term 2 FOLLOWER]: Refusing update from remote peer bc20559563fe44efa0a2b7b084aafbce: Log matching property violated. Preceding OpId in replica: term: 1 index: 499. Preceding OpId from leader: term: 2 index: 500. (index mismatch)
I20240502 21:27:04.844182 18473 raft_consensus.cc:1261] T ed78d40318fe4ab5bee4a01b182a34b6 P faadabfb099b4abf90a466395be585b5 [term 2 FOLLOWER]: Refusing update from remote peer bc20559563fe44efa0a2b7b084aafbce: Log matching property violated. Preceding OpId in replica: term: 1 index: 499. Preceding OpId from leader: term: 2 index: 500. (index mismatch)
I20240502 21:27:04.846318 22948 consensus_queue.cc:1035] T ed78d40318fe4ab5bee4a01b182a34b6 P bc20559563fe44efa0a2b7b084aafbce [LEADER]: Connected to new peer: Peer: permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 500, Last known committed idx: 499, Time since last communication: 0.000s
I20240502 21:27:04.848276 22948 consensus_queue.cc:1035] T ed78d40318fe4ab5bee4a01b182a34b6 P bc20559563fe44efa0a2b7b084aafbce [LEADER]: Connected to new peer: Peer: permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 500, Last known committed idx: 499, Time since last communication: 0.000s
I20240502 21:27:04.901612 23202 tablet_bootstrap.cc:492] T b0852f54089944ce935f8ee4cbf2bc5a P bc20559563fe44efa0a2b7b084aafbce: Bootstrap replayed 1/1 log segments. Stats: ops{read=453 overwritten=0 applied=453 ignored=0} inserts{seen=1081 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:27:04.906392 23202 tablet_bootstrap.cc:492] T b0852f54089944ce935f8ee4cbf2bc5a P bc20559563fe44efa0a2b7b084aafbce: Bootstrap complete.
I20240502 21:27:04.907248 23202 ts_tablet_manager.cc:1367] T b0852f54089944ce935f8ee4cbf2bc5a P bc20559563fe44efa0a2b7b084aafbce: Time spent bootstrapping tablet: real 0.562s user 0.462s sys 0.038s
I20240502 21:27:04.908738 23202 raft_consensus.cc:348] T b0852f54089944ce935f8ee4cbf2bc5a P bc20559563fe44efa0a2b7b084aafbce [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 453 OBSOLETE_local: false peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false replace: true } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: true } }
I20240502 21:27:04.910284 23202 raft_consensus.cc:729] T b0852f54089944ce935f8ee4cbf2bc5a P bc20559563fe44efa0a2b7b084aafbce [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: bc20559563fe44efa0a2b7b084aafbce, State: Initialized, Role: LEARNER
I20240502 21:27:04.910724 23202 consensus_queue.cc:260] T b0852f54089944ce935f8ee4cbf2bc5a P bc20559563fe44efa0a2b7b084aafbce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 453, Last appended: 1.453, Last appended by leader: 453, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 453 OBSOLETE_local: false peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false replace: true } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: true } }
I20240502 21:27:04.912691 23202 ts_tablet_manager.cc:1398] T b0852f54089944ce935f8ee4cbf2bc5a P bc20559563fe44efa0a2b7b084aafbce: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240502 21:27:04.914568 19166 tablet_copy_service.cc:342] P 2877108756304e05bd60ba1186b870da: Request end of tablet copy session bc20559563fe44efa0a2b7b084aafbce-b0852f54089944ce935f8ee4cbf2bc5a received from {username='slave'} at 127.15.255.136:57045
I20240502 21:27:04.914721 19166 tablet_copy_service.cc:433] P 2877108756304e05bd60ba1186b870da: ending tablet copy session bc20559563fe44efa0a2b7b084aafbce-b0852f54089944ce935f8ee4cbf2bc5a on tablet b0852f54089944ce935f8ee4cbf2bc5a with peer bc20559563fe44efa0a2b7b084aafbce
I20240502 21:27:05.107003 23214 tablet_bootstrap.cc:492] T 37e9dcc6191c42d3a5c1503dedc5511e P bc20559563fe44efa0a2b7b084aafbce: Bootstrap replayed 1/1 log segments. Stats: ops{read=532 overwritten=0 applied=532 ignored=0} inserts{seen=1214 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:27:05.107456 23214 tablet_bootstrap.cc:492] T 37e9dcc6191c42d3a5c1503dedc5511e P bc20559563fe44efa0a2b7b084aafbce: Bootstrap complete.
I20240502 21:27:05.107785 23214 ts_tablet_manager.cc:1367] T 37e9dcc6191c42d3a5c1503dedc5511e P bc20559563fe44efa0a2b7b084aafbce: Time spent bootstrapping tablet: real 0.758s user 0.549s sys 0.075s
I20240502 21:27:05.108590 23214 raft_consensus.cc:348] T 37e9dcc6191c42d3a5c1503dedc5511e P bc20559563fe44efa0a2b7b084aafbce [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 532 OBSOLETE_local: false peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } attrs { replace: true } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } attrs { promote: false } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: true } }
I20240502 21:27:05.108875 23214 raft_consensus.cc:729] T 37e9dcc6191c42d3a5c1503dedc5511e P bc20559563fe44efa0a2b7b084aafbce [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: bc20559563fe44efa0a2b7b084aafbce, State: Initialized, Role: LEARNER
I20240502 21:27:05.109112 23214 consensus_queue.cc:260] T 37e9dcc6191c42d3a5c1503dedc5511e P bc20559563fe44efa0a2b7b084aafbce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 532, Last appended: 1.532, Last appended by leader: 532, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 532 OBSOLETE_local: false peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } attrs { replace: true } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } attrs { promote: false } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: true } }
I20240502 21:27:05.109788 23214 ts_tablet_manager.cc:1398] T 37e9dcc6191c42d3a5c1503dedc5511e P bc20559563fe44efa0a2b7b084aafbce: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240502 21:27:05.110671 18898 tablet_copy_service.cc:342] P c75831edb3b44dd59c00ef151adca49d: Request end of tablet copy session bc20559563fe44efa0a2b7b084aafbce-37e9dcc6191c42d3a5c1503dedc5511e received from {username='slave'} at 127.15.255.136:39477
I20240502 21:27:05.110816 18898 tablet_copy_service.cc:433] P c75831edb3b44dd59c00ef151adca49d: ending tablet copy session bc20559563fe44efa0a2b7b084aafbce-37e9dcc6191c42d3a5c1503dedc5511e on tablet 37e9dcc6191c42d3a5c1503dedc5511e with peer bc20559563fe44efa0a2b7b084aafbce
I20240502 21:27:05.132876 22992 raft_consensus.cc:1050] T 40d4f61d0f6d4d31a5626c7743c88133 P faadabfb099b4abf90a466395be585b5: attempting to promote NON_VOTER bc20559563fe44efa0a2b7b084aafbce to VOTER
I20240502 21:27:05.133742 22992 consensus_queue.cc:237] T 40d4f61d0f6d4d31a5626c7743c88133 P faadabfb099b4abf90a466395be585b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } attrs { replace: true } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: false } }
I20240502 21:27:05.136169 22631 raft_consensus.cc:1261] T 40d4f61d0f6d4d31a5626c7743c88133 P bc20559563fe44efa0a2b7b084aafbce [term 1 LEARNER]: Refusing update from remote peer faadabfb099b4abf90a466395be585b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:27:05.136893 18744 raft_consensus.cc:1261] T 40d4f61d0f6d4d31a5626c7743c88133 P 501e3c0d44fe4d70af9ff926d9622417 [term 1 FOLLOWER]: Refusing update from remote peer faadabfb099b4abf90a466395be585b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:27:05.138000 20167 raft_consensus.cc:1261] T 40d4f61d0f6d4d31a5626c7743c88133 P 90e329b457c24caf93a9f880345eb288 [term 1 FOLLOWER]: Refusing update from remote peer faadabfb099b4abf90a466395be585b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:27:05.138263 19005 raft_consensus.cc:1261] T 40d4f61d0f6d4d31a5626c7743c88133 P 9be399440ded4e2ebe6b6e0bf486cbb1 [term 1 FOLLOWER]: Refusing update from remote peer faadabfb099b4abf90a466395be585b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:27:05.138640 23299 consensus_queue.cc:1035] T 40d4f61d0f6d4d31a5626c7743c88133 P faadabfb099b4abf90a466395be585b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:27:05.139128 23298 consensus_queue.cc:1035] T 40d4f61d0f6d4d31a5626c7743c88133 P faadabfb099b4abf90a466395be585b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:27:05.139276 18610 raft_consensus.cc:1261] T 40d4f61d0f6d4d31a5626c7743c88133 P 82f50f4b66f3450aa9f58a0b317f5cea [term 1 FOLLOWER]: Refusing update from remote peer faadabfb099b4abf90a466395be585b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 21:27:05.139741 23298 consensus_queue.cc:1035] T 40d4f61d0f6d4d31a5626c7743c88133 P faadabfb099b4abf90a466395be585b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:27:05.140236 23298 consensus_queue.cc:1035] T 40d4f61d0f6d4d31a5626c7743c88133 P faadabfb099b4abf90a466395be585b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:27:05.142313 23080 consensus_queue.cc:1035] T 40d4f61d0f6d4d31a5626c7743c88133 P faadabfb099b4abf90a466395be585b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 21:27:05.146238 23298 raft_consensus.cc:2932] T 40d4f61d0f6d4d31a5626c7743c88133 P faadabfb099b4abf90a466395be585b5 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, bc20559563fe44efa0a2b7b084aafbce (127.15.255.136) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } attrs { replace: true } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: false } } }
I20240502 21:27:05.147425 22631 raft_consensus.cc:2932] T 40d4f61d0f6d4d31a5626c7743c88133 P bc20559563fe44efa0a2b7b084aafbce [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, bc20559563fe44efa0a2b7b084aafbce (127.15.255.136) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } attrs { replace: true } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: false } } }
I20240502 21:27:05.148344 18744 raft_consensus.cc:2932] T 40d4f61d0f6d4d31a5626c7743c88133 P 501e3c0d44fe4d70af9ff926d9622417 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, bc20559563fe44efa0a2b7b084aafbce (127.15.255.136) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } attrs { replace: true } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: false } } }
I20240502 21:27:05.152391 19005 raft_consensus.cc:2932] T 40d4f61d0f6d4d31a5626c7743c88133 P 9be399440ded4e2ebe6b6e0bf486cbb1 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, bc20559563fe44efa0a2b7b084aafbce (127.15.255.136) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } attrs { replace: true } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: false } } }
I20240502 21:27:05.153988 18610 raft_consensus.cc:2932] T 40d4f61d0f6d4d31a5626c7743c88133 P 82f50f4b66f3450aa9f58a0b317f5cea [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, bc20559563fe44efa0a2b7b084aafbce (127.15.255.136) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } attrs { replace: true } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: false } } }
I20240502 21:27:05.156174 20167 raft_consensus.cc:2932] T 40d4f61d0f6d4d31a5626c7743c88133 P 90e329b457c24caf93a9f880345eb288 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, bc20559563fe44efa0a2b7b084aafbce (127.15.255.136) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } attrs { replace: true } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: false } } }
I20240502 21:27:05.161916 18327 catalog_manager.cc:5484] T 40d4f61d0f6d4d31a5626c7743c88133 P 9be399440ded4e2ebe6b6e0bf486cbb1 reported cstate change: config changed from index 2 to 3, bc20559563fe44efa0a2b7b084aafbce (127.15.255.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "faadabfb099b4abf90a466395be585b5" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } attrs { replace: true } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: false } } }
I20240502 21:27:05.178243 22631 raft_consensus.cc:1203] T b0852f54089944ce935f8ee4cbf2bc5a P bc20559563fe44efa0a2b7b084aafbce [term 1 LEARNER]: Deduplicated request from leader. Original: 1.452->[1.453-1.453] Dedup: 1.453->[]
I20240502 21:27:05.191919 18473 consensus_queue.cc:237] T 40d4f61d0f6d4d31a5626c7743c88133 P faadabfb099b4abf90a466395be585b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: false } }
I20240502 21:27:05.194909 19005 raft_consensus.cc:1261] T 40d4f61d0f6d4d31a5626c7743c88133 P 9be399440ded4e2ebe6b6e0bf486cbb1 [term 1 FOLLOWER]: Refusing update from remote peer faadabfb099b4abf90a466395be585b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:27:05.194963 18610 raft_consensus.cc:1261] T 40d4f61d0f6d4d31a5626c7743c88133 P 82f50f4b66f3450aa9f58a0b317f5cea [term 1 FOLLOWER]: Refusing update from remote peer faadabfb099b4abf90a466395be585b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:27:05.195107 22631 raft_consensus.cc:1261] T 40d4f61d0f6d4d31a5626c7743c88133 P bc20559563fe44efa0a2b7b084aafbce [term 1 FOLLOWER]: Refusing update from remote peer faadabfb099b4abf90a466395be585b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:27:05.195734 20167 raft_consensus.cc:1261] T 40d4f61d0f6d4d31a5626c7743c88133 P 90e329b457c24caf93a9f880345eb288 [term 1 FOLLOWER]: Refusing update from remote peer faadabfb099b4abf90a466395be585b5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:27:05.195735 23298 consensus_queue.cc:1035] T 40d4f61d0f6d4d31a5626c7743c88133 P faadabfb099b4abf90a466395be585b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:27:05.196388 23298 consensus_queue.cc:1035] T 40d4f61d0f6d4d31a5626c7743c88133 P faadabfb099b4abf90a466395be585b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:27:05.197047 23305 consensus_queue.cc:1035] T 40d4f61d0f6d4d31a5626c7743c88133 P faadabfb099b4abf90a466395be585b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:27:05.198288 23299 consensus_queue.cc:1035] T 40d4f61d0f6d4d31a5626c7743c88133 P faadabfb099b4abf90a466395be585b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:27:05.200295 23080 raft_consensus.cc:2932] T 40d4f61d0f6d4d31a5626c7743c88133 P faadabfb099b4abf90a466395be585b5 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 501e3c0d44fe4d70af9ff926d9622417 (127.15.255.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: false } } }
I20240502 21:27:05.201284 22631 raft_consensus.cc:2932] T 40d4f61d0f6d4d31a5626c7743c88133 P bc20559563fe44efa0a2b7b084aafbce [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 501e3c0d44fe4d70af9ff926d9622417 (127.15.255.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: false } } }
I20240502 21:27:05.206605 18313 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 40d4f61d0f6d4d31a5626c7743c88133 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:27:05.207998 18610 raft_consensus.cc:2932] T 40d4f61d0f6d4d31a5626c7743c88133 P 82f50f4b66f3450aa9f58a0b317f5cea [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 501e3c0d44fe4d70af9ff926d9622417 (127.15.255.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: false } } }
I20240502 21:27:05.209018 18324 catalog_manager.cc:5484] T 40d4f61d0f6d4d31a5626c7743c88133 P bc20559563fe44efa0a2b7b084aafbce reported cstate change: config changed from index 3 to 4, VOTER 501e3c0d44fe4d70af9ff926d9622417 (127.15.255.131) evicted. New cstate: current_term: 1 leader_uuid: "faadabfb099b4abf90a466395be585b5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: false } } }
I20240502 21:27:05.209734 19005 raft_consensus.cc:2932] T 40d4f61d0f6d4d31a5626c7743c88133 P 9be399440ded4e2ebe6b6e0bf486cbb1 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 501e3c0d44fe4d70af9ff926d9622417 (127.15.255.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: false } } }
I20240502 21:27:05.214443 20167 raft_consensus.cc:2932] T 40d4f61d0f6d4d31a5626c7743c88133 P 90e329b457c24caf93a9f880345eb288 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 501e3c0d44fe4d70af9ff926d9622417 (127.15.255.131) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: false } } }
I20240502 21:27:05.221596 18722 tablet_service.cc:1514] Processing DeleteTablet for tablet 40d4f61d0f6d4d31a5626c7743c88133 with delete_type TABLET_DATA_TOMBSTONED (TS 501e3c0d44fe4d70af9ff926d9622417 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:56558
I20240502 21:27:05.230049 23311 tablet_replica.cc:330] T 40d4f61d0f6d4d31a5626c7743c88133 P 501e3c0d44fe4d70af9ff926d9622417: stopping tablet replica
I20240502 21:27:05.230424 23311 raft_consensus.cc:2229] T 40d4f61d0f6d4d31a5626c7743c88133 P 501e3c0d44fe4d70af9ff926d9622417 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:05.230855 23311 raft_consensus.cc:2258] T 40d4f61d0f6d4d31a5626c7743c88133 P 501e3c0d44fe4d70af9ff926d9622417 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:05.234257 23311 ts_tablet_manager.cc:1875] T 40d4f61d0f6d4d31a5626c7743c88133 P 501e3c0d44fe4d70af9ff926d9622417: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:27:05.239161 22631 raft_consensus.cc:1203] T 37e9dcc6191c42d3a5c1503dedc5511e P bc20559563fe44efa0a2b7b084aafbce [term 1 LEARNER]: Deduplicated request from leader. Original: 1.531->[1.532-1.532] Dedup: 1.532->[]
I20240502 21:27:05.241500 23311 ts_tablet_manager.cc:1888] T 40d4f61d0f6d4d31a5626c7743c88133 P 501e3c0d44fe4d70af9ff926d9622417: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 21:27:05.241708 23311 log.cc:1196] T 40d4f61d0f6d4d31a5626c7743c88133 P 501e3c0d44fe4d70af9ff926d9622417: Deleting WAL directory at /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-2/wal/wals/40d4f61d0f6d4d31a5626c7743c88133
I20240502 21:27:05.243103 18312 catalog_manager.cc:4830] TS 501e3c0d44fe4d70af9ff926d9622417 (127.15.255.131:42501): tablet 40d4f61d0f6d4d31a5626c7743c88133 (table TestTable [id=f30df752b55b4a89883ebf66115566aa]) successfully deleted
I20240502 21:27:05.300052 18610 consensus_queue.cc:237] T 2d111096a9af46068a2476a6231d6520 P 82f50f4b66f3450aa9f58a0b317f5cea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 600, Committed index: 600, Last appended: 1.600, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 601 OBSOLETE_local: false peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } attrs { replace: true } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: true } }
I20240502 21:27:05.303172 18876 raft_consensus.cc:1261] T 2d111096a9af46068a2476a6231d6520 P c75831edb3b44dd59c00ef151adca49d [term 1 FOLLOWER]: Refusing update from remote peer 82f50f4b66f3450aa9f58a0b317f5cea: Log matching property violated. Preceding OpId in replica: term: 1 index: 600. Preceding OpId from leader: term: 1 index: 601. (index mismatch)
W20240502 21:27:05.303738 18543 consensus_peers.cc:485] T 2d111096a9af46068a2476a6231d6520 P 82f50f4b66f3450aa9f58a0b317f5cea -> Peer bc20559563fe44efa0a2b7b084aafbce (127.15.255.136:34965): Couldn't send request to peer bc20559563fe44efa0a2b7b084aafbce. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2d111096a9af46068a2476a6231d6520. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:05.303725 18744 raft_consensus.cc:1261] T 2d111096a9af46068a2476a6231d6520 P 501e3c0d44fe4d70af9ff926d9622417 [term 1 FOLLOWER]: Refusing update from remote peer 82f50f4b66f3450aa9f58a0b317f5cea: Log matching property violated. Preceding OpId in replica: term: 1 index: 600. Preceding OpId from leader: term: 1 index: 601. (index mismatch)
I20240502 21:27:05.304299 19146 raft_consensus.cc:1261] T 2d111096a9af46068a2476a6231d6520 P 2877108756304e05bd60ba1186b870da [term 1 FOLLOWER]: Refusing update from remote peer 82f50f4b66f3450aa9f58a0b317f5cea: Log matching property violated. Preceding OpId in replica: term: 1 index: 600. Preceding OpId from leader: term: 1 index: 601. (index mismatch)
I20240502 21:27:05.304297 23343 consensus_queue.cc:1035] T 2d111096a9af46068a2476a6231d6520 P 82f50f4b66f3450aa9f58a0b317f5cea [LEADER]: Connected to new peer: Peer: permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 601, Last known committed idx: 600, Time since last communication: 0.000s
I20240502 21:27:05.305132 23199 consensus_queue.cc:1035] T 2d111096a9af46068a2476a6231d6520 P 82f50f4b66f3450aa9f58a0b317f5cea [LEADER]: Connected to new peer: Peer: permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 601, Last known committed idx: 600, Time since last communication: 0.000s
I20240502 21:27:05.305454 22925 consensus_queue.cc:1035] T 2d111096a9af46068a2476a6231d6520 P 82f50f4b66f3450aa9f58a0b317f5cea [LEADER]: Connected to new peer: Peer: permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 601, Last known committed idx: 600, Time since last communication: 0.000s
I20240502 21:27:05.307102 18473 raft_consensus.cc:1261] T 2d111096a9af46068a2476a6231d6520 P faadabfb099b4abf90a466395be585b5 [term 1 FOLLOWER]: Refusing update from remote peer 82f50f4b66f3450aa9f58a0b317f5cea: Log matching property violated. Preceding OpId in replica: term: 1 index: 600. Preceding OpId from leader: term: 1 index: 601. (index mismatch)
I20240502 21:27:05.307598 23225 consensus_queue.cc:1035] T 2d111096a9af46068a2476a6231d6520 P 82f50f4b66f3450aa9f58a0b317f5cea [LEADER]: Connected to new peer: Peer: permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 601, Last known committed idx: 600, Time since last communication: 0.000s
I20240502 21:27:05.311409 23199 raft_consensus.cc:2932] T 2d111096a9af46068a2476a6231d6520 P 82f50f4b66f3450aa9f58a0b317f5cea [term 1 LEADER]: Committing config change with OpId 1.601: config changed from index -1 to 601, NON_VOTER bc20559563fe44efa0a2b7b084aafbce (127.15.255.136) added. New config: { opid_index: 601 OBSOLETE_local: false peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } attrs { replace: true } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: true } } }
I20240502 21:27:05.312187 18876 raft_consensus.cc:2932] T 2d111096a9af46068a2476a6231d6520 P c75831edb3b44dd59c00ef151adca49d [term 1 FOLLOWER]: Committing config change with OpId 1.601: config changed from index -1 to 601, NON_VOTER bc20559563fe44efa0a2b7b084aafbce (127.15.255.136) added. New config: { opid_index: 601 OBSOLETE_local: false peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } attrs { replace: true } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: true } } }
I20240502 21:27:05.312808 18473 raft_consensus.cc:2932] T 2d111096a9af46068a2476a6231d6520 P faadabfb099b4abf90a466395be585b5 [term 1 FOLLOWER]: Committing config change with OpId 1.601: config changed from index -1 to 601, NON_VOTER bc20559563fe44efa0a2b7b084aafbce (127.15.255.136) added. New config: { opid_index: 601 OBSOLETE_local: false peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } attrs { replace: true } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: true } } }
I20240502 21:27:05.314447 23193 tablet_bootstrap.cc:492] T 9844e983dd0a403abe0ac90cd96690d6 P bc20559563fe44efa0a2b7b084aafbce: Bootstrap replayed 1/1 log segments. Stats: ops{read=406 overwritten=0 applied=406 ignored=0} inserts{seen=1078 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:27:05.314862 23193 tablet_bootstrap.cc:492] T 9844e983dd0a403abe0ac90cd96690d6 P bc20559563fe44efa0a2b7b084aafbce: Bootstrap complete.
I20240502 21:27:05.315184 23193 ts_tablet_manager.cc:1367] T 9844e983dd0a403abe0ac90cd96690d6 P bc20559563fe44efa0a2b7b084aafbce: Time spent bootstrapping tablet: real 0.536s user 0.419s sys 0.068s
I20240502 21:27:05.315521 19146 raft_consensus.cc:2932] T 2d111096a9af46068a2476a6231d6520 P 2877108756304e05bd60ba1186b870da [term 1 FOLLOWER]: Committing config change with OpId 1.601: config changed from index -1 to 601, NON_VOTER bc20559563fe44efa0a2b7b084aafbce (127.15.255.136) added. New config: { opid_index: 601 OBSOLETE_local: false peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } attrs { replace: true } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: true } } }
I20240502 21:27:05.316913 23193 raft_consensus.cc:348] T 9844e983dd0a403abe0ac90cd96690d6 P bc20559563fe44efa0a2b7b084aafbce [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 406 OBSOLETE_local: false peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } attrs { replace: true } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: true } }
I20240502 21:27:05.317250 23193 raft_consensus.cc:729] T 9844e983dd0a403abe0ac90cd96690d6 P bc20559563fe44efa0a2b7b084aafbce [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: bc20559563fe44efa0a2b7b084aafbce, State: Initialized, Role: LEARNER
I20240502 21:27:05.317461 23193 consensus_queue.cc:260] T 9844e983dd0a403abe0ac90cd96690d6 P bc20559563fe44efa0a2b7b084aafbce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 406, Last appended: 1.406, Last appended by leader: 406, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 406 OBSOLETE_local: false peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } attrs { replace: true } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: true } }
I20240502 21:27:05.318109 23193 ts_tablet_manager.cc:1398] T 9844e983dd0a403abe0ac90cd96690d6 P bc20559563fe44efa0a2b7b084aafbce: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20240502 21:27:05.318482 18324 catalog_manager.cc:5484] T 2d111096a9af46068a2476a6231d6520 P c75831edb3b44dd59c00ef151adca49d reported cstate change: config changed from index -1 to 601, NON_VOTER bc20559563fe44efa0a2b7b084aafbce (127.15.255.136) added. New cstate: current_term: 1 leader_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" committed_config { opid_index: 601 OBSOLETE_local: false peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } attrs { replace: true } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: true } } }
I20240502 21:27:05.320714 18744 raft_consensus.cc:2932] T 2d111096a9af46068a2476a6231d6520 P 501e3c0d44fe4d70af9ff926d9622417 [term 1 FOLLOWER]: Committing config change with OpId 1.601: config changed from index -1 to 601, NON_VOTER bc20559563fe44efa0a2b7b084aafbce (127.15.255.136) added. New config: { opid_index: 601 OBSOLETE_local: false peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } attrs { replace: true } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: true } } }
I20240502 21:27:05.323851 18632 tablet_copy_service.cc:342] P 82f50f4b66f3450aa9f58a0b317f5cea: Request end of tablet copy session bc20559563fe44efa0a2b7b084aafbce-9844e983dd0a403abe0ac90cd96690d6 received from {username='slave'} at 127.15.255.136:48763
I20240502 21:27:05.323989 18632 tablet_copy_service.cc:433] P 82f50f4b66f3450aa9f58a0b317f5cea: ending tablet copy session bc20559563fe44efa0a2b7b084aafbce-9844e983dd0a403abe0ac90cd96690d6 on tablet 9844e983dd0a403abe0ac90cd96690d6 with peer bc20559563fe44efa0a2b7b084aafbce
I20240502 21:27:05.349280 22631 consensus_queue.cc:237] T ed78d40318fe4ab5bee4a01b182a34b6 P bc20559563fe44efa0a2b7b084aafbce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 500, Committed index: 500, Last appended: 2.500, Last appended by leader: 499, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 501 OBSOLETE_local: false peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: false } }
I20240502 21:27:05.351959 19005 raft_consensus.cc:1261] T ed78d40318fe4ab5bee4a01b182a34b6 P 9be399440ded4e2ebe6b6e0bf486cbb1 [term 2 FOLLOWER]: Refusing update from remote peer bc20559563fe44efa0a2b7b084aafbce: Log matching property violated. Preceding OpId in replica: term: 2 index: 500. Preceding OpId from leader: term: 2 index: 501. (index mismatch)
I20240502 21:27:05.352458 18876 raft_consensus.cc:1261] T ed78d40318fe4ab5bee4a01b182a34b6 P c75831edb3b44dd59c00ef151adca49d [term 2 FOLLOWER]: Refusing update from remote peer bc20559563fe44efa0a2b7b084aafbce: Log matching property violated. Preceding OpId in replica: term: 2 index: 500. Preceding OpId from leader: term: 2 index: 501. (index mismatch)
I20240502 21:27:05.352608 22948 consensus_queue.cc:1035] T ed78d40318fe4ab5bee4a01b182a34b6 P bc20559563fe44efa0a2b7b084aafbce [LEADER]: Connected to new peer: Peer: permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 501, Last known committed idx: 500, Time since last communication: 0.000s
I20240502 21:27:05.353319 18744 raft_consensus.cc:1261] T ed78d40318fe4ab5bee4a01b182a34b6 P 501e3c0d44fe4d70af9ff926d9622417 [term 2 FOLLOWER]: Refusing update from remote peer bc20559563fe44efa0a2b7b084aafbce: Log matching property violated. Preceding OpId in replica: term: 2 index: 500. Preceding OpId from leader: term: 2 index: 501. (index mismatch)
I20240502 21:27:05.353878 22948 consensus_queue.cc:1035] T ed78d40318fe4ab5bee4a01b182a34b6 P bc20559563fe44efa0a2b7b084aafbce [LEADER]: Connected to new peer: Peer: permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 501, Last known committed idx: 500, Time since last communication: 0.000s
I20240502 21:27:05.351768 19146 raft_consensus.cc:1261] T ed78d40318fe4ab5bee4a01b182a34b6 P 2877108756304e05bd60ba1186b870da [term 2 FOLLOWER]: Refusing update from remote peer bc20559563fe44efa0a2b7b084aafbce: Log matching property violated. Preceding OpId in replica: term: 2 index: 500. Preceding OpId from leader: term: 2 index: 501. (index mismatch)
I20240502 21:27:05.354326 22948 consensus_queue.cc:1035] T ed78d40318fe4ab5bee4a01b182a34b6 P bc20559563fe44efa0a2b7b084aafbce [LEADER]: Connected to new peer: Peer: permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 501, Last known committed idx: 500, Time since last communication: 0.000s
I20240502 21:27:05.354615 23272 consensus_queue.cc:1035] T ed78d40318fe4ab5bee4a01b182a34b6 P bc20559563fe44efa0a2b7b084aafbce [LEADER]: Connected to new peer: Peer: permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 501, Last known committed idx: 500, Time since last communication: 0.000s
I20240502 21:27:05.357779 23220 raft_consensus.cc:2932] T ed78d40318fe4ab5bee4a01b182a34b6 P bc20559563fe44efa0a2b7b084aafbce [term 2 LEADER]: Committing config change with OpId 2.501: config changed from index 499 to 501, VOTER faadabfb099b4abf90a466395be585b5 (127.15.255.129) evicted. New config: { opid_index: 501 OBSOLETE_local: false peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: false } } }
I20240502 21:27:05.359946 19146 raft_consensus.cc:2932] T ed78d40318fe4ab5bee4a01b182a34b6 P 2877108756304e05bd60ba1186b870da [term 2 FOLLOWER]: Committing config change with OpId 2.501: config changed from index 499 to 501, VOTER faadabfb099b4abf90a466395be585b5 (127.15.255.129) evicted. New config: { opid_index: 501 OBSOLETE_local: false peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: false } } }
I20240502 21:27:05.360854 18744 raft_consensus.cc:2932] T ed78d40318fe4ab5bee4a01b182a34b6 P 501e3c0d44fe4d70af9ff926d9622417 [term 2 FOLLOWER]: Committing config change with OpId 2.501: config changed from index 499 to 501, VOTER faadabfb099b4abf90a466395be585b5 (127.15.255.129) evicted. New config: { opid_index: 501 OBSOLETE_local: false peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: false } } }
I20240502 21:27:05.364157 18312 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet ed78d40318fe4ab5bee4a01b182a34b6 with cas_config_opid_index 499: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:27:05.364656 18876 raft_consensus.cc:2932] T ed78d40318fe4ab5bee4a01b182a34b6 P c75831edb3b44dd59c00ef151adca49d [term 2 FOLLOWER]: Committing config change with OpId 2.501: config changed from index 499 to 501, VOTER faadabfb099b4abf90a466395be585b5 (127.15.255.129) evicted. New config: { opid_index: 501 OBSOLETE_local: false peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: false } } }
I20240502 21:27:05.365365 19005 raft_consensus.cc:2932] T ed78d40318fe4ab5bee4a01b182a34b6 P 9be399440ded4e2ebe6b6e0bf486cbb1 [term 2 FOLLOWER]: Committing config change with OpId 2.501: config changed from index 499 to 501, VOTER faadabfb099b4abf90a466395be585b5 (127.15.255.129) evicted. New config: { opid_index: 501 OBSOLETE_local: false peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: false } } }
I20240502 21:27:05.368192 18319 catalog_manager.cc:5484] T ed78d40318fe4ab5bee4a01b182a34b6 P 2877108756304e05bd60ba1186b870da reported cstate change: config changed from index 499 to 501, VOTER faadabfb099b4abf90a466395be585b5 (127.15.255.129) evicted. New cstate: current_term: 2 leader_uuid: "bc20559563fe44efa0a2b7b084aafbce" committed_config { opid_index: 501 OBSOLETE_local: false peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: false } } }
I20240502 21:27:05.376834 18451 tablet_service.cc:1514] Processing DeleteTablet for tablet ed78d40318fe4ab5bee4a01b182a34b6 with delete_type TABLET_DATA_TOMBSTONED (TS faadabfb099b4abf90a466395be585b5 not found in new config with opid_index 501) from {username='slave'} at 127.0.0.1:39378
I20240502 21:27:05.380795 23368 tablet_replica.cc:330] T ed78d40318fe4ab5bee4a01b182a34b6 P faadabfb099b4abf90a466395be585b5: stopping tablet replica
I20240502 21:27:05.381006 23368 raft_consensus.cc:2229] T ed78d40318fe4ab5bee4a01b182a34b6 P faadabfb099b4abf90a466395be585b5 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:05.381242 23368 raft_consensus.cc:2258] T ed78d40318fe4ab5bee4a01b182a34b6 P faadabfb099b4abf90a466395be585b5 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:05.383584 23368 ts_tablet_manager.cc:1875] T ed78d40318fe4ab5bee4a01b182a34b6 P faadabfb099b4abf90a466395be585b5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:27:05.388794 23368 ts_tablet_manager.cc:1888] T ed78d40318fe4ab5bee4a01b182a34b6 P faadabfb099b4abf90a466395be585b5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.500
I20240502 21:27:05.388943 23368 log.cc:1196] T ed78d40318fe4ab5bee4a01b182a34b6 P faadabfb099b4abf90a466395be585b5: Deleting WAL directory at /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-0/wal/wals/ed78d40318fe4ab5bee4a01b182a34b6
I20240502 21:27:05.389695 18313 catalog_manager.cc:4830] TS faadabfb099b4abf90a466395be585b5 (127.15.255.129:32983): tablet ed78d40318fe4ab5bee4a01b182a34b6 (table rebalance_test_table_0 [id=086ede98fc604333833bce759f342073]) successfully deleted
I20240502 21:27:05.551638 22631 raft_consensus.cc:1203] T 9844e983dd0a403abe0ac90cd96690d6 P bc20559563fe44efa0a2b7b084aafbce [term 1 LEARNER]: Deduplicated request from leader. Original: 1.405->[1.406-1.406] Dedup: 1.406->[]
I20240502 21:27:05.565981 23399 raft_consensus.cc:1050] T b0852f54089944ce935f8ee4cbf2bc5a P 2877108756304e05bd60ba1186b870da: attempting to promote NON_VOTER bc20559563fe44efa0a2b7b084aafbce to VOTER
I20240502 21:27:05.566829 23399 consensus_queue.cc:237] T b0852f54089944ce935f8ee4cbf2bc5a P 2877108756304e05bd60ba1186b870da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 453, Committed index: 453, Last appended: 1.453, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 454 OBSOLETE_local: false peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false replace: true } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: false } }
I20240502 21:27:05.569356 22631 raft_consensus.cc:1261] T b0852f54089944ce935f8ee4cbf2bc5a P bc20559563fe44efa0a2b7b084aafbce [term 1 LEARNER]: Refusing update from remote peer 2877108756304e05bd60ba1186b870da: Log matching property violated. Preceding OpId in replica: term: 1 index: 453. Preceding OpId from leader: term: 1 index: 454. (index mismatch)
I20240502 21:27:05.570029 18744 raft_consensus.cc:1261] T b0852f54089944ce935f8ee4cbf2bc5a P 501e3c0d44fe4d70af9ff926d9622417 [term 1 FOLLOWER]: Refusing update from remote peer 2877108756304e05bd60ba1186b870da: Log matching property violated. Preceding OpId in replica: term: 1 index: 453. Preceding OpId from leader: term: 1 index: 454. (index mismatch)
I20240502 21:27:05.570017 20167 raft_consensus.cc:1261] T b0852f54089944ce935f8ee4cbf2bc5a P 90e329b457c24caf93a9f880345eb288 [term 1 FOLLOWER]: Refusing update from remote peer 2877108756304e05bd60ba1186b870da: Log matching property violated. Preceding OpId in replica: term: 1 index: 453. Preceding OpId from leader: term: 1 index: 454. (index mismatch)
I20240502 21:27:05.570199 18610 raft_consensus.cc:1261] T b0852f54089944ce935f8ee4cbf2bc5a P 82f50f4b66f3450aa9f58a0b317f5cea [term 1 FOLLOWER]: Refusing update from remote peer 2877108756304e05bd60ba1186b870da: Log matching property violated. Preceding OpId in replica: term: 1 index: 453. Preceding OpId from leader: term: 1 index: 454. (index mismatch)
I20240502 21:27:05.570259 19005 raft_consensus.cc:1261] T b0852f54089944ce935f8ee4cbf2bc5a P 9be399440ded4e2ebe6b6e0bf486cbb1 [term 1 FOLLOWER]: Refusing update from remote peer 2877108756304e05bd60ba1186b870da: Log matching property violated. Preceding OpId in replica: term: 1 index: 453. Preceding OpId from leader: term: 1 index: 454. (index mismatch)
I20240502 21:27:05.570477 22419 consensus_queue.cc:1035] T b0852f54089944ce935f8ee4cbf2bc5a P 2877108756304e05bd60ba1186b870da [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 454, Last known committed idx: 453, Time since last communication: 0.000s
I20240502 21:27:05.571051 23093 consensus_queue.cc:1035] T b0852f54089944ce935f8ee4cbf2bc5a P 2877108756304e05bd60ba1186b870da [LEADER]: Connected to new peer: Peer: permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 454, Last known committed idx: 453, Time since last communication: 0.000s
I20240502 21:27:05.571353 23401 consensus_queue.cc:1035] T b0852f54089944ce935f8ee4cbf2bc5a P 2877108756304e05bd60ba1186b870da [LEADER]: Connected to new peer: Peer: permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 454, Last known committed idx: 453, Time since last communication: 0.000s
I20240502 21:27:05.571642 23399 consensus_queue.cc:1035] T b0852f54089944ce935f8ee4cbf2bc5a P 2877108756304e05bd60ba1186b870da [LEADER]: Connected to new peer: Peer: permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 454, Last known committed idx: 453, Time since last communication: 0.000s
I20240502 21:27:05.572569 23402 consensus_queue.cc:1035] T b0852f54089944ce935f8ee4cbf2bc5a P 2877108756304e05bd60ba1186b870da [LEADER]: Connected to new peer: Peer: permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 454, Last known committed idx: 453, Time since last communication: 0.000s
I20240502 21:27:05.599831 23402 raft_consensus.cc:2932] T b0852f54089944ce935f8ee4cbf2bc5a P 2877108756304e05bd60ba1186b870da [term 1 LEADER]: Committing config change with OpId 1.454: config changed from index 453 to 454, bc20559563fe44efa0a2b7b084aafbce (127.15.255.136) changed from NON_VOTER to VOTER. New config: { opid_index: 454 OBSOLETE_local: false peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false replace: true } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: false } } }
I20240502 21:27:05.602172 22632 raft_consensus.cc:2932] T b0852f54089944ce935f8ee4cbf2bc5a P bc20559563fe44efa0a2b7b084aafbce [term 1 FOLLOWER]: Committing config change with OpId 1.454: config changed from index 453 to 454, bc20559563fe44efa0a2b7b084aafbce (127.15.255.136) changed from NON_VOTER to VOTER. New config: { opid_index: 454 OBSOLETE_local: false peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false replace: true } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: false } } }
I20240502 21:27:05.603505 20168 raft_consensus.cc:2932] T b0852f54089944ce935f8ee4cbf2bc5a P 90e329b457c24caf93a9f880345eb288 [term 1 FOLLOWER]: Committing config change with OpId 1.454: config changed from index 453 to 454, bc20559563fe44efa0a2b7b084aafbce (127.15.255.136) changed from NON_VOTER to VOTER. New config: { opid_index: 454 OBSOLETE_local: false peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false replace: true } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: false } } }
I20240502 21:27:05.606505 18744 raft_consensus.cc:2932] T b0852f54089944ce935f8ee4cbf2bc5a P 501e3c0d44fe4d70af9ff926d9622417 [term 1 FOLLOWER]: Committing config change with OpId 1.454: config changed from index 453 to 454, bc20559563fe44efa0a2b7b084aafbce (127.15.255.136) changed from NON_VOTER to VOTER. New config: { opid_index: 454 OBSOLETE_local: false peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false replace: true } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: false } } }
I20240502 21:27:05.607545 18321 catalog_manager.cc:5484] T b0852f54089944ce935f8ee4cbf2bc5a P 2877108756304e05bd60ba1186b870da reported cstate change: config changed from index 453 to 454, bc20559563fe44efa0a2b7b084aafbce (127.15.255.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "2877108756304e05bd60ba1186b870da" committed_config { opid_index: 454 OBSOLETE_local: false peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:27:05.607486 19005 raft_consensus.cc:2932] T b0852f54089944ce935f8ee4cbf2bc5a P 9be399440ded4e2ebe6b6e0bf486cbb1 [term 1 FOLLOWER]: Committing config change with OpId 1.454: config changed from index 453 to 454, bc20559563fe44efa0a2b7b084aafbce (127.15.255.136) changed from NON_VOTER to VOTER. New config: { opid_index: 454 OBSOLETE_local: false peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false replace: true } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: false } } }
I20240502 21:27:05.611641 18610 raft_consensus.cc:2932] T b0852f54089944ce935f8ee4cbf2bc5a P 82f50f4b66f3450aa9f58a0b317f5cea [term 1 FOLLOWER]: Committing config change with OpId 1.454: config changed from index 453 to 454, bc20559563fe44efa0a2b7b084aafbce (127.15.255.136) changed from NON_VOTER to VOTER. New config: { opid_index: 454 OBSOLETE_local: false peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false replace: true } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: false } } }
I20240502 21:27:05.618263 19146 consensus_queue.cc:237] T b0852f54089944ce935f8ee4cbf2bc5a P 2877108756304e05bd60ba1186b870da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 454, Committed index: 454, Last appended: 1.454, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 455 OBSOLETE_local: false peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: false } }
I20240502 21:27:05.619961 18744 raft_consensus.cc:1261] T b0852f54089944ce935f8ee4cbf2bc5a P 501e3c0d44fe4d70af9ff926d9622417 [term 1 FOLLOWER]: Refusing update from remote peer 2877108756304e05bd60ba1186b870da: Log matching property violated. Preceding OpId in replica: term: 1 index: 454. Preceding OpId from leader: term: 1 index: 455. (index mismatch)
I20240502 21:27:05.620106 22632 raft_consensus.cc:1261] T b0852f54089944ce935f8ee4cbf2bc5a P bc20559563fe44efa0a2b7b084aafbce [term 1 FOLLOWER]: Refusing update from remote peer 2877108756304e05bd60ba1186b870da: Log matching property violated. Preceding OpId in replica: term: 1 index: 454. Preceding OpId from leader: term: 1 index: 455. (index mismatch)
I20240502 21:27:05.620548 23399 consensus_queue.cc:1035] T b0852f54089944ce935f8ee4cbf2bc5a P 2877108756304e05bd60ba1186b870da [LEADER]: Connected to new peer: Peer: permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 455, Last known committed idx: 454, Time since last communication: 0.000s
I20240502 21:27:05.620841 23093 consensus_queue.cc:1035] T b0852f54089944ce935f8ee4cbf2bc5a P 2877108756304e05bd60ba1186b870da [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 455, Last known committed idx: 454, Time since last communication: 0.000s
I20240502 21:27:05.621294 19005 raft_consensus.cc:1261] T b0852f54089944ce935f8ee4cbf2bc5a P 9be399440ded4e2ebe6b6e0bf486cbb1 [term 1 FOLLOWER]: Refusing update from remote peer 2877108756304e05bd60ba1186b870da: Log matching property violated. Preceding OpId in replica: term: 1 index: 454. Preceding OpId from leader: term: 1 index: 455. (index mismatch)
I20240502 21:27:05.621824 23093 consensus_queue.cc:1035] T b0852f54089944ce935f8ee4cbf2bc5a P 2877108756304e05bd60ba1186b870da [LEADER]: Connected to new peer: Peer: permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 455, Last known committed idx: 454, Time since last communication: 0.000s
I20240502 21:27:05.622061 18610 raft_consensus.cc:1261] T b0852f54089944ce935f8ee4cbf2bc5a P 82f50f4b66f3450aa9f58a0b317f5cea [term 1 FOLLOWER]: Refusing update from remote peer 2877108756304e05bd60ba1186b870da: Log matching property violated. Preceding OpId in replica: term: 1 index: 454. Preceding OpId from leader: term: 1 index: 455. (index mismatch)
I20240502 21:27:05.622660 23093 consensus_queue.cc:1035] T b0852f54089944ce935f8ee4cbf2bc5a P 2877108756304e05bd60ba1186b870da [LEADER]: Connected to new peer: Peer: permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 455, Last known committed idx: 454, Time since last communication: 0.000s
I20240502 21:27:05.624464 23399 raft_consensus.cc:2932] T b0852f54089944ce935f8ee4cbf2bc5a P 2877108756304e05bd60ba1186b870da [term 1 LEADER]: Committing config change with OpId 1.455: config changed from index 454 to 455, VOTER 90e329b457c24caf93a9f880345eb288 (127.15.255.135) evicted. New config: { opid_index: 455 OBSOLETE_local: false peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: false } } }
I20240502 21:27:05.625069 19005 raft_consensus.cc:2932] T b0852f54089944ce935f8ee4cbf2bc5a P 9be399440ded4e2ebe6b6e0bf486cbb1 [term 1 FOLLOWER]: Committing config change with OpId 1.455: config changed from index 454 to 455, VOTER 90e329b457c24caf93a9f880345eb288 (127.15.255.135) evicted. New config: { opid_index: 455 OBSOLETE_local: false peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: false } } }
I20240502 21:27:05.626821 18610 raft_consensus.cc:2932] T b0852f54089944ce935f8ee4cbf2bc5a P 82f50f4b66f3450aa9f58a0b317f5cea [term 1 FOLLOWER]: Committing config change with OpId 1.455: config changed from index 454 to 455, VOTER 90e329b457c24caf93a9f880345eb288 (127.15.255.135) evicted. New config: { opid_index: 455 OBSOLETE_local: false peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: false } } }
I20240502 21:27:05.626787 18739 raft_consensus.cc:2932] T b0852f54089944ce935f8ee4cbf2bc5a P 501e3c0d44fe4d70af9ff926d9622417 [term 1 FOLLOWER]: Committing config change with OpId 1.455: config changed from index 454 to 455, VOTER 90e329b457c24caf93a9f880345eb288 (127.15.255.135) evicted. New config: { opid_index: 455 OBSOLETE_local: false peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: false } } }
I20240502 21:27:05.629451 18315 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet b0852f54089944ce935f8ee4cbf2bc5a with cas_config_opid_index 454: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:27:05.630172 18320 catalog_manager.cc:5484] T b0852f54089944ce935f8ee4cbf2bc5a P 9be399440ded4e2ebe6b6e0bf486cbb1 reported cstate change: config changed from index 454 to 455, VOTER 90e329b457c24caf93a9f880345eb288 (127.15.255.135) evicted. New cstate: current_term: 1 leader_uuid: "2877108756304e05bd60ba1186b870da" committed_config { opid_index: 455 OBSOLETE_local: false peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: false } } }
I20240502 21:27:05.631753 22632 raft_consensus.cc:2932] T b0852f54089944ce935f8ee4cbf2bc5a P bc20559563fe44efa0a2b7b084aafbce [term 1 FOLLOWER]: Committing config change with OpId 1.455: config changed from index 454 to 455, VOTER 90e329b457c24caf93a9f880345eb288 (127.15.255.135) evicted. New config: { opid_index: 455 OBSOLETE_local: false peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: false } } }
I20240502 21:27:05.645617 20147 tablet_service.cc:1514] Processing DeleteTablet for tablet b0852f54089944ce935f8ee4cbf2bc5a with delete_type TABLET_DATA_TOMBSTONED (TS 90e329b457c24caf93a9f880345eb288 not found in new config with opid_index 455) from {username='slave'} at 127.0.0.1:41252
I20240502 21:27:05.646389 23409 tablet_replica.cc:330] T b0852f54089944ce935f8ee4cbf2bc5a P 90e329b457c24caf93a9f880345eb288: stopping tablet replica
I20240502 21:27:05.646754 23409 raft_consensus.cc:2229] T b0852f54089944ce935f8ee4cbf2bc5a P 90e329b457c24caf93a9f880345eb288 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:05.647114 23409 raft_consensus.cc:2258] T b0852f54089944ce935f8ee4cbf2bc5a P 90e329b457c24caf93a9f880345eb288 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:05.648831 23409 ts_tablet_manager.cc:1875] T b0852f54089944ce935f8ee4cbf2bc5a P 90e329b457c24caf93a9f880345eb288: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:27:05.654354 23409 ts_tablet_manager.cc:1888] T b0852f54089944ce935f8ee4cbf2bc5a P 90e329b457c24caf93a9f880345eb288: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.454
I20240502 21:27:05.654479 23409 log.cc:1196] T b0852f54089944ce935f8ee4cbf2bc5a P 90e329b457c24caf93a9f880345eb288: Deleting WAL directory at /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-6/wal/wals/b0852f54089944ce935f8ee4cbf2bc5a
I20240502 21:27:05.655292 18314 catalog_manager.cc:4830] TS 90e329b457c24caf93a9f880345eb288 (127.15.255.135:43395): tablet b0852f54089944ce935f8ee4cbf2bc5a (table rebalance_test_table_4 [id=c79c94b660074ffc94fcea415b4caae8]) successfully deleted
I20240502 21:27:05.667258 23258 raft_consensus.cc:1050] T 37e9dcc6191c42d3a5c1503dedc5511e P c75831edb3b44dd59c00ef151adca49d: attempting to promote NON_VOTER bc20559563fe44efa0a2b7b084aafbce to VOTER
I20240502 21:27:05.668078 23258 consensus_queue.cc:237] T 37e9dcc6191c42d3a5c1503dedc5511e P c75831edb3b44dd59c00ef151adca49d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 532, Committed index: 532, Last appended: 1.532, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 533 OBSOLETE_local: false peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } attrs { replace: true } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } attrs { promote: false } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: false } }
I20240502 21:27:05.670888 22632 raft_consensus.cc:1261] T 37e9dcc6191c42d3a5c1503dedc5511e P bc20559563fe44efa0a2b7b084aafbce [term 1 LEARNER]: Refusing update from remote peer c75831edb3b44dd59c00ef151adca49d: Log matching property violated. Preceding OpId in replica: term: 1 index: 532. Preceding OpId from leader: term: 1 index: 533. (index mismatch)
I20240502 21:27:05.671262 18473 raft_consensus.cc:1261] T 37e9dcc6191c42d3a5c1503dedc5511e P faadabfb099b4abf90a466395be585b5 [term 1 FOLLOWER]: Refusing update from remote peer c75831edb3b44dd59c00ef151adca49d: Log matching property violated. Preceding OpId in replica: term: 1 index: 532. Preceding OpId from leader: term: 1 index: 533. (index mismatch)
I20240502 21:27:05.671772 19146 raft_consensus.cc:1261] T 37e9dcc6191c42d3a5c1503dedc5511e P 2877108756304e05bd60ba1186b870da [term 1 FOLLOWER]: Refusing update from remote peer c75831edb3b44dd59c00ef151adca49d: Log matching property violated. Preceding OpId in replica: term: 1 index: 532. Preceding OpId from leader: term: 1 index: 533. (index mismatch)
I20240502 21:27:05.671968 23047 consensus_queue.cc:1035] T 37e9dcc6191c42d3a5c1503dedc5511e P c75831edb3b44dd59c00ef151adca49d [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 533, Last known committed idx: 532, Time since last communication: 0.000s
I20240502 21:27:05.673285 23260 consensus_queue.cc:1035] T 37e9dcc6191c42d3a5c1503dedc5511e P c75831edb3b44dd59c00ef151adca49d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 533, Last known committed idx: 532, Time since last communication: 0.000s
I20240502 21:27:05.673849 23260 consensus_queue.cc:1035] T 37e9dcc6191c42d3a5c1503dedc5511e P c75831edb3b44dd59c00ef151adca49d [LEADER]: Connected to new peer: Peer: permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 533, Last known committed idx: 532, Time since last communication: 0.000s
I20240502 21:27:05.683069 19005 raft_consensus.cc:1261] T 37e9dcc6191c42d3a5c1503dedc5511e P 9be399440ded4e2ebe6b6e0bf486cbb1 [term 1 FOLLOWER]: Refusing update from remote peer c75831edb3b44dd59c00ef151adca49d: Log matching property violated. Preceding OpId in replica: term: 1 index: 532. Preceding OpId from leader: term: 1 index: 533. (index mismatch)
I20240502 21:27:05.683677 23258 consensus_queue.cc:1035] T 37e9dcc6191c42d3a5c1503dedc5511e P c75831edb3b44dd59c00ef151adca49d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 533, Last known committed idx: 532, Time since last communication: 0.000s
I20240502 21:27:05.684558 18610 raft_consensus.cc:1261] T 37e9dcc6191c42d3a5c1503dedc5511e P 82f50f4b66f3450aa9f58a0b317f5cea [term 1 FOLLOWER]: Refusing update from remote peer c75831edb3b44dd59c00ef151adca49d: Log matching property violated. Preceding OpId in replica: term: 1 index: 532. Preceding OpId from leader: term: 1 index: 533. (index mismatch)
I20240502 21:27:05.685102 23258 consensus_queue.cc:1035] T 37e9dcc6191c42d3a5c1503dedc5511e P c75831edb3b44dd59c00ef151adca49d [LEADER]: Connected to new peer: Peer: permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 533, Last known committed idx: 532, Time since last communication: 0.000s
I20240502 21:27:05.690768 23421 raft_consensus.cc:2932] T 37e9dcc6191c42d3a5c1503dedc5511e P c75831edb3b44dd59c00ef151adca49d [term 1 LEADER]: Committing config change with OpId 1.533: config changed from index 532 to 533, bc20559563fe44efa0a2b7b084aafbce (127.15.255.136) changed from NON_VOTER to VOTER. New config: { opid_index: 533 OBSOLETE_local: false peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } attrs { replace: true } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } attrs { promote: false } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: false } } }
I20240502 21:27:05.695509 18472 raft_consensus.cc:2932] T 37e9dcc6191c42d3a5c1503dedc5511e P faadabfb099b4abf90a466395be585b5 [term 1 FOLLOWER]: Committing config change with OpId 1.533: config changed from index 532 to 533, bc20559563fe44efa0a2b7b084aafbce (127.15.255.136) changed from NON_VOTER to VOTER. New config: { opid_index: 533 OBSOLETE_local: false peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } attrs { replace: true } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } attrs { promote: false } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: false } } }
I20240502 21:27:05.696835 19005 raft_consensus.cc:2932] T 37e9dcc6191c42d3a5c1503dedc5511e P 9be399440ded4e2ebe6b6e0bf486cbb1 [term 1 FOLLOWER]: Committing config change with OpId 1.533: config changed from index 532 to 533, bc20559563fe44efa0a2b7b084aafbce (127.15.255.136) changed from NON_VOTER to VOTER. New config: { opid_index: 533 OBSOLETE_local: false peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } attrs { replace: true } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } attrs { promote: false } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: false } } }
I20240502 21:27:05.696795 22631 raft_consensus.cc:2932] T 37e9dcc6191c42d3a5c1503dedc5511e P bc20559563fe44efa0a2b7b084aafbce [term 1 FOLLOWER]: Committing config change with OpId 1.533: config changed from index 532 to 533, bc20559563fe44efa0a2b7b084aafbce (127.15.255.136) changed from NON_VOTER to VOTER. New config: { opid_index: 533 OBSOLETE_local: false peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } attrs { replace: true } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } attrs { promote: false } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: false } } }
I20240502 21:27:05.701602 18319 catalog_manager.cc:5484] T 37e9dcc6191c42d3a5c1503dedc5511e P faadabfb099b4abf90a466395be585b5 reported cstate change: config changed from index 532 to 533, bc20559563fe44efa0a2b7b084aafbce (127.15.255.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c75831edb3b44dd59c00ef151adca49d" committed_config { opid_index: 533 OBSOLETE_local: false peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } attrs { replace: true } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } attrs { promote: false } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: false } } }
I20240502 21:27:05.701500 19146 raft_consensus.cc:2932] T 37e9dcc6191c42d3a5c1503dedc5511e P 2877108756304e05bd60ba1186b870da [term 1 FOLLOWER]: Committing config change with OpId 1.533: config changed from index 532 to 533, bc20559563fe44efa0a2b7b084aafbce (127.15.255.136) changed from NON_VOTER to VOTER. New config: { opid_index: 533 OBSOLETE_local: false peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } attrs { replace: true } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } attrs { promote: false } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: false } } }
I20240502 21:27:05.709924 18610 raft_consensus.cc:2932] T 37e9dcc6191c42d3a5c1503dedc5511e P 82f50f4b66f3450aa9f58a0b317f5cea [term 1 FOLLOWER]: Committing config change with OpId 1.533: config changed from index 532 to 533, bc20559563fe44efa0a2b7b084aafbce (127.15.255.136) changed from NON_VOTER to VOTER. New config: { opid_index: 533 OBSOLETE_local: false peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } attrs { replace: true } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } attrs { promote: false } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: false } } }
I20240502 21:27:05.711432 18876 consensus_queue.cc:237] T 37e9dcc6191c42d3a5c1503dedc5511e P c75831edb3b44dd59c00ef151adca49d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 533, Committed index: 533, Last appended: 1.533, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 534 OBSOLETE_local: false peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } attrs { promote: false } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: false } }
I20240502 21:27:05.715111 19146 raft_consensus.cc:1261] T 37e9dcc6191c42d3a5c1503dedc5511e P 2877108756304e05bd60ba1186b870da [term 1 FOLLOWER]: Refusing update from remote peer c75831edb3b44dd59c00ef151adca49d: Log matching property violated. Preceding OpId in replica: term: 1 index: 533. Preceding OpId from leader: term: 1 index: 534. (index mismatch)
I20240502 21:27:05.715811 22632 raft_consensus.cc:1261] T 37e9dcc6191c42d3a5c1503dedc5511e P bc20559563fe44efa0a2b7b084aafbce [term 1 FOLLOWER]: Refusing update from remote peer c75831edb3b44dd59c00ef151adca49d: Log matching property violated. Preceding OpId in replica: term: 1 index: 533. Preceding OpId from leader: term: 1 index: 534. (index mismatch)
I20240502 21:27:05.716095 23421 consensus_queue.cc:1035] T 37e9dcc6191c42d3a5c1503dedc5511e P c75831edb3b44dd59c00ef151adca49d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 534, Last known committed idx: 533, Time since last communication: 0.000s
I20240502 21:27:05.716452 23260 consensus_queue.cc:1035] T 37e9dcc6191c42d3a5c1503dedc5511e P c75831edb3b44dd59c00ef151adca49d [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 534, Last known committed idx: 533, Time since last communication: 0.000s
I20240502 21:27:05.716593 18472 raft_consensus.cc:1261] T 37e9dcc6191c42d3a5c1503dedc5511e P faadabfb099b4abf90a466395be585b5 [term 1 FOLLOWER]: Refusing update from remote peer c75831edb3b44dd59c00ef151adca49d: Log matching property violated. Preceding OpId in replica: term: 1 index: 533. Preceding OpId from leader: term: 1 index: 534. (index mismatch)
I20240502 21:27:05.716909 19005 raft_consensus.cc:1261] T 37e9dcc6191c42d3a5c1503dedc5511e P 9be399440ded4e2ebe6b6e0bf486cbb1 [term 1 FOLLOWER]: Refusing update from remote peer c75831edb3b44dd59c00ef151adca49d: Log matching property violated. Preceding OpId in replica: term: 1 index: 533. Preceding OpId from leader: term: 1 index: 534. (index mismatch)
I20240502 21:27:05.717641 23047 consensus_queue.cc:1035] T 37e9dcc6191c42d3a5c1503dedc5511e P c75831edb3b44dd59c00ef151adca49d [LEADER]: Connected to new peer: Peer: permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 534, Last known committed idx: 533, Time since last communication: 0.000s
I20240502 21:27:05.718148 23047 consensus_queue.cc:1035] T 37e9dcc6191c42d3a5c1503dedc5511e P c75831edb3b44dd59c00ef151adca49d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 534, Last known committed idx: 533, Time since last communication: 0.000s
I20240502 21:27:05.721676 23260 raft_consensus.cc:2932] T 37e9dcc6191c42d3a5c1503dedc5511e P c75831edb3b44dd59c00ef151adca49d [term 1 LEADER]: Committing config change with OpId 1.534: config changed from index 533 to 534, VOTER 82f50f4b66f3450aa9f58a0b317f5cea (127.15.255.130) evicted. New config: { opid_index: 534 OBSOLETE_local: false peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } attrs { promote: false } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: false } } }
I20240502 21:27:05.725312 18472 raft_consensus.cc:2932] T 37e9dcc6191c42d3a5c1503dedc5511e P faadabfb099b4abf90a466395be585b5 [term 1 FOLLOWER]: Committing config change with OpId 1.534: config changed from index 533 to 534, VOTER 82f50f4b66f3450aa9f58a0b317f5cea (127.15.255.130) evicted. New config: { opid_index: 534 OBSOLETE_local: false peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } attrs { promote: false } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: false } } }
I20240502 21:27:05.725155 19005 raft_consensus.cc:2932] T 37e9dcc6191c42d3a5c1503dedc5511e P 9be399440ded4e2ebe6b6e0bf486cbb1 [term 1 FOLLOWER]: Committing config change with OpId 1.534: config changed from index 533 to 534, VOTER 82f50f4b66f3450aa9f58a0b317f5cea (127.15.255.130) evicted. New config: { opid_index: 534 OBSOLETE_local: false peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } attrs { promote: false } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: false } } }
I20240502 21:27:05.726697 18314 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 37e9dcc6191c42d3a5c1503dedc5511e with cas_config_opid_index 533: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:27:05.727813 19146 raft_consensus.cc:2932] T 37e9dcc6191c42d3a5c1503dedc5511e P 2877108756304e05bd60ba1186b870da [term 1 FOLLOWER]: Committing config change with OpId 1.534: config changed from index 533 to 534, VOTER 82f50f4b66f3450aa9f58a0b317f5cea (127.15.255.130) evicted. New config: { opid_index: 534 OBSOLETE_local: false peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } attrs { promote: false } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: false } } }
I20240502 21:27:05.733057 18320 catalog_manager.cc:5484] T 37e9dcc6191c42d3a5c1503dedc5511e P 9be399440ded4e2ebe6b6e0bf486cbb1 reported cstate change: config changed from index 533 to 534, VOTER 82f50f4b66f3450aa9f58a0b317f5cea (127.15.255.130) evicted. New cstate: current_term: 1 leader_uuid: "c75831edb3b44dd59c00ef151adca49d" committed_config { opid_index: 534 OBSOLETE_local: false peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } attrs { promote: false } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: false } } }
I20240502 21:27:05.737429 18590 tablet_service.cc:1514] Processing DeleteTablet for tablet 37e9dcc6191c42d3a5c1503dedc5511e with delete_type TABLET_DATA_TOMBSTONED (TS 82f50f4b66f3450aa9f58a0b317f5cea not found in new config with opid_index 534) from {username='slave'} at 127.0.0.1:33750
I20240502 21:27:05.738401 23430 tablet_replica.cc:330] T 37e9dcc6191c42d3a5c1503dedc5511e P 82f50f4b66f3450aa9f58a0b317f5cea: stopping tablet replica
I20240502 21:27:05.738705 23430 raft_consensus.cc:2229] T 37e9dcc6191c42d3a5c1503dedc5511e P 82f50f4b66f3450aa9f58a0b317f5cea [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:05.739131 23430 raft_consensus.cc:2258] T 37e9dcc6191c42d3a5c1503dedc5511e P 82f50f4b66f3450aa9f58a0b317f5cea [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:05.741473 23430 ts_tablet_manager.cc:1875] T 37e9dcc6191c42d3a5c1503dedc5511e P 82f50f4b66f3450aa9f58a0b317f5cea: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:27:05.741928 22631 raft_consensus.cc:2932] T 37e9dcc6191c42d3a5c1503dedc5511e P bc20559563fe44efa0a2b7b084aafbce [term 1 FOLLOWER]: Committing config change with OpId 1.534: config changed from index 533 to 534, VOTER 82f50f4b66f3450aa9f58a0b317f5cea (127.15.255.130) evicted. New config: { opid_index: 534 OBSOLETE_local: false peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } attrs { promote: false } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: false } } }
I20240502 21:27:05.748487 23430 ts_tablet_manager.cc:1888] T 37e9dcc6191c42d3a5c1503dedc5511e P 82f50f4b66f3450aa9f58a0b317f5cea: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.533
I20240502 21:27:05.748677 23430 log.cc:1196] T 37e9dcc6191c42d3a5c1503dedc5511e P 82f50f4b66f3450aa9f58a0b317f5cea: Deleting WAL directory at /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-1/wal/wals/37e9dcc6191c42d3a5c1503dedc5511e
I20240502 21:27:05.749590 18315 catalog_manager.cc:4830] TS 82f50f4b66f3450aa9f58a0b317f5cea (127.15.255.130:45139): tablet 37e9dcc6191c42d3a5c1503dedc5511e (table rebalance_test_table_1 [id=76deee1e4597471ebedfc30f6d635eaa]) successfully deleted
I20240502 21:27:05.876989 23448 ts_tablet_manager.cc:902] T 2d111096a9af46068a2476a6231d6520 P bc20559563fe44efa0a2b7b084aafbce: Initiating tablet copy from peer 82f50f4b66f3450aa9f58a0b317f5cea (127.15.255.130:45139)
I20240502 21:27:05.878062 23448 tablet_copy_client.cc:308] T 2d111096a9af46068a2476a6231d6520 P bc20559563fe44efa0a2b7b084aafbce: tablet copy: Beginning tablet copy session from remote peer at address 127.15.255.130:45139
I20240502 21:27:05.878613 18632 tablet_copy_service.cc:140] P 82f50f4b66f3450aa9f58a0b317f5cea: Received BeginTabletCopySession request for tablet 2d111096a9af46068a2476a6231d6520 from peer bc20559563fe44efa0a2b7b084aafbce ({username='slave'} at 127.15.255.136:48763)
I20240502 21:27:05.878791 18632 tablet_copy_service.cc:161] P 82f50f4b66f3450aa9f58a0b317f5cea: Beginning new tablet copy session on tablet 2d111096a9af46068a2476a6231d6520 from peer bc20559563fe44efa0a2b7b084aafbce at {username='slave'} at 127.15.255.136:48763: session id = bc20559563fe44efa0a2b7b084aafbce-2d111096a9af46068a2476a6231d6520
I20240502 21:27:05.880764 18632 tablet_copy_source_session.cc:205] T 2d111096a9af46068a2476a6231d6520 P 82f50f4b66f3450aa9f58a0b317f5cea: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:27:05.881860 23448 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d111096a9af46068a2476a6231d6520. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:05.887823 23448 tablet_copy_client.cc:783] T 2d111096a9af46068a2476a6231d6520 P bc20559563fe44efa0a2b7b084aafbce: tablet copy: Starting download of 0 data blocks...
I20240502 21:27:05.888031 23448 tablet_copy_client.cc:647] T 2d111096a9af46068a2476a6231d6520 P bc20559563fe44efa0a2b7b084aafbce: tablet copy: Starting download of 1 WAL segments...
I20240502 21:27:05.890695 23448 tablet_copy_client.cc:523] T 2d111096a9af46068a2476a6231d6520 P bc20559563fe44efa0a2b7b084aafbce: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:27:05.894223 23448 tablet_bootstrap.cc:492] T 2d111096a9af46068a2476a6231d6520 P bc20559563fe44efa0a2b7b084aafbce: Bootstrap starting.
I20240502 21:27:06.089138 23348 raft_consensus.cc:1050] T 9844e983dd0a403abe0ac90cd96690d6 P 82f50f4b66f3450aa9f58a0b317f5cea: attempting to promote NON_VOTER bc20559563fe44efa0a2b7b084aafbce to VOTER
I20240502 21:27:06.090133 23348 consensus_queue.cc:237] T 9844e983dd0a403abe0ac90cd96690d6 P 82f50f4b66f3450aa9f58a0b317f5cea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 406, Committed index: 406, Last appended: 1.406, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 407 OBSOLETE_local: false peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } attrs { replace: true } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: false } }
I20240502 21:27:06.093483 18876 raft_consensus.cc:1261] T 9844e983dd0a403abe0ac90cd96690d6 P c75831edb3b44dd59c00ef151adca49d [term 1 FOLLOWER]: Refusing update from remote peer 82f50f4b66f3450aa9f58a0b317f5cea: Log matching property violated. Preceding OpId in replica: term: 1 index: 406. Preceding OpId from leader: term: 1 index: 407. (index mismatch)
I20240502 21:27:06.094466 22631 raft_consensus.cc:1261] T 9844e983dd0a403abe0ac90cd96690d6 P bc20559563fe44efa0a2b7b084aafbce [term 1 LEARNER]: Refusing update from remote peer 82f50f4b66f3450aa9f58a0b317f5cea: Log matching property violated. Preceding OpId in replica: term: 1 index: 406. Preceding OpId from leader: term: 1 index: 407. (index mismatch)
I20240502 21:27:06.094578 19005 raft_consensus.cc:1261] T 9844e983dd0a403abe0ac90cd96690d6 P 9be399440ded4e2ebe6b6e0bf486cbb1 [term 1 FOLLOWER]: Refusing update from remote peer 82f50f4b66f3450aa9f58a0b317f5cea: Log matching property violated. Preceding OpId in replica: term: 1 index: 406. Preceding OpId from leader: term: 1 index: 407. (index mismatch)
I20240502 21:27:06.094955 23464 consensus_queue.cc:1035] T 9844e983dd0a403abe0ac90cd96690d6 P 82f50f4b66f3450aa9f58a0b317f5cea [LEADER]: Connected to new peer: Peer: permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 407, Last known committed idx: 406, Time since last communication: 0.000s
I20240502 21:27:06.095659 23465 consensus_queue.cc:1035] T 9844e983dd0a403abe0ac90cd96690d6 P 82f50f4b66f3450aa9f58a0b317f5cea [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 407, Last known committed idx: 406, Time since last communication: 0.000s
I20240502 21:27:06.095937 23464 consensus_queue.cc:1035] T 9844e983dd0a403abe0ac90cd96690d6 P 82f50f4b66f3450aa9f58a0b317f5cea [LEADER]: Connected to new peer: Peer: permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 407, Last known committed idx: 406, Time since last communication: 0.000s
I20240502 21:27:06.097270 18739 raft_consensus.cc:1261] T 9844e983dd0a403abe0ac90cd96690d6 P 501e3c0d44fe4d70af9ff926d9622417 [term 1 FOLLOWER]: Refusing update from remote peer 82f50f4b66f3450aa9f58a0b317f5cea: Log matching property violated. Preceding OpId in replica: term: 1 index: 406. Preceding OpId from leader: term: 1 index: 407. (index mismatch)
I20240502 21:27:06.097846 23464 consensus_queue.cc:1035] T 9844e983dd0a403abe0ac90cd96690d6 P 82f50f4b66f3450aa9f58a0b317f5cea [LEADER]: Connected to new peer: Peer: permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 407, Last known committed idx: 406, Time since last communication: 0.000s
I20240502 21:27:06.098098 18472 raft_consensus.cc:1261] T 9844e983dd0a403abe0ac90cd96690d6 P faadabfb099b4abf90a466395be585b5 [term 1 FOLLOWER]: Refusing update from remote peer 82f50f4b66f3450aa9f58a0b317f5cea: Log matching property violated. Preceding OpId in replica: term: 1 index: 406. Preceding OpId from leader: term: 1 index: 407. (index mismatch)
I20240502 21:27:06.098757 23465 consensus_queue.cc:1035] T 9844e983dd0a403abe0ac90cd96690d6 P 82f50f4b66f3450aa9f58a0b317f5cea [LEADER]: Connected to new peer: Peer: permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 407, Last known committed idx: 406, Time since last communication: 0.000s
I20240502 21:27:06.117072 18873 raft_consensus.cc:2932] T 9844e983dd0a403abe0ac90cd96690d6 P c75831edb3b44dd59c00ef151adca49d [term 1 FOLLOWER]: Committing config change with OpId 1.407: config changed from index 406 to 407, bc20559563fe44efa0a2b7b084aafbce (127.15.255.136) changed from NON_VOTER to VOTER. New config: { opid_index: 407 OBSOLETE_local: false peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } attrs { replace: true } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: false } } }
I20240502 21:27:06.117993 19009 raft_consensus.cc:2932] T 9844e983dd0a403abe0ac90cd96690d6 P 9be399440ded4e2ebe6b6e0bf486cbb1 [term 1 FOLLOWER]: Committing config change with OpId 1.407: config changed from index 406 to 407, bc20559563fe44efa0a2b7b084aafbce (127.15.255.136) changed from NON_VOTER to VOTER. New config: { opid_index: 407 OBSOLETE_local: false peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } attrs { replace: true } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: false } } }
I20240502 21:27:06.122848 18320 catalog_manager.cc:5484] T 9844e983dd0a403abe0ac90cd96690d6 P c75831edb3b44dd59c00ef151adca49d reported cstate change: config changed from index 406 to 407, bc20559563fe44efa0a2b7b084aafbce (127.15.255.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" committed_config { opid_index: 407 OBSOLETE_local: false peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } attrs { replace: true } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: false } } }
I20240502 21:27:06.116832 23348 raft_consensus.cc:2932] T 9844e983dd0a403abe0ac90cd96690d6 P 82f50f4b66f3450aa9f58a0b317f5cea [term 1 LEADER]: Committing config change with OpId 1.407: config changed from index 406 to 407, bc20559563fe44efa0a2b7b084aafbce (127.15.255.136) changed from NON_VOTER to VOTER. New config: { opid_index: 407 OBSOLETE_local: false peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } attrs { replace: true } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: false } } }
I20240502 21:27:06.128870 18739 raft_consensus.cc:2932] T 9844e983dd0a403abe0ac90cd96690d6 P 501e3c0d44fe4d70af9ff926d9622417 [term 1 FOLLOWER]: Committing config change with OpId 1.407: config changed from index 406 to 407, bc20559563fe44efa0a2b7b084aafbce (127.15.255.136) changed from NON_VOTER to VOTER. New config: { opid_index: 407 OBSOLETE_local: false peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } attrs { replace: true } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: false } } }
I20240502 21:27:06.131420 22631 raft_consensus.cc:2932] T 9844e983dd0a403abe0ac90cd96690d6 P bc20559563fe44efa0a2b7b084aafbce [term 1 FOLLOWER]: Committing config change with OpId 1.407: config changed from index 406 to 407, bc20559563fe44efa0a2b7b084aafbce (127.15.255.136) changed from NON_VOTER to VOTER. New config: { opid_index: 407 OBSOLETE_local: false peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } attrs { replace: true } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: false } } }
I20240502 21:27:06.137756 18472 raft_consensus.cc:2932] T 9844e983dd0a403abe0ac90cd96690d6 P faadabfb099b4abf90a466395be585b5 [term 1 FOLLOWER]: Committing config change with OpId 1.407: config changed from index 406 to 407, bc20559563fe44efa0a2b7b084aafbce (127.15.255.136) changed from NON_VOTER to VOTER. New config: { opid_index: 407 OBSOLETE_local: false peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } attrs { replace: true } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: false } } }
I20240502 21:27:06.148726 18610 consensus_queue.cc:237] T 9844e983dd0a403abe0ac90cd96690d6 P 82f50f4b66f3450aa9f58a0b317f5cea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 407, Committed index: 407, Last appended: 1.407, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 408 OBSOLETE_local: false peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: false } }
I20240502 21:27:06.151247 18873 raft_consensus.cc:1261] T 9844e983dd0a403abe0ac90cd96690d6 P c75831edb3b44dd59c00ef151adca49d [term 1 FOLLOWER]: Refusing update from remote peer 82f50f4b66f3450aa9f58a0b317f5cea: Log matching property violated. Preceding OpId in replica: term: 1 index: 407. Preceding OpId from leader: term: 1 index: 408. (index mismatch)
I20240502 21:27:06.151787 23348 consensus_queue.cc:1035] T 9844e983dd0a403abe0ac90cd96690d6 P 82f50f4b66f3450aa9f58a0b317f5cea [LEADER]: Connected to new peer: Peer: permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 408, Last known committed idx: 407, Time since last communication: 0.000s
I20240502 21:27:06.152156 22631 raft_consensus.cc:1261] T 9844e983dd0a403abe0ac90cd96690d6 P bc20559563fe44efa0a2b7b084aafbce [term 1 FOLLOWER]: Refusing update from remote peer 82f50f4b66f3450aa9f58a0b317f5cea: Log matching property violated. Preceding OpId in replica: term: 1 index: 407. Preceding OpId from leader: term: 1 index: 408. (index mismatch)
I20240502 21:27:06.154176 23348 consensus_queue.cc:1035] T 9844e983dd0a403abe0ac90cd96690d6 P 82f50f4b66f3450aa9f58a0b317f5cea [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 408, Last known committed idx: 407, Time since last communication: 0.000s
I20240502 21:27:06.154477 18472 raft_consensus.cc:1261] T 9844e983dd0a403abe0ac90cd96690d6 P faadabfb099b4abf90a466395be585b5 [term 1 FOLLOWER]: Refusing update from remote peer 82f50f4b66f3450aa9f58a0b317f5cea: Log matching property violated. Preceding OpId in replica: term: 1 index: 407. Preceding OpId from leader: term: 1 index: 408. (index mismatch)
I20240502 21:27:06.154966 23348 consensus_queue.cc:1035] T 9844e983dd0a403abe0ac90cd96690d6 P 82f50f4b66f3450aa9f58a0b317f5cea [LEADER]: Connected to new peer: Peer: permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 408, Last known committed idx: 407, Time since last communication: 0.000s
I20240502 21:27:06.155297 18739 raft_consensus.cc:1261] T 9844e983dd0a403abe0ac90cd96690d6 P 501e3c0d44fe4d70af9ff926d9622417 [term 1 FOLLOWER]: Refusing update from remote peer 82f50f4b66f3450aa9f58a0b317f5cea: Log matching property violated. Preceding OpId in replica: term: 1 index: 407. Preceding OpId from leader: term: 1 index: 408. (index mismatch)
I20240502 21:27:06.155900 23467 consensus_queue.cc:1035] T 9844e983dd0a403abe0ac90cd96690d6 P 82f50f4b66f3450aa9f58a0b317f5cea [LEADER]: Connected to new peer: Peer: permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 408, Last known committed idx: 407, Time since last communication: 0.000s
I20240502 21:27:06.156533 23465 raft_consensus.cc:2932] T 9844e983dd0a403abe0ac90cd96690d6 P 82f50f4b66f3450aa9f58a0b317f5cea [term 1 LEADER]: Committing config change with OpId 1.408: config changed from index 407 to 408, VOTER 9be399440ded4e2ebe6b6e0bf486cbb1 (127.15.255.133) evicted. New config: { opid_index: 408 OBSOLETE_local: false peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: false } } }
I20240502 21:27:06.158821 18472 raft_consensus.cc:2932] T 9844e983dd0a403abe0ac90cd96690d6 P faadabfb099b4abf90a466395be585b5 [term 1 FOLLOWER]: Committing config change with OpId 1.408: config changed from index 407 to 408, VOTER 9be399440ded4e2ebe6b6e0bf486cbb1 (127.15.255.133) evicted. New config: { opid_index: 408 OBSOLETE_local: false peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: false } } }
I20240502 21:27:06.161011 18315 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 9844e983dd0a403abe0ac90cd96690d6 with cas_config_opid_index 407: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:27:06.161648 18873 raft_consensus.cc:2932] T 9844e983dd0a403abe0ac90cd96690d6 P c75831edb3b44dd59c00ef151adca49d [term 1 FOLLOWER]: Committing config change with OpId 1.408: config changed from index 407 to 408, VOTER 9be399440ded4e2ebe6b6e0bf486cbb1 (127.15.255.133) evicted. New config: { opid_index: 408 OBSOLETE_local: false peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: false } } }
I20240502 21:27:06.162223 22632 raft_consensus.cc:2932] T 9844e983dd0a403abe0ac90cd96690d6 P bc20559563fe44efa0a2b7b084aafbce [term 1 FOLLOWER]: Committing config change with OpId 1.408: config changed from index 407 to 408, VOTER 9be399440ded4e2ebe6b6e0bf486cbb1 (127.15.255.133) evicted. New config: { opid_index: 408 OBSOLETE_local: false peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: false } } }
I20240502 21:27:06.166052 18739 raft_consensus.cc:2932] T 9844e983dd0a403abe0ac90cd96690d6 P 501e3c0d44fe4d70af9ff926d9622417 [term 1 FOLLOWER]: Committing config change with OpId 1.408: config changed from index 407 to 408, VOTER 9be399440ded4e2ebe6b6e0bf486cbb1 (127.15.255.133) evicted. New config: { opid_index: 408 OBSOLETE_local: false peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: false } } }
I20240502 21:27:06.166792 18321 catalog_manager.cc:5484] T 9844e983dd0a403abe0ac90cd96690d6 P bc20559563fe44efa0a2b7b084aafbce reported cstate change: config changed from index 407 to 408, VOTER 9be399440ded4e2ebe6b6e0bf486cbb1 (127.15.255.133) evicted. New cstate: current_term: 1 leader_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" committed_config { opid_index: 408 OBSOLETE_local: false peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: false } } }
I20240502 21:27:06.173496 18989 tablet_service.cc:1514] Processing DeleteTablet for tablet 9844e983dd0a403abe0ac90cd96690d6 with delete_type TABLET_DATA_TOMBSTONED (TS 9be399440ded4e2ebe6b6e0bf486cbb1 not found in new config with opid_index 408) from {username='slave'} at 127.0.0.1:40858
I20240502 21:27:06.179211 23473 tablet_replica.cc:330] T 9844e983dd0a403abe0ac90cd96690d6 P 9be399440ded4e2ebe6b6e0bf486cbb1: stopping tablet replica
I20240502 21:27:06.179493 23473 raft_consensus.cc:2229] T 9844e983dd0a403abe0ac90cd96690d6 P 9be399440ded4e2ebe6b6e0bf486cbb1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:06.179841 23473 raft_consensus.cc:2258] T 9844e983dd0a403abe0ac90cd96690d6 P 9be399440ded4e2ebe6b6e0bf486cbb1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:06.181674 23473 ts_tablet_manager.cc:1875] T 9844e983dd0a403abe0ac90cd96690d6 P 9be399440ded4e2ebe6b6e0bf486cbb1: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:27:06.188165 23473 ts_tablet_manager.cc:1888] T 9844e983dd0a403abe0ac90cd96690d6 P 9be399440ded4e2ebe6b6e0bf486cbb1: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.407
I20240502 21:27:06.188314 23473 log.cc:1196] T 9844e983dd0a403abe0ac90cd96690d6 P 9be399440ded4e2ebe6b6e0bf486cbb1: Deleting WAL directory at /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-4/wal/wals/9844e983dd0a403abe0ac90cd96690d6
I20240502 21:27:06.189184 18313 catalog_manager.cc:4830] TS 9be399440ded4e2ebe6b6e0bf486cbb1 (127.15.255.133:34217): tablet 9844e983dd0a403abe0ac90cd96690d6 (table rebalance_test_table_3 [id=5af23c390f054d07b5a7985c2611ca30]) successfully deleted
I20240502 21:27:06.502912 23448 tablet_bootstrap.cc:492] T 2d111096a9af46068a2476a6231d6520 P bc20559563fe44efa0a2b7b084aafbce: Bootstrap replayed 1/1 log segments. Stats: ops{read=601 overwritten=0 applied=601 ignored=0} inserts{seen=1286 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:27:06.503389 23448 tablet_bootstrap.cc:492] T 2d111096a9af46068a2476a6231d6520 P bc20559563fe44efa0a2b7b084aafbce: Bootstrap complete.
I20240502 21:27:06.503700 23448 ts_tablet_manager.cc:1367] T 2d111096a9af46068a2476a6231d6520 P bc20559563fe44efa0a2b7b084aafbce: Time spent bootstrapping tablet: real 0.610s user 0.524s sys 0.072s
I20240502 21:27:06.504408 23448 raft_consensus.cc:348] T 2d111096a9af46068a2476a6231d6520 P bc20559563fe44efa0a2b7b084aafbce [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 601 OBSOLETE_local: false peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } attrs { replace: true } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: true } }
I20240502 21:27:06.504686 23448 raft_consensus.cc:729] T 2d111096a9af46068a2476a6231d6520 P bc20559563fe44efa0a2b7b084aafbce [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: bc20559563fe44efa0a2b7b084aafbce, State: Initialized, Role: LEARNER
I20240502 21:27:06.504878 23448 consensus_queue.cc:260] T 2d111096a9af46068a2476a6231d6520 P bc20559563fe44efa0a2b7b084aafbce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 601, Last appended: 1.601, Last appended by leader: 601, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 601 OBSOLETE_local: false peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } attrs { replace: true } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: true } }
I20240502 21:27:06.505497 23448 ts_tablet_manager.cc:1398] T 2d111096a9af46068a2476a6231d6520 P bc20559563fe44efa0a2b7b084aafbce: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240502 21:27:06.506294 18632 tablet_copy_service.cc:342] P 82f50f4b66f3450aa9f58a0b317f5cea: Request end of tablet copy session bc20559563fe44efa0a2b7b084aafbce-2d111096a9af46068a2476a6231d6520 received from {username='slave'} at 127.15.255.136:48763
I20240502 21:27:06.506420 18632 tablet_copy_service.cc:433] P 82f50f4b66f3450aa9f58a0b317f5cea: ending tablet copy session bc20559563fe44efa0a2b7b084aafbce-2d111096a9af46068a2476a6231d6520 on tablet 2d111096a9af46068a2476a6231d6520 with peer bc20559563fe44efa0a2b7b084aafbce
I20240502 21:27:06.876821 22632 raft_consensus.cc:1203] T 2d111096a9af46068a2476a6231d6520 P bc20559563fe44efa0a2b7b084aafbce [term 1 LEARNER]: Deduplicated request from leader. Original: 1.600->[1.601-1.601] Dedup: 1.601->[]
I20240502 21:27:07.387183 23517 raft_consensus.cc:1050] T 2d111096a9af46068a2476a6231d6520 P 82f50f4b66f3450aa9f58a0b317f5cea: attempting to promote NON_VOTER bc20559563fe44efa0a2b7b084aafbce to VOTER
I20240502 21:27:07.388044 23517 consensus_queue.cc:237] T 2d111096a9af46068a2476a6231d6520 P 82f50f4b66f3450aa9f58a0b317f5cea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 601, Committed index: 601, Last appended: 1.601, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 602 OBSOLETE_local: false peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } attrs { replace: true } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: false } }
I20240502 21:27:07.390967 18873 raft_consensus.cc:1261] T 2d111096a9af46068a2476a6231d6520 P c75831edb3b44dd59c00ef151adca49d [term 1 FOLLOWER]: Refusing update from remote peer 82f50f4b66f3450aa9f58a0b317f5cea: Log matching property violated. Preceding OpId in replica: term: 1 index: 601. Preceding OpId from leader: term: 1 index: 602. (index mismatch)
I20240502 21:27:07.391261 18744 raft_consensus.cc:1261] T 2d111096a9af46068a2476a6231d6520 P 501e3c0d44fe4d70af9ff926d9622417 [term 1 FOLLOWER]: Refusing update from remote peer 82f50f4b66f3450aa9f58a0b317f5cea: Log matching property violated. Preceding OpId in replica: term: 1 index: 601. Preceding OpId from leader: term: 1 index: 602. (index mismatch)
I20240502 21:27:07.391561 23467 consensus_queue.cc:1035] T 2d111096a9af46068a2476a6231d6520 P 82f50f4b66f3450aa9f58a0b317f5cea [LEADER]: Connected to new peer: Peer: permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 602, Last known committed idx: 601, Time since last communication: 0.000s
I20240502 21:27:07.391774 22632 raft_consensus.cc:1261] T 2d111096a9af46068a2476a6231d6520 P bc20559563fe44efa0a2b7b084aafbce [term 1 LEARNER]: Refusing update from remote peer 82f50f4b66f3450aa9f58a0b317f5cea: Log matching property violated. Preceding OpId in replica: term: 1 index: 601. Preceding OpId from leader: term: 1 index: 602. (index mismatch)
I20240502 21:27:07.392071 19146 raft_consensus.cc:1261] T 2d111096a9af46068a2476a6231d6520 P 2877108756304e05bd60ba1186b870da [term 1 FOLLOWER]: Refusing update from remote peer 82f50f4b66f3450aa9f58a0b317f5cea: Log matching property violated. Preceding OpId in replica: term: 1 index: 601. Preceding OpId from leader: term: 1 index: 602. (index mismatch)
I20240502 21:27:07.392334 23518 consensus_queue.cc:1035] T 2d111096a9af46068a2476a6231d6520 P 82f50f4b66f3450aa9f58a0b317f5cea [LEADER]: Connected to new peer: Peer: permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 602, Last known committed idx: 601, Time since last communication: 0.000s
I20240502 21:27:07.392778 23517 consensus_queue.cc:1035] T 2d111096a9af46068a2476a6231d6520 P 82f50f4b66f3450aa9f58a0b317f5cea [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 602, Last known committed idx: 601, Time since last communication: 0.000s
I20240502 21:27:07.393064 23518 consensus_queue.cc:1035] T 2d111096a9af46068a2476a6231d6520 P 82f50f4b66f3450aa9f58a0b317f5cea [LEADER]: Connected to new peer: Peer: permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 602, Last known committed idx: 601, Time since last communication: 0.000s
I20240502 21:27:07.394335 18473 raft_consensus.cc:1261] T 2d111096a9af46068a2476a6231d6520 P faadabfb099b4abf90a466395be585b5 [term 1 FOLLOWER]: Refusing update from remote peer 82f50f4b66f3450aa9f58a0b317f5cea: Log matching property violated. Preceding OpId in replica: term: 1 index: 601. Preceding OpId from leader: term: 1 index: 602. (index mismatch)
I20240502 21:27:07.395195 23517 consensus_queue.cc:1035] T 2d111096a9af46068a2476a6231d6520 P 82f50f4b66f3450aa9f58a0b317f5cea [LEADER]: Connected to new peer: Peer: permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 602, Last known committed idx: 601, Time since last communication: 0.000s
I20240502 21:27:07.396196 23517 raft_consensus.cc:2932] T 2d111096a9af46068a2476a6231d6520 P 82f50f4b66f3450aa9f58a0b317f5cea [term 1 LEADER]: Committing config change with OpId 1.602: config changed from index 601 to 602, bc20559563fe44efa0a2b7b084aafbce (127.15.255.136) changed from NON_VOTER to VOTER. New config: { opid_index: 602 OBSOLETE_local: false peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } attrs { replace: true } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: false } } }
I20240502 21:27:07.396829 19146 raft_consensus.cc:2932] T 2d111096a9af46068a2476a6231d6520 P 2877108756304e05bd60ba1186b870da [term 1 FOLLOWER]: Committing config change with OpId 1.602: config changed from index 601 to 602, bc20559563fe44efa0a2b7b084aafbce (127.15.255.136) changed from NON_VOTER to VOTER. New config: { opid_index: 602 OBSOLETE_local: false peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } attrs { replace: true } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: false } } }
I20240502 21:27:07.397490 22632 raft_consensus.cc:2932] T 2d111096a9af46068a2476a6231d6520 P bc20559563fe44efa0a2b7b084aafbce [term 1 FOLLOWER]: Committing config change with OpId 1.602: config changed from index 601 to 602, bc20559563fe44efa0a2b7b084aafbce (127.15.255.136) changed from NON_VOTER to VOTER. New config: { opid_index: 602 OBSOLETE_local: false peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } attrs { replace: true } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: false } } }
I20240502 21:27:07.402316 18320 catalog_manager.cc:5484] T 2d111096a9af46068a2476a6231d6520 P 2877108756304e05bd60ba1186b870da reported cstate change: config changed from index 601 to 602, bc20559563fe44efa0a2b7b084aafbce (127.15.255.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" committed_config { opid_index: 602 OBSOLETE_local: false peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } attrs { replace: true } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: false } } }
I20240502 21:27:07.402879 18744 raft_consensus.cc:2932] T 2d111096a9af46068a2476a6231d6520 P 501e3c0d44fe4d70af9ff926d9622417 [term 1 FOLLOWER]: Committing config change with OpId 1.602: config changed from index 601 to 602, bc20559563fe44efa0a2b7b084aafbce (127.15.255.136) changed from NON_VOTER to VOTER. New config: { opid_index: 602 OBSOLETE_local: false peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } attrs { replace: true } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: false } } }
I20240502 21:27:07.403079 18873 raft_consensus.cc:2932] T 2d111096a9af46068a2476a6231d6520 P c75831edb3b44dd59c00ef151adca49d [term 1 FOLLOWER]: Committing config change with OpId 1.602: config changed from index 601 to 602, bc20559563fe44efa0a2b7b084aafbce (127.15.255.136) changed from NON_VOTER to VOTER. New config: { opid_index: 602 OBSOLETE_local: false peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } attrs { replace: true } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: false } } }
I20240502 21:27:07.411331 18473 raft_consensus.cc:2932] T 2d111096a9af46068a2476a6231d6520 P faadabfb099b4abf90a466395be585b5 [term 1 FOLLOWER]: Committing config change with OpId 1.602: config changed from index 601 to 602, bc20559563fe44efa0a2b7b084aafbce (127.15.255.136) changed from NON_VOTER to VOTER. New config: { opid_index: 602 OBSOLETE_local: false peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } attrs { replace: true } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: false } } }
I20240502 21:27:07.415112 18611 consensus_queue.cc:237] T 2d111096a9af46068a2476a6231d6520 P 82f50f4b66f3450aa9f58a0b317f5cea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 602, Committed index: 602, Last appended: 1.602, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 603 OBSOLETE_local: false peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: false } }
I20240502 21:27:07.417114 22632 raft_consensus.cc:1261] T 2d111096a9af46068a2476a6231d6520 P bc20559563fe44efa0a2b7b084aafbce [term 1 FOLLOWER]: Refusing update from remote peer 82f50f4b66f3450aa9f58a0b317f5cea: Log matching property violated. Preceding OpId in replica: term: 1 index: 602. Preceding OpId from leader: term: 1 index: 603. (index mismatch)
I20240502 21:27:07.417397 19146 raft_consensus.cc:1261] T 2d111096a9af46068a2476a6231d6520 P 2877108756304e05bd60ba1186b870da [term 1 FOLLOWER]: Refusing update from remote peer 82f50f4b66f3450aa9f58a0b317f5cea: Log matching property violated. Preceding OpId in replica: term: 1 index: 602. Preceding OpId from leader: term: 1 index: 603. (index mismatch)
I20240502 21:27:07.418090 18744 raft_consensus.cc:1261] T 2d111096a9af46068a2476a6231d6520 P 501e3c0d44fe4d70af9ff926d9622417 [term 1 FOLLOWER]: Refusing update from remote peer 82f50f4b66f3450aa9f58a0b317f5cea: Log matching property violated. Preceding OpId in replica: term: 1 index: 602. Preceding OpId from leader: term: 1 index: 603. (index mismatch)
I20240502 21:27:07.418890 23518 consensus_queue.cc:1035] T 2d111096a9af46068a2476a6231d6520 P 82f50f4b66f3450aa9f58a0b317f5cea [LEADER]: Connected to new peer: Peer: permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 603, Last known committed idx: 602, Time since last communication: 0.000s
I20240502 21:27:07.419178 18873 raft_consensus.cc:1261] T 2d111096a9af46068a2476a6231d6520 P c75831edb3b44dd59c00ef151adca49d [term 1 FOLLOWER]: Refusing update from remote peer 82f50f4b66f3450aa9f58a0b317f5cea: Log matching property violated. Preceding OpId in replica: term: 1 index: 602. Preceding OpId from leader: term: 1 index: 603. (index mismatch)
I20240502 21:27:07.419317 23518 consensus_queue.cc:1035] T 2d111096a9af46068a2476a6231d6520 P 82f50f4b66f3450aa9f58a0b317f5cea [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 603, Last known committed idx: 602, Time since last communication: 0.000s
I20240502 21:27:07.419742 23518 consensus_queue.cc:1035] T 2d111096a9af46068a2476a6231d6520 P 82f50f4b66f3450aa9f58a0b317f5cea [LEADER]: Connected to new peer: Peer: permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 603, Last known committed idx: 602, Time since last communication: 0.000s
I20240502 21:27:07.420104 23518 consensus_queue.cc:1035] T 2d111096a9af46068a2476a6231d6520 P 82f50f4b66f3450aa9f58a0b317f5cea [LEADER]: Connected to new peer: Peer: permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 603, Last known committed idx: 602, Time since last communication: 0.000s
I20240502 21:27:07.423323 23518 raft_consensus.cc:2932] T 2d111096a9af46068a2476a6231d6520 P 82f50f4b66f3450aa9f58a0b317f5cea [term 1 LEADER]: Committing config change with OpId 1.603: config changed from index 602 to 603, VOTER faadabfb099b4abf90a466395be585b5 (127.15.255.129) evicted. New config: { opid_index: 603 OBSOLETE_local: false peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: false } } }
I20240502 21:27:07.423709 19146 raft_consensus.cc:2932] T 2d111096a9af46068a2476a6231d6520 P 2877108756304e05bd60ba1186b870da [term 1 FOLLOWER]: Committing config change with OpId 1.603: config changed from index 602 to 603, VOTER faadabfb099b4abf90a466395be585b5 (127.15.255.129) evicted. New config: { opid_index: 603 OBSOLETE_local: false peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: false } } }
I20240502 21:27:07.424247 18873 raft_consensus.cc:2932] T 2d111096a9af46068a2476a6231d6520 P c75831edb3b44dd59c00ef151adca49d [term 1 FOLLOWER]: Committing config change with OpId 1.603: config changed from index 602 to 603, VOTER faadabfb099b4abf90a466395be585b5 (127.15.255.129) evicted. New config: { opid_index: 603 OBSOLETE_local: false peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: false } } }
I20240502 21:27:07.427105 18315 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 2d111096a9af46068a2476a6231d6520 with cas_config_opid_index 602: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:27:07.428114 18744 raft_consensus.cc:2932] T 2d111096a9af46068a2476a6231d6520 P 501e3c0d44fe4d70af9ff926d9622417 [term 1 FOLLOWER]: Committing config change with OpId 1.603: config changed from index 602 to 603, VOTER faadabfb099b4abf90a466395be585b5 (127.15.255.129) evicted. New config: { opid_index: 603 OBSOLETE_local: false peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: false } } }
I20240502 21:27:07.429317 18319 catalog_manager.cc:5484] T 2d111096a9af46068a2476a6231d6520 P 2877108756304e05bd60ba1186b870da reported cstate change: config changed from index 602 to 603, VOTER faadabfb099b4abf90a466395be585b5 (127.15.255.129) evicted. New cstate: current_term: 1 leader_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" committed_config { opid_index: 603 OBSOLETE_local: false peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: false } } }
I20240502 21:27:07.433068 18451 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d111096a9af46068a2476a6231d6520 with delete_type TABLET_DATA_TOMBSTONED (TS faadabfb099b4abf90a466395be585b5 not found in new config with opid_index 603) from {username='slave'} at 127.0.0.1:39378
I20240502 21:27:07.433517 22632 raft_consensus.cc:2932] T 2d111096a9af46068a2476a6231d6520 P bc20559563fe44efa0a2b7b084aafbce [term 1 FOLLOWER]: Committing config change with OpId 1.603: config changed from index 602 to 603, VOTER faadabfb099b4abf90a466395be585b5 (127.15.255.129) evicted. New config: { opid_index: 603 OBSOLETE_local: false peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: false } } }
I20240502 21:27:07.444900 23530 tablet_replica.cc:330] T 2d111096a9af46068a2476a6231d6520 P faadabfb099b4abf90a466395be585b5: stopping tablet replica
I20240502 21:27:07.445123 23530 raft_consensus.cc:2229] T 2d111096a9af46068a2476a6231d6520 P faadabfb099b4abf90a466395be585b5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:07.445555 23530 raft_consensus.cc:2258] T 2d111096a9af46068a2476a6231d6520 P faadabfb099b4abf90a466395be585b5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:07.446995 23530 ts_tablet_manager.cc:1875] T 2d111096a9af46068a2476a6231d6520 P faadabfb099b4abf90a466395be585b5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:27:07.451651 23530 ts_tablet_manager.cc:1888] T 2d111096a9af46068a2476a6231d6520 P faadabfb099b4abf90a466395be585b5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.602
I20240502 21:27:07.451752 23530 log.cc:1196] T 2d111096a9af46068a2476a6231d6520 P faadabfb099b4abf90a466395be585b5: Deleting WAL directory at /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-0/wal/wals/2d111096a9af46068a2476a6231d6520
I20240502 21:27:07.452497 18313 catalog_manager.cc:4830] TS faadabfb099b4abf90a466395be585b5 (127.15.255.129:32983): tablet 2d111096a9af46068a2476a6231d6520 (table rebalance_test_table_2 [id=59b7e3f7d6ce4a008110576d81d3cf39]) successfully deleted
I20240502 21:27:07.641957 19124 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:27:07.663326 18590 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:27:07.665484 20147 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:07.667286 18989 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:27:07.691427 18722 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:27:07.706287 22613 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:27:07.760553 18857 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
W20240502 21:27:07.776675 18679 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.132:43607, user_credentials={real_user=slave}} blocked reactor thread for 61156.9us
I20240502 21:27:07.820333 18451 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:27:07.962416 19124 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:27:07.966353 18722 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:27:07.985134 18590 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:27:08.005821 20147 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:08.008765 18989 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:27:08.033357 22613 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:27:08.048929 18857 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:27:08.052810 18451 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:27:08.372954 19124 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:27:08.377539 18722 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:27:08.400985 18590 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:27:08.419977 20147 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:08.461843 22613 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:27:08.480387 18857 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:27:08.500666 18989 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:27:08.514479 18451 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
132c427694cc4f858dd86045029e179e | 127.15.255.190:36947 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
allow_unsafe_replication_factor | true | runtime,unsafe | all 1 server(s) checked
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714685120732928-16382-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.15.255.148:40185 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+---------+----------+----------------+-----------------
2877108756304e05bd60ba1186b870da | 127.15.255.134:43635 | HEALTHY | <none> | 1 | 0
501e3c0d44fe4d70af9ff926d9622417 | 127.15.255.131:42501 | HEALTHY | <none> | 1 | 0
82f50f4b66f3450aa9f58a0b317f5cea | 127.15.255.130:45139 | HEALTHY | <none> | 3 | 0
90e329b457c24caf93a9f880345eb288 | 127.15.255.135:43395 | HEALTHY | <none> | 0 | 0
9be399440ded4e2ebe6b6e0bf486cbb1 | 127.15.255.133:34217 | HEALTHY | <none> | 2 | 0
bc20559563fe44efa0a2b7b084aafbce | 127.15.255.136:34965 | HEALTHY | <none> | 1 | 0
c75831edb3b44dd59c00ef151adca49d | 127.15.255.132:43607 | HEALTHY | <none> | 3 | 0
faadabfb099b4abf90a466395be585b5 | 127.15.255.129:32983 | HEALTHY | <none> | 5 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 8
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 8 server(s) checked
local_ip_for_outbound_sockets | 127.15.255.129 | experimental | 127.15.255.129:32983
local_ip_for_outbound_sockets | 127.15.255.130 | experimental | 127.15.255.130:45139
local_ip_for_outbound_sockets | 127.15.255.131 | experimental | 127.15.255.131:42501
local_ip_for_outbound_sockets | 127.15.255.132 | experimental | 127.15.255.132:43607
local_ip_for_outbound_sockets | 127.15.255.133 | experimental | 127.15.255.133:34217
local_ip_for_outbound_sockets | 127.15.255.134 | experimental | 127.15.255.134:43635
local_ip_for_outbound_sockets | 127.15.255.135 | experimental | 127.15.255.135:43395
local_ip_for_outbound_sockets | 127.15.255.136 | experimental | 127.15.255.136:34965
never_fsync | true | unsafe,advanced | all 8 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 8 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 8 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 8 server(s) checked
server_dump_info_format | pb | hidden | all 8 server(s) checked
server_dump_info_path | /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.15.255.129:32983
server_dump_info_path | /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.15.255.130:45139
server_dump_info_path | /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.15.255.131:42501
server_dump_info_path | /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.15.255.132:43607
server_dump_info_path | /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.15.255.133:34217
server_dump_info_path | /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.15.255.134:43635
server_dump_info_path | /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-6/data/info.pb | hidden | 127.15.255.135:43395
server_dump_info_path | /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalanceParamTest.Rebalance_7.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-7/data/info.pb | hidden | 127.15.255.136:34965
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.15.255.148:40185 | all 8 server(s) checked
time_source | builtin | all 8 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 9 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------------------+----+---------+---------------+---------+------------+------------------+-------------
TestTable | 5 | HEALTHY | 1 | 1 | 0 | 0 | 0
rebalance_test_table_0 | 5 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_1 | 5 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_2 | 5 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_3 | 5 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_4 | 5 | HEALTHY | 3 | 3 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 10
First Quartile | 10
Median | 10
Third Quartile | 10
Maximum | 10
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 8
Tables | 6
Tablets | 16
Replicas | 80
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20240502 21:27:08.569860 16382 log_verifier.cc:126] Checking tablet 1df9d3ad5ddc438798543900b6c67b46
I20240502 21:27:09.021198 16382 log_verifier.cc:177] Verified matching terms for 504 ops in tablet 1df9d3ad5ddc438798543900b6c67b46
I20240502 21:27:09.023193 16382 log_verifier.cc:126] Checking tablet 2d111096a9af46068a2476a6231d6520
I20240502 21:27:09.579972 16382 log_verifier.cc:177] Verified matching terms for 603 ops in tablet 2d111096a9af46068a2476a6231d6520
I20240502 21:27:09.581357 16382 log_verifier.cc:126] Checking tablet 32c9fc5a18c54207881124fda86f42ff
I20240502 21:27:10.111364 16382 log_verifier.cc:177] Verified matching terms for 534 ops in tablet 32c9fc5a18c54207881124fda86f42ff
I20240502 21:27:10.112577 16382 log_verifier.cc:126] Checking tablet 37e9dcc6191c42d3a5c1503dedc5511e
I20240502 21:27:10.522272 16382 log_verifier.cc:177] Verified matching terms for 534 ops in tablet 37e9dcc6191c42d3a5c1503dedc5511e
I20240502 21:27:10.523475 16382 log_verifier.cc:126] Checking tablet 40d4f61d0f6d4d31a5626c7743c88133
I20240502 21:27:10.582904 16382 log_verifier.cc:177] Verified matching terms for 4 ops in tablet 40d4f61d0f6d4d31a5626c7743c88133
I20240502 21:27:10.582988 16382 log_verifier.cc:126] Checking tablet 4aff104e3b6b424baa628bbe612b62e8
I20240502 21:27:10.910032 16382 log_verifier.cc:177] Verified matching terms for 410 ops in tablet 4aff104e3b6b424baa628bbe612b62e8
I20240502 21:27:10.911007 16382 log_verifier.cc:126] Checking tablet 51edafdf2a9348548a494119668bfd31
I20240502 21:27:11.373687 16382 log_verifier.cc:177] Verified matching terms for 584 ops in tablet 51edafdf2a9348548a494119668bfd31
I20240502 21:27:11.374980 16382 log_verifier.cc:126] Checking tablet 693a8666bbe54ae1b774f7f4da5ba67e
I20240502 21:27:11.857656 16382 log_verifier.cc:177] Verified matching terms for 586 ops in tablet 693a8666bbe54ae1b774f7f4da5ba67e
I20240502 21:27:11.859122 16382 log_verifier.cc:126] Checking tablet 6f0107b0690c4c64b0a4f2472a1c8322
I20240502 21:27:12.242545 16382 log_verifier.cc:177] Verified matching terms for 500 ops in tablet 6f0107b0690c4c64b0a4f2472a1c8322
I20240502 21:27:12.243573 16382 log_verifier.cc:126] Checking tablet 84b235fd89634168938c0dddb534462a
I20240502 21:27:12.701103 16382 log_verifier.cc:177] Verified matching terms for 571 ops in tablet 84b235fd89634168938c0dddb534462a
I20240502 21:27:12.702493 16382 log_verifier.cc:126] Checking tablet 9844e983dd0a403abe0ac90cd96690d6
I20240502 21:27:13.066313 16382 log_verifier.cc:177] Verified matching terms for 408 ops in tablet 9844e983dd0a403abe0ac90cd96690d6
I20240502 21:27:13.067323 16382 log_verifier.cc:126] Checking tablet a7a9256f2b784abcb780b190e2ff3983
I20240502 21:27:13.469012 16382 log_verifier.cc:177] Verified matching terms for 513 ops in tablet a7a9256f2b784abcb780b190e2ff3983
I20240502 21:27:13.470144 16382 log_verifier.cc:126] Checking tablet b0852f54089944ce935f8ee4cbf2bc5a
I20240502 21:27:13.884953 16382 log_verifier.cc:177] Verified matching terms for 455 ops in tablet b0852f54089944ce935f8ee4cbf2bc5a
I20240502 21:27:13.886036 16382 log_verifier.cc:126] Checking tablet e0e69b88ad014363b97d1f650df693a8
I20240502 21:27:14.348852 16382 log_verifier.cc:177] Verified matching terms for 508 ops in tablet e0e69b88ad014363b97d1f650df693a8
I20240502 21:27:14.350272 16382 log_verifier.cc:126] Checking tablet ebf7371ff0ef4de99ced8fec0468af28
I20240502 21:27:14.736979 16382 log_verifier.cc:177] Verified matching terms for 389 ops in tablet ebf7371ff0ef4de99ced8fec0468af28
I20240502 21:27:14.737783 16382 log_verifier.cc:126] Checking tablet ed78d40318fe4ab5bee4a01b182a34b6
I20240502 21:27:15.174664 16382 log_verifier.cc:177] Verified matching terms for 501 ops in tablet ed78d40318fe4ab5bee4a01b182a34b6
I20240502 21:27:15.181522 16382 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvNciQZ/build/asan/bin/kudu pid 18389
I20240502 21:27:15.192351 18519 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 21:27:15.504150 16382 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvNciQZ/build/asan/bin/kudu with pid 18389
W20240502 21:27:15.548313 18312 connection.cc:535] server connection from 127.15.255.129:53125 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 21:27:15.548477 18544 connection.cc:535] server connection from 127.15.255.129:48059 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 21:27:15.548352 19076 connection.cc:535] server connection from 127.15.255.129:40621 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 21:27:15.548940 16382 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvNciQZ/build/asan/bin/kudu pid 18528
I20240502 21:27:15.561043 18653 generic_service.cc:196] Checking for leaks (request via RPC)
W20240502 21:27:15.646199 18810 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.255.129:32983: connect: Connection refused (error 111)
W20240502 21:27:15.647485 18810 consensus_peers.cc:485] T 37e9dcc6191c42d3a5c1503dedc5511e P c75831edb3b44dd59c00ef151adca49d -> Peer faadabfb099b4abf90a466395be585b5 (127.15.255.129:32983): Couldn't send request to peer faadabfb099b4abf90a466395be585b5. Status: Network error: Client connection negotiation failed: client connection to 127.15.255.129:32983: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:27:15.654101 18944 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.255.129:32983: connect: Connection refused (error 111)
W20240502 21:27:15.655366 18944 consensus_peers.cc:485] T 4aff104e3b6b424baa628bbe612b62e8 P 9be399440ded4e2ebe6b6e0bf486cbb1 -> Peer faadabfb099b4abf90a466395be585b5 (127.15.255.129:32983): Couldn't send request to peer faadabfb099b4abf90a466395be585b5. Status: Network error: Client connection negotiation failed: client connection to 127.15.255.129:32983: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:27:15.852607 18678 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.255.129:32983: connect: Connection refused (error 111)
W20240502 21:27:15.857074 18678 consensus_peers.cc:485] T 1df9d3ad5ddc438798543900b6c67b46 P 501e3c0d44fe4d70af9ff926d9622417 -> Peer faadabfb099b4abf90a466395be585b5 (127.15.255.129:32983): Couldn't send request to peer faadabfb099b4abf90a466395be585b5. Status: Network error: Client connection negotiation failed: client connection to 127.15.255.129:32983: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:15.858222 16382 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvNciQZ/build/asan/bin/kudu with pid 18528
W20240502 21:27:15.900825 18812 connection.cc:535] server connection from 127.15.255.130:48719 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 21:27:15.900966 18946 connection.cc:535] client connection to 127.15.255.130:45139 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 21:27:15.901037 18812 connection.cc:535] client connection to 127.15.255.130:45139 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 21:27:15.901698 16382 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvNciQZ/build/asan/bin/kudu pid 18661
W20240502 21:27:15.902218 18946 consensus_peers.cc:485] T 4aff104e3b6b424baa628bbe612b62e8 P 9be399440ded4e2ebe6b6e0bf486cbb1 -> Peer 82f50f4b66f3450aa9f58a0b317f5cea (127.15.255.130:45139): Couldn't send request to peer 82f50f4b66f3450aa9f58a0b317f5cea. Status: Network error: Client connection negotiation failed: client connection to 127.15.255.130:45139: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:27:15.902246 18812 consensus_peers.cc:485] T 51edafdf2a9348548a494119668bfd31 P c75831edb3b44dd59c00ef151adca49d -> Peer 82f50f4b66f3450aa9f58a0b317f5cea (127.15.255.130:45139): Couldn't send request to peer 82f50f4b66f3450aa9f58a0b317f5cea. Status: Network error: Client connection negotiation failed: client connection to 127.15.255.130:45139: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:15.907090 18786 generic_service.cc:196] Checking for leaks (request via RPC)
W20240502 21:27:16.101127 19079 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.255.130:45139: connect: Connection refused (error 111)
W20240502 21:27:16.102933 19079 consensus_peers.cc:485] T b0852f54089944ce935f8ee4cbf2bc5a P 2877108756304e05bd60ba1186b870da -> Peer 82f50f4b66f3450aa9f58a0b317f5cea (127.15.255.130:45139): Couldn't send request to peer 82f50f4b66f3450aa9f58a0b317f5cea. Status: Network error: Client connection negotiation failed: client connection to 127.15.255.130:45139: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:16.182614 16382 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvNciQZ/build/asan/bin/kudu with pid 18661
I20240502 21:27:16.198032 23684 raft_consensus.cc:482] T 6f0107b0690c4c64b0a4f2472a1c8322 P c75831edb3b44dd59c00ef151adca49d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader faadabfb099b4abf90a466395be585b5)
I20240502 21:27:16.198208 23684 raft_consensus.cc:504] T 6f0107b0690c4c64b0a4f2472a1c8322 P c75831edb3b44dd59c00ef151adca49d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 134 OBSOLETE_local: false peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } }
I20240502 21:27:16.198995 23684 leader_election.cc:288] T 6f0107b0690c4c64b0a4f2472a1c8322 P c75831edb3b44dd59c00ef151adca49d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 82f50f4b66f3450aa9f58a0b317f5cea (127.15.255.130:45139), faadabfb099b4abf90a466395be585b5 (127.15.255.129:32983), 9be399440ded4e2ebe6b6e0bf486cbb1 (127.15.255.133:34217), 90e329b457c24caf93a9f880345eb288 (127.15.255.135:43395)
I20240502 21:27:16.199843 20168 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f0107b0690c4c64b0a4f2472a1c8322" candidate_uuid: "c75831edb3b44dd59c00ef151adca49d" candidate_term: 2 candidate_status { last_received { term: 1 index: 500 } } ignore_live_leader: false dest_uuid: "90e329b457c24caf93a9f880345eb288" is_pre_election: true
I20240502 21:27:16.200639 19009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f0107b0690c4c64b0a4f2472a1c8322" candidate_uuid: "c75831edb3b44dd59c00ef151adca49d" candidate_term: 2 candidate_status { last_received { term: 1 index: 500 } } ignore_live_leader: false dest_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" is_pre_election: true
W20240502 21:27:16.200994 18812 leader_election.cc:334] T 6f0107b0690c4c64b0a4f2472a1c8322 P c75831edb3b44dd59c00ef151adca49d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 82f50f4b66f3450aa9f58a0b317f5cea (127.15.255.130:45139): Network error: Client connection negotiation failed: client connection to 127.15.255.130:45139: connect: Connection refused (error 111)
I20240502 21:27:16.201248 18810 leader_election.cc:302] T 6f0107b0690c4c64b0a4f2472a1c8322 P c75831edb3b44dd59c00ef151adca49d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 5 voters: 1 yes votes; 3 no votes. yes voters: c75831edb3b44dd59c00ef151adca49d; no voters: 82f50f4b66f3450aa9f58a0b317f5cea, 90e329b457c24caf93a9f880345eb288, 9be399440ded4e2ebe6b6e0bf486cbb1
W20240502 21:27:16.201483 18810 leader_election.cc:334] T 6f0107b0690c4c64b0a4f2472a1c8322 P c75831edb3b44dd59c00ef151adca49d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer faadabfb099b4abf90a466395be585b5 (127.15.255.129:32983): Network error: Client connection negotiation failed: client connection to 127.15.255.129:32983: connect: Connection refused (error 111)
I20240502 21:27:16.201490 23684 raft_consensus.cc:2726] T 6f0107b0690c4c64b0a4f2472a1c8322 P c75831edb3b44dd59c00ef151adca49d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 21:27:16.217140 23690 raft_consensus.cc:482] T 84b235fd89634168938c0dddb534462a P 90e329b457c24caf93a9f880345eb288 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader faadabfb099b4abf90a466395be585b5)
I20240502 21:27:16.217342 23690 raft_consensus.cc:504] T 84b235fd89634168938c0dddb534462a P 90e329b457c24caf93a9f880345eb288 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 134 OBSOLETE_local: false peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } }
I20240502 21:27:16.219218 19140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84b235fd89634168938c0dddb534462a" candidate_uuid: "90e329b457c24caf93a9f880345eb288" candidate_term: 2 candidate_status { last_received { term: 1 index: 571 } } ignore_live_leader: false dest_uuid: "2877108756304e05bd60ba1186b870da" is_pre_election: true
I20240502 21:27:16.219303 23690 leader_election.cc:288] T 84b235fd89634168938c0dddb534462a P 90e329b457c24caf93a9f880345eb288 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2877108756304e05bd60ba1186b870da (127.15.255.134:43635), 9be399440ded4e2ebe6b6e0bf486cbb1 (127.15.255.133:34217), faadabfb099b4abf90a466395be585b5 (127.15.255.129:32983), 82f50f4b66f3450aa9f58a0b317f5cea (127.15.255.130:45139)
I20240502 21:27:16.219300 19009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84b235fd89634168938c0dddb534462a" candidate_uuid: "90e329b457c24caf93a9f880345eb288" candidate_term: 2 candidate_status { last_received { term: 1 index: 571 } } ignore_live_leader: false dest_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" is_pre_election: true
W20240502 21:27:16.220373 20098 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.255.130:45139: connect: Connection refused (error 111)
W20240502 21:27:16.221768 20098 leader_election.cc:334] T 84b235fd89634168938c0dddb534462a P 90e329b457c24caf93a9f880345eb288 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 82f50f4b66f3450aa9f58a0b317f5cea (127.15.255.130:45139): Network error: Client connection negotiation failed: client connection to 127.15.255.130:45139: connect: Connection refused (error 111)
I20240502 21:27:16.221891 20098 leader_election.cc:302] T 84b235fd89634168938c0dddb534462a P 90e329b457c24caf93a9f880345eb288 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 5 voters: 1 yes votes; 3 no votes. yes voters: 90e329b457c24caf93a9f880345eb288; no voters: 2877108756304e05bd60ba1186b870da, 82f50f4b66f3450aa9f58a0b317f5cea, 9be399440ded4e2ebe6b6e0bf486cbb1
W20240502 21:27:16.222083 20096 leader_election.cc:334] T 84b235fd89634168938c0dddb534462a P 90e329b457c24caf93a9f880345eb288 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer faadabfb099b4abf90a466395be585b5 (127.15.255.129:32983): Network error: Client connection negotiation failed: client connection to 127.15.255.129:32983: connect: Connection refused (error 111)
I20240502 21:27:16.222182 23690 raft_consensus.cc:2726] T 84b235fd89634168938c0dddb534462a P 90e329b457c24caf93a9f880345eb288 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 21:27:16.227707 18943 connection.cc:535] client connection to 127.15.255.131:42501 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 21:27:16.228596 16382 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvNciQZ/build/asan/bin/kudu pid 18794
W20240502 21:27:16.228655 18943 consensus_peers.cc:485] T e0e69b88ad014363b97d1f650df693a8 P 9be399440ded4e2ebe6b6e0bf486cbb1 -> Peer 501e3c0d44fe4d70af9ff926d9622417 (127.15.255.131:42501): Couldn't send request to peer 501e3c0d44fe4d70af9ff926d9622417. Status: Network error: Client connection negotiation failed: client connection to 127.15.255.131:42501: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:16.233927 18919 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 21:27:16.241920 23690 raft_consensus.cc:482] T 32c9fc5a18c54207881124fda86f42ff P 90e329b457c24caf93a9f880345eb288 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader faadabfb099b4abf90a466395be585b5)
I20240502 21:27:16.242048 23690 raft_consensus.cc:504] T 32c9fc5a18c54207881124fda86f42ff P 90e329b457c24caf93a9f880345eb288 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 228 OBSOLETE_local: false peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } }
I20240502 21:27:16.242838 23690 leader_election.cc:288] T 32c9fc5a18c54207881124fda86f42ff P 90e329b457c24caf93a9f880345eb288 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c75831edb3b44dd59c00ef151adca49d (127.15.255.132:43607), 82f50f4b66f3450aa9f58a0b317f5cea (127.15.255.130:45139), faadabfb099b4abf90a466395be585b5 (127.15.255.129:32983), 501e3c0d44fe4d70af9ff926d9622417 (127.15.255.131:42501)
W20240502 21:27:16.244057 20098 leader_election.cc:334] T 32c9fc5a18c54207881124fda86f42ff P 90e329b457c24caf93a9f880345eb288 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 82f50f4b66f3450aa9f58a0b317f5cea (127.15.255.130:45139): Network error: Client connection negotiation failed: client connection to 127.15.255.130:45139: connect: Connection refused (error 111)
W20240502 21:27:16.244361 20094 leader_election.cc:334] T 32c9fc5a18c54207881124fda86f42ff P 90e329b457c24caf93a9f880345eb288 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 501e3c0d44fe4d70af9ff926d9622417 (127.15.255.131:42501): Network error: Client connection negotiation failed: client connection to 127.15.255.131:42501: connect: Connection refused (error 111)
W20240502 21:27:16.244488 20096 leader_election.cc:334] T 32c9fc5a18c54207881124fda86f42ff P 90e329b457c24caf93a9f880345eb288 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer faadabfb099b4abf90a466395be585b5 (127.15.255.129:32983): Network error: Client connection negotiation failed: client connection to 127.15.255.129:32983: connect: Connection refused (error 111)
I20240502 21:27:16.244563 20096 leader_election.cc:302] T 32c9fc5a18c54207881124fda86f42ff P 90e329b457c24caf93a9f880345eb288 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 5 voters: 1 yes votes; 3 no votes. yes voters: 90e329b457c24caf93a9f880345eb288; no voters: 501e3c0d44fe4d70af9ff926d9622417, 82f50f4b66f3450aa9f58a0b317f5cea, faadabfb099b4abf90a466395be585b5
I20240502 21:27:16.244758 23690 raft_consensus.cc:2726] T 32c9fc5a18c54207881124fda86f42ff P 90e329b457c24caf93a9f880345eb288 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 21:27:16.439152 22566 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.255.131:42501: connect: Connection refused (error 111)
W20240502 21:27:16.440691 22566 consensus_peers.cc:485] T ed78d40318fe4ab5bee4a01b182a34b6 P bc20559563fe44efa0a2b7b084aafbce -> Peer 501e3c0d44fe4d70af9ff926d9622417 (127.15.255.131:42501): Couldn't send request to peer 501e3c0d44fe4d70af9ff926d9622417. Status: Network error: Client connection negotiation failed: client connection to 127.15.255.131:42501: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:27:16.455649 19076 consensus_peers.cc:485] T b0852f54089944ce935f8ee4cbf2bc5a P 2877108756304e05bd60ba1186b870da -> Peer 501e3c0d44fe4d70af9ff926d9622417 (127.15.255.131:42501): Couldn't send request to peer 501e3c0d44fe4d70af9ff926d9622417. Status: Network error: Client connection negotiation failed: client connection to 127.15.255.131:42501: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:16.489176 23685 raft_consensus.cc:482] T 40d4f61d0f6d4d31a5626c7743c88133 P 9be399440ded4e2ebe6b6e0bf486cbb1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader faadabfb099b4abf90a466395be585b5)
I20240502 21:27:16.489391 23685 raft_consensus.cc:504] T 40d4f61d0f6d4d31a5626c7743c88133 P 9be399440ded4e2ebe6b6e0bf486cbb1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: false } }
I20240502 21:27:16.490370 23685 leader_election.cc:288] T 40d4f61d0f6d4d31a5626c7743c88133 P 9be399440ded4e2ebe6b6e0bf486cbb1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 82f50f4b66f3450aa9f58a0b317f5cea (127.15.255.130:45139), 90e329b457c24caf93a9f880345eb288 (127.15.255.135:43395), faadabfb099b4abf90a466395be585b5 (127.15.255.129:32983), bc20559563fe44efa0a2b7b084aafbce (127.15.255.136:34965)
I20240502 21:27:16.491061 20168 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40d4f61d0f6d4d31a5626c7743c88133" candidate_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "90e329b457c24caf93a9f880345eb288" is_pre_election: true
W20240502 21:27:16.492242 18946 leader_election.cc:334] T 40d4f61d0f6d4d31a5626c7743c88133 P 9be399440ded4e2ebe6b6e0bf486cbb1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 82f50f4b66f3450aa9f58a0b317f5cea (127.15.255.130:45139): Network error: Client connection negotiation failed: client connection to 127.15.255.130:45139: connect: Connection refused (error 111)
W20240502 21:27:16.495535 18944 leader_election.cc:334] T 40d4f61d0f6d4d31a5626c7743c88133 P 9be399440ded4e2ebe6b6e0bf486cbb1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer faadabfb099b4abf90a466395be585b5 (127.15.255.129:32983): Network error: Client connection negotiation failed: client connection to 127.15.255.129:32983: connect: Connection refused (error 111)
I20240502 21:27:16.495648 18944 leader_election.cc:302] T 40d4f61d0f6d4d31a5626c7743c88133 P 9be399440ded4e2ebe6b6e0bf486cbb1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 5 voters: 1 yes votes; 3 no votes. yes voters: 9be399440ded4e2ebe6b6e0bf486cbb1; no voters: 82f50f4b66f3450aa9f58a0b317f5cea, 90e329b457c24caf93a9f880345eb288, faadabfb099b4abf90a466395be585b5
I20240502 21:27:16.495920 23685 raft_consensus.cc:2726] T 40d4f61d0f6d4d31a5626c7743c88133 P 9be399440ded4e2ebe6b6e0bf486cbb1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 21:27:16.498680 22631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40d4f61d0f6d4d31a5626c7743c88133" candidate_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "bc20559563fe44efa0a2b7b084aafbce" is_pre_election: true
I20240502 21:27:16.522550 18873 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32c9fc5a18c54207881124fda86f42ff" candidate_uuid: "90e329b457c24caf93a9f880345eb288" candidate_term: 2 candidate_status { last_received { term: 1 index: 534 } } ignore_live_leader: false dest_uuid: "c75831edb3b44dd59c00ef151adca49d" is_pre_election: true
I20240502 21:27:16.522787 18873 raft_consensus.cc:2454] T 32c9fc5a18c54207881124fda86f42ff P c75831edb3b44dd59c00ef151adca49d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90e329b457c24caf93a9f880345eb288 in term 1.
I20240502 21:27:16.523350 16382 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvNciQZ/build/asan/bin/kudu with pid 18794
I20240502 21:27:16.554143 23685 raft_consensus.cc:482] T 693a8666bbe54ae1b774f7f4da5ba67e P 9be399440ded4e2ebe6b6e0bf486cbb1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader faadabfb099b4abf90a466395be585b5)
I20240502 21:27:16.554385 23685 raft_consensus.cc:504] T 693a8666bbe54ae1b774f7f4da5ba67e P 9be399440ded4e2ebe6b6e0bf486cbb1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 586 OBSOLETE_local: false peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: false } }
I20240502 21:27:16.555446 23685 leader_election.cc:288] T 693a8666bbe54ae1b774f7f4da5ba67e P 9be399440ded4e2ebe6b6e0bf486cbb1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 501e3c0d44fe4d70af9ff926d9622417 (127.15.255.131:42501), faadabfb099b4abf90a466395be585b5 (127.15.255.129:32983), 90e329b457c24caf93a9f880345eb288 (127.15.255.135:43395), bc20559563fe44efa0a2b7b084aafbce (127.15.255.136:34965)
I20240502 21:27:16.556017 20168 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "693a8666bbe54ae1b774f7f4da5ba67e" candidate_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" candidate_term: 2 candidate_status { last_received { term: 1 index: 586 } } ignore_live_leader: false dest_uuid: "90e329b457c24caf93a9f880345eb288" is_pre_election: true
I20240502 21:27:16.556767 22631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "693a8666bbe54ae1b774f7f4da5ba67e" candidate_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" candidate_term: 2 candidate_status { last_received { term: 1 index: 586 } } ignore_live_leader: false dest_uuid: "bc20559563fe44efa0a2b7b084aafbce" is_pre_election: true
W20240502 21:27:16.557396 18943 leader_election.cc:334] T 693a8666bbe54ae1b774f7f4da5ba67e P 9be399440ded4e2ebe6b6e0bf486cbb1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 501e3c0d44fe4d70af9ff926d9622417 (127.15.255.131:42501): Network error: Client connection negotiation failed: client connection to 127.15.255.131:42501: connect: Connection refused (error 111)
I20240502 21:27:16.557534 18943 leader_election.cc:302] T 693a8666bbe54ae1b774f7f4da5ba67e P 9be399440ded4e2ebe6b6e0bf486cbb1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 5 voters: 1 yes votes; 3 no votes. yes voters: 9be399440ded4e2ebe6b6e0bf486cbb1; no voters: 501e3c0d44fe4d70af9ff926d9622417, 90e329b457c24caf93a9f880345eb288, bc20559563fe44efa0a2b7b084aafbce
W20240502 21:27:16.557716 18944 leader_election.cc:334] T 693a8666bbe54ae1b774f7f4da5ba67e P 9be399440ded4e2ebe6b6e0bf486cbb1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer faadabfb099b4abf90a466395be585b5 (127.15.255.129:32983): Network error: Client connection negotiation failed: client connection to 127.15.255.129:32983: connect: Connection refused (error 111)
I20240502 21:27:16.557822 23685 raft_consensus.cc:2726] T 693a8666bbe54ae1b774f7f4da5ba67e P 9be399440ded4e2ebe6b6e0bf486cbb1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 21:27:16.568652 20096 connection.cc:535] server connection from 127.15.255.132:37735 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 21:27:16.568792 19076 connection.cc:535] server connection from 127.15.255.132:40959 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 21:27:16.569109 18312 connection.cc:535] server connection from 127.15.255.132:34713 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 21:27:16.569679 16382 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvNciQZ/build/asan/bin/kudu pid 18927
W20240502 21:27:16.570068 22568 consensus_peers.cc:485] T ed78d40318fe4ab5bee4a01b182a34b6 P bc20559563fe44efa0a2b7b084aafbce -> Peer c75831edb3b44dd59c00ef151adca49d (127.15.255.132:43607): Couldn't send request to peer c75831edb3b44dd59c00ef151adca49d. Status: Network error: Client connection negotiation failed: client connection to 127.15.255.132:43607: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:27:16.570058 20097 leader_election.cc:334] T 32c9fc5a18c54207881124fda86f42ff P 90e329b457c24caf93a9f880345eb288 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c75831edb3b44dd59c00ef151adca49d (127.15.255.132:43607): Network error: Client connection negotiation failed: client connection to 127.15.255.132:43607: connect: Connection refused (error 111)
I20240502 21:27:16.575182 19052 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 21:27:16.652935 23690 raft_consensus.cc:482] T 40d4f61d0f6d4d31a5626c7743c88133 P 90e329b457c24caf93a9f880345eb288 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader faadabfb099b4abf90a466395be585b5)
I20240502 21:27:16.653101 23690 raft_consensus.cc:504] T 40d4f61d0f6d4d31a5626c7743c88133 P 90e329b457c24caf93a9f880345eb288 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: false } }
I20240502 21:27:16.653944 23690 leader_election.cc:288] T 40d4f61d0f6d4d31a5626c7743c88133 P 90e329b457c24caf93a9f880345eb288 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 82f50f4b66f3450aa9f58a0b317f5cea (127.15.255.130:45139), faadabfb099b4abf90a466395be585b5 (127.15.255.129:32983), 9be399440ded4e2ebe6b6e0bf486cbb1 (127.15.255.133:34217), bc20559563fe44efa0a2b7b084aafbce (127.15.255.136:34965)
W20240502 21:27:16.655540 20096 leader_election.cc:334] T 40d4f61d0f6d4d31a5626c7743c88133 P 90e329b457c24caf93a9f880345eb288 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer faadabfb099b4abf90a466395be585b5 (127.15.255.129:32983): Network error: Client connection negotiation failed: client connection to 127.15.255.129:32983: connect: Connection refused (error 111)
W20240502 21:27:16.655673 20098 leader_election.cc:334] T 40d4f61d0f6d4d31a5626c7743c88133 P 90e329b457c24caf93a9f880345eb288 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 82f50f4b66f3450aa9f58a0b317f5cea (127.15.255.130:45139): Network error: Client connection negotiation failed: client connection to 127.15.255.130:45139: connect: Connection refused (error 111)
I20240502 21:27:16.659457 22631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40d4f61d0f6d4d31a5626c7743c88133" candidate_uuid: "90e329b457c24caf93a9f880345eb288" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "bc20559563fe44efa0a2b7b084aafbce" is_pre_election: true
I20240502 21:27:16.659926 20094 leader_election.cc:302] T 40d4f61d0f6d4d31a5626c7743c88133 P 90e329b457c24caf93a9f880345eb288 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 5 voters: 1 yes votes; 3 no votes. yes voters: 90e329b457c24caf93a9f880345eb288; no voters: 82f50f4b66f3450aa9f58a0b317f5cea, bc20559563fe44efa0a2b7b084aafbce, faadabfb099b4abf90a466395be585b5
I20240502 21:27:16.660214 23690 raft_consensus.cc:2726] T 40d4f61d0f6d4d31a5626c7743c88133 P 90e329b457c24caf93a9f880345eb288 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 21:27:16.704958 23646 raft_consensus.cc:482] T a7a9256f2b784abcb780b190e2ff3983 P bc20559563fe44efa0a2b7b084aafbce [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 82f50f4b66f3450aa9f58a0b317f5cea)
I20240502 21:27:16.705185 23646 raft_consensus.cc:504] T a7a9256f2b784abcb780b190e2ff3983 P bc20559563fe44efa0a2b7b084aafbce [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 513 OBSOLETE_local: false peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: false } }
I20240502 21:27:16.706640 23646 leader_election.cc:288] T a7a9256f2b784abcb780b190e2ff3983 P bc20559563fe44efa0a2b7b084aafbce [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 501e3c0d44fe4d70af9ff926d9622417 (127.15.255.131:42501), faadabfb099b4abf90a466395be585b5 (127.15.255.129:32983), 82f50f4b66f3450aa9f58a0b317f5cea (127.15.255.130:45139), 90e329b457c24caf93a9f880345eb288 (127.15.255.135:43395)
W20240502 21:27:16.708907 22566 leader_election.cc:334] T a7a9256f2b784abcb780b190e2ff3983 P bc20559563fe44efa0a2b7b084aafbce [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 501e3c0d44fe4d70af9ff926d9622417 (127.15.255.131:42501): Network error: Client connection negotiation failed: client connection to 127.15.255.131:42501: connect: Connection refused (error 111)
W20240502 21:27:16.709426 22567 leader_election.cc:334] T a7a9256f2b784abcb780b190e2ff3983 P bc20559563fe44efa0a2b7b084aafbce [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer faadabfb099b4abf90a466395be585b5 (127.15.255.129:32983): Network error: Client connection negotiation failed: client connection to 127.15.255.129:32983: connect: Connection refused (error 111)
W20240502 21:27:16.709626 22569 leader_election.cc:334] T a7a9256f2b784abcb780b190e2ff3983 P bc20559563fe44efa0a2b7b084aafbce [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 82f50f4b66f3450aa9f58a0b317f5cea (127.15.255.130:45139): Network error: Client connection negotiation failed: client connection to 127.15.255.130:45139: connect: Connection refused (error 111)
I20240502 21:27:16.709759 22569 leader_election.cc:302] T a7a9256f2b784abcb780b190e2ff3983 P bc20559563fe44efa0a2b7b084aafbce [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 5 voters: 1 yes votes; 3 no votes. yes voters: bc20559563fe44efa0a2b7b084aafbce; no voters: 501e3c0d44fe4d70af9ff926d9622417, 82f50f4b66f3450aa9f58a0b317f5cea, faadabfb099b4abf90a466395be585b5
I20240502 21:27:16.710060 23646 raft_consensus.cc:2726] T a7a9256f2b784abcb780b190e2ff3983 P bc20559563fe44efa0a2b7b084aafbce [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 21:27:16.714716 20168 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7a9256f2b784abcb780b190e2ff3983" candidate_uuid: "bc20559563fe44efa0a2b7b084aafbce" candidate_term: 2 candidate_status { last_received { term: 1 index: 513 } } ignore_live_leader: false dest_uuid: "90e329b457c24caf93a9f880345eb288" is_pre_election: true
I20240502 21:27:16.868710 23685 raft_consensus.cc:482] T 6f0107b0690c4c64b0a4f2472a1c8322 P 9be399440ded4e2ebe6b6e0bf486cbb1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader faadabfb099b4abf90a466395be585b5)
I20240502 21:27:16.868953 23685 raft_consensus.cc:504] T 6f0107b0690c4c64b0a4f2472a1c8322 P 9be399440ded4e2ebe6b6e0bf486cbb1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 134 OBSOLETE_local: false peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } }
I20240502 21:27:16.870000 23685 leader_election.cc:288] T 6f0107b0690c4c64b0a4f2472a1c8322 P 9be399440ded4e2ebe6b6e0bf486cbb1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 82f50f4b66f3450aa9f58a0b317f5cea (127.15.255.130:45139), c75831edb3b44dd59c00ef151adca49d (127.15.255.132:43607), faadabfb099b4abf90a466395be585b5 (127.15.255.129:32983), 90e329b457c24caf93a9f880345eb288 (127.15.255.135:43395)
I20240502 21:27:16.870471 19009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40d4f61d0f6d4d31a5626c7743c88133" candidate_uuid: "90e329b457c24caf93a9f880345eb288" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" is_pre_election: true
I20240502 21:27:16.870484 20167 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f0107b0690c4c64b0a4f2472a1c8322" candidate_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" candidate_term: 2 candidate_status { last_received { term: 1 index: 500 } } ignore_live_leader: false dest_uuid: "90e329b457c24caf93a9f880345eb288" is_pre_election: true
I20240502 21:27:16.870692 19009 raft_consensus.cc:2454] T 40d4f61d0f6d4d31a5626c7743c88133 P 9be399440ded4e2ebe6b6e0bf486cbb1 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90e329b457c24caf93a9f880345eb288 in term 1.
I20240502 21:27:16.871071 16382 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvNciQZ/build/asan/bin/kudu with pid 18927
I20240502 21:27:16.896399 23711 raft_consensus.cc:482] T 2d111096a9af46068a2476a6231d6520 P bc20559563fe44efa0a2b7b084aafbce [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 82f50f4b66f3450aa9f58a0b317f5cea)
I20240502 21:27:16.896652 23711 raft_consensus.cc:504] T 2d111096a9af46068a2476a6231d6520 P bc20559563fe44efa0a2b7b084aafbce [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 603 OBSOLETE_local: false peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: false } }
I20240502 21:27:16.898461 23711 leader_election.cc:288] T 2d111096a9af46068a2476a6231d6520 P bc20559563fe44efa0a2b7b084aafbce [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 82f50f4b66f3450aa9f58a0b317f5cea (127.15.255.130:45139), 2877108756304e05bd60ba1186b870da (127.15.255.134:43635), 501e3c0d44fe4d70af9ff926d9622417 (127.15.255.131:42501), c75831edb3b44dd59c00ef151adca49d (127.15.255.132:43607)
I20240502 21:27:16.898598 19140 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d111096a9af46068a2476a6231d6520" candidate_uuid: "bc20559563fe44efa0a2b7b084aafbce" candidate_term: 2 candidate_status { last_received { term: 1 index: 603 } } ignore_live_leader: false dest_uuid: "2877108756304e05bd60ba1186b870da" is_pre_election: true
W20240502 21:27:16.899479 22566 leader_election.cc:334] T 2d111096a9af46068a2476a6231d6520 P bc20559563fe44efa0a2b7b084aafbce [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 501e3c0d44fe4d70af9ff926d9622417 (127.15.255.131:42501): Network error: Client connection negotiation failed: client connection to 127.15.255.131:42501: connect: Connection refused (error 111)
W20240502 21:27:16.899703 22568 leader_election.cc:334] T 2d111096a9af46068a2476a6231d6520 P bc20559563fe44efa0a2b7b084aafbce [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c75831edb3b44dd59c00ef151adca49d (127.15.255.132:43607): Network error: Client connection negotiation failed: client connection to 127.15.255.132:43607: connect: Connection refused (error 111)
I20240502 21:27:16.899849 22568 leader_election.cc:302] T 2d111096a9af46068a2476a6231d6520 P bc20559563fe44efa0a2b7b084aafbce [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 5 voters: 1 yes votes; 3 no votes. yes voters: bc20559563fe44efa0a2b7b084aafbce; no voters: 2877108756304e05bd60ba1186b870da, 501e3c0d44fe4d70af9ff926d9622417, c75831edb3b44dd59c00ef151adca49d
W20240502 21:27:16.900081 22569 leader_election.cc:334] T 2d111096a9af46068a2476a6231d6520 P bc20559563fe44efa0a2b7b084aafbce [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 82f50f4b66f3450aa9f58a0b317f5cea (127.15.255.130:45139): Network error: Client connection negotiation failed: client connection to 127.15.255.130:45139: connect: Connection refused (error 111)
I20240502 21:27:16.900135 23711 raft_consensus.cc:2726] T 2d111096a9af46068a2476a6231d6520 P bc20559563fe44efa0a2b7b084aafbce [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 21:27:16.915946 20097 connection.cc:535] server connection from 127.15.255.133:53733 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 21:27:16.917088 23691 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 21:27:16.916576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.15.255.133:34217 (local address 127.15.255.135:38439)
0502 21:27:16.916812 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0502 21:27:16.916884 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.15.255.133:34217: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":140}
I20240502 21:27:16.917133 16382 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvNciQZ/build/asan/bin/kudu pid 19061
W20240502 21:27:16.917273 20096 leader_election.cc:334] T 40d4f61d0f6d4d31a5626c7743c88133 P 90e329b457c24caf93a9f880345eb288 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9be399440ded4e2ebe6b6e0bf486cbb1 (127.15.255.133:34217): Network error: Client connection negotiation failed: client connection to 127.15.255.133:34217: connect: Connection reset by peer (error 104)
I20240502 21:27:16.922516 19187 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 21:27:16.982592 23711 raft_consensus.cc:482] T 9844e983dd0a403abe0ac90cd96690d6 P bc20559563fe44efa0a2b7b084aafbce [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 82f50f4b66f3450aa9f58a0b317f5cea)
I20240502 21:27:16.982800 23711 raft_consensus.cc:504] T 9844e983dd0a403abe0ac90cd96690d6 P bc20559563fe44efa0a2b7b084aafbce [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 408 OBSOLETE_local: false peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: false } }
I20240502 21:27:16.983731 23711 leader_election.cc:288] T 9844e983dd0a403abe0ac90cd96690d6 P bc20559563fe44efa0a2b7b084aafbce [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers faadabfb099b4abf90a466395be585b5 (127.15.255.129:32983), 501e3c0d44fe4d70af9ff926d9622417 (127.15.255.131:42501), 82f50f4b66f3450aa9f58a0b317f5cea (127.15.255.130:45139), c75831edb3b44dd59c00ef151adca49d (127.15.255.132:43607)
W20240502 21:27:16.985435 22568 leader_election.cc:334] T 9844e983dd0a403abe0ac90cd96690d6 P bc20559563fe44efa0a2b7b084aafbce [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c75831edb3b44dd59c00ef151adca49d (127.15.255.132:43607): Network error: Client connection negotiation failed: client connection to 127.15.255.132:43607: connect: Connection refused (error 111)
W20240502 21:27:16.985677 22566 leader_election.cc:334] T 9844e983dd0a403abe0ac90cd96690d6 P bc20559563fe44efa0a2b7b084aafbce [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 501e3c0d44fe4d70af9ff926d9622417 (127.15.255.131:42501): Network error: Client connection negotiation failed: client connection to 127.15.255.131:42501: connect: Connection refused (error 111)
W20240502 21:27:16.985836 22569 leader_election.cc:334] T 9844e983dd0a403abe0ac90cd96690d6 P bc20559563fe44efa0a2b7b084aafbce [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 82f50f4b66f3450aa9f58a0b317f5cea (127.15.255.130:45139): Network error: Client connection negotiation failed: client connection to 127.15.255.130:45139: connect: Connection refused (error 111)
I20240502 21:27:16.985978 22569 leader_election.cc:302] T 9844e983dd0a403abe0ac90cd96690d6 P bc20559563fe44efa0a2b7b084aafbce [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 5 voters: 1 yes votes; 3 no votes. yes voters: bc20559563fe44efa0a2b7b084aafbce; no voters: 501e3c0d44fe4d70af9ff926d9622417, 82f50f4b66f3450aa9f58a0b317f5cea, c75831edb3b44dd59c00ef151adca49d
W20240502 21:27:16.986163 22567 leader_election.cc:334] T 9844e983dd0a403abe0ac90cd96690d6 P bc20559563fe44efa0a2b7b084aafbce [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer faadabfb099b4abf90a466395be585b5 (127.15.255.129:32983): Network error: Client connection negotiation failed: client connection to 127.15.255.129:32983: connect: Connection refused (error 111)
I20240502 21:27:16.986265 23711 raft_consensus.cc:2726] T 9844e983dd0a403abe0ac90cd96690d6 P bc20559563fe44efa0a2b7b084aafbce [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 21:27:17.001888 23711 raft_consensus.cc:482] T 40d4f61d0f6d4d31a5626c7743c88133 P bc20559563fe44efa0a2b7b084aafbce [term 1 FOLLOWER]: Starting pre-election (detected failure of leader faadabfb099b4abf90a466395be585b5)
I20240502 21:27:17.002049 23711 raft_consensus.cc:504] T 40d4f61d0f6d4d31a5626c7743c88133 P bc20559563fe44efa0a2b7b084aafbce [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: false } }
I20240502 21:27:17.002764 23712 raft_consensus.cc:482] T 693a8666bbe54ae1b774f7f4da5ba67e P bc20559563fe44efa0a2b7b084aafbce [term 1 FOLLOWER]: Starting pre-election (detected failure of leader faadabfb099b4abf90a466395be585b5)
I20240502 21:27:17.002981 23711 leader_election.cc:288] T 40d4f61d0f6d4d31a5626c7743c88133 P bc20559563fe44efa0a2b7b084aafbce [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 82f50f4b66f3450aa9f58a0b317f5cea (127.15.255.130:45139), 90e329b457c24caf93a9f880345eb288 (127.15.255.135:43395), faadabfb099b4abf90a466395be585b5 (127.15.255.129:32983), 9be399440ded4e2ebe6b6e0bf486cbb1 (127.15.255.133:34217)
I20240502 21:27:17.002924 23712 raft_consensus.cc:504] T 693a8666bbe54ae1b774f7f4da5ba67e P bc20559563fe44efa0a2b7b084aafbce [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 586 OBSOLETE_local: false peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: false } }
I20240502 21:27:17.003130 23690 raft_consensus.cc:482] T 6f0107b0690c4c64b0a4f2472a1c8322 P 90e329b457c24caf93a9f880345eb288 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader faadabfb099b4abf90a466395be585b5)
I20240502 21:27:17.003286 23690 raft_consensus.cc:504] T 6f0107b0690c4c64b0a4f2472a1c8322 P 90e329b457c24caf93a9f880345eb288 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 134 OBSOLETE_local: false peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } }
I20240502 21:27:17.003962 23712 leader_election.cc:288] T 693a8666bbe54ae1b774f7f4da5ba67e P bc20559563fe44efa0a2b7b084aafbce [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9be399440ded4e2ebe6b6e0bf486cbb1 (127.15.255.133:34217), 501e3c0d44fe4d70af9ff926d9622417 (127.15.255.131:42501), faadabfb099b4abf90a466395be585b5 (127.15.255.129:32983), 90e329b457c24caf93a9f880345eb288 (127.15.255.135:43395)
I20240502 21:27:17.004099 23690 leader_election.cc:288] T 6f0107b0690c4c64b0a4f2472a1c8322 P 90e329b457c24caf93a9f880345eb288 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 82f50f4b66f3450aa9f58a0b317f5cea (127.15.255.130:45139), c75831edb3b44dd59c00ef151adca49d (127.15.255.132:43607), faadabfb099b4abf90a466395be585b5 (127.15.255.129:32983), 9be399440ded4e2ebe6b6e0bf486cbb1 (127.15.255.133:34217)
I20240502 21:27:17.004226 23690 raft_consensus.cc:482] T 693a8666bbe54ae1b774f7f4da5ba67e P 90e329b457c24caf93a9f880345eb288 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader faadabfb099b4abf90a466395be585b5)
I20240502 21:27:17.004333 23690 raft_consensus.cc:504] T 693a8666bbe54ae1b774f7f4da5ba67e P 90e329b457c24caf93a9f880345eb288 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 586 OBSOLETE_local: false peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: false } }
I20240502 21:27:17.005075 23690 leader_election.cc:288] T 693a8666bbe54ae1b774f7f4da5ba67e P 90e329b457c24caf93a9f880345eb288 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9be399440ded4e2ebe6b6e0bf486cbb1 (127.15.255.133:34217), 501e3c0d44fe4d70af9ff926d9622417 (127.15.255.131:42501), faadabfb099b4abf90a466395be585b5 (127.15.255.129:32983), bc20559563fe44efa0a2b7b084aafbce (127.15.255.136:34965)
I20240502 21:27:17.006074 20168 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "693a8666bbe54ae1b774f7f4da5ba67e" candidate_uuid: "bc20559563fe44efa0a2b7b084aafbce" candidate_term: 2 candidate_status { last_received { term: 1 index: 586 } } ignore_live_leader: false dest_uuid: "90e329b457c24caf93a9f880345eb288" is_pre_election: true
I20240502 21:27:17.006287 20168 raft_consensus.cc:2454] T 693a8666bbe54ae1b774f7f4da5ba67e P 90e329b457c24caf93a9f880345eb288 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bc20559563fe44efa0a2b7b084aafbce in term 1.
I20240502 21:27:17.006608 20167 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40d4f61d0f6d4d31a5626c7743c88133" candidate_uuid: "bc20559563fe44efa0a2b7b084aafbce" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "90e329b457c24caf93a9f880345eb288" is_pre_election: true
I20240502 21:27:17.006798 20167 raft_consensus.cc:2454] T 40d4f61d0f6d4d31a5626c7743c88133 P 90e329b457c24caf93a9f880345eb288 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bc20559563fe44efa0a2b7b084aafbce in term 1.
W20240502 21:27:17.007232 20094 leader_election.cc:334] T 693a8666bbe54ae1b774f7f4da5ba67e P 90e329b457c24caf93a9f880345eb288 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 501e3c0d44fe4d70af9ff926d9622417 (127.15.255.131:42501): Network error: Client connection negotiation failed: client connection to 127.15.255.131:42501: connect: Connection refused (error 111)
W20240502 21:27:17.008046 22566 leader_election.cc:334] T 693a8666bbe54ae1b774f7f4da5ba67e P bc20559563fe44efa0a2b7b084aafbce [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 501e3c0d44fe4d70af9ff926d9622417 (127.15.255.131:42501): Network error: Client connection negotiation failed: client connection to 127.15.255.131:42501: connect: Connection refused (error 111)
W20240502 21:27:17.008117 20097 leader_election.cc:334] T 6f0107b0690c4c64b0a4f2472a1c8322 P 90e329b457c24caf93a9f880345eb288 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c75831edb3b44dd59c00ef151adca49d (127.15.255.132:43607): Network error: Client connection negotiation failed: client connection to 127.15.255.132:43607: connect: Connection refused (error 111)
W20240502 21:27:17.008543 20096 leader_election.cc:334] T 6f0107b0690c4c64b0a4f2472a1c8322 P 90e329b457c24caf93a9f880345eb288 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer faadabfb099b4abf90a466395be585b5 (127.15.255.129:32983): Network error: Client connection negotiation failed: client connection to 127.15.255.129:32983: connect: Connection refused (error 111)
W20240502 21:27:17.008687 20096 leader_election.cc:334] T 693a8666bbe54ae1b774f7f4da5ba67e P 90e329b457c24caf93a9f880345eb288 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer faadabfb099b4abf90a466395be585b5 (127.15.255.129:32983): Network error: Client connection negotiation failed: client connection to 127.15.255.129:32983: connect: Connection refused (error 111)
W20240502 21:27:17.008730 20098 leader_election.cc:334] T 6f0107b0690c4c64b0a4f2472a1c8322 P 90e329b457c24caf93a9f880345eb288 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 82f50f4b66f3450aa9f58a0b317f5cea (127.15.255.130:45139): Network error: Client connection negotiation failed: client connection to 127.15.255.130:45139: connect: Connection refused (error 111)
I20240502 21:27:17.008848 20098 leader_election.cc:302] T 6f0107b0690c4c64b0a4f2472a1c8322 P 90e329b457c24caf93a9f880345eb288 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 5 voters: 1 yes votes; 3 no votes. yes voters: 90e329b457c24caf93a9f880345eb288; no voters: 82f50f4b66f3450aa9f58a0b317f5cea, c75831edb3b44dd59c00ef151adca49d, faadabfb099b4abf90a466395be585b5
W20240502 21:27:17.009097 22569 leader_election.cc:334] T 40d4f61d0f6d4d31a5626c7743c88133 P bc20559563fe44efa0a2b7b084aafbce [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 82f50f4b66f3450aa9f58a0b317f5cea (127.15.255.130:45139): Network error: Client connection negotiation failed: client connection to 127.15.255.130:45139: connect: Connection refused (error 111)
I20240502 21:27:17.009305 23690 raft_consensus.cc:2726] T 6f0107b0690c4c64b0a4f2472a1c8322 P 90e329b457c24caf93a9f880345eb288 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 21:27:17.009550 20096 leader_election.cc:334] T 6f0107b0690c4c64b0a4f2472a1c8322 P 90e329b457c24caf93a9f880345eb288 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9be399440ded4e2ebe6b6e0bf486cbb1 (127.15.255.133:34217): Network error: Client connection negotiation failed: client connection to 127.15.255.133:34217: connect: Connection refused (error 111)
W20240502 21:27:17.009788 22567 leader_election.cc:334] T 40d4f61d0f6d4d31a5626c7743c88133 P bc20559563fe44efa0a2b7b084aafbce [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer faadabfb099b4abf90a466395be585b5 (127.15.255.129:32983): Network error: Client connection negotiation failed: client connection to 127.15.255.129:32983: connect: Connection refused (error 111)
W20240502 21:27:17.009820 20096 leader_election.cc:334] T 693a8666bbe54ae1b774f7f4da5ba67e P 90e329b457c24caf93a9f880345eb288 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9be399440ded4e2ebe6b6e0bf486cbb1 (127.15.255.133:34217): Network error: Client connection negotiation failed: client connection to 127.15.255.133:34217: connect: Connection refused (error 111)
I20240502 21:27:17.009860 22631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "693a8666bbe54ae1b774f7f4da5ba67e" candidate_uuid: "90e329b457c24caf93a9f880345eb288" candidate_term: 2 candidate_status { last_received { term: 1 index: 586 } } ignore_live_leader: false dest_uuid: "bc20559563fe44efa0a2b7b084aafbce" is_pre_election: true
I20240502 21:27:17.009908 20096 leader_election.cc:302] T 693a8666bbe54ae1b774f7f4da5ba67e P 90e329b457c24caf93a9f880345eb288 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 5 voters: 1 yes votes; 3 no votes. yes voters: 90e329b457c24caf93a9f880345eb288; no voters: 501e3c0d44fe4d70af9ff926d9622417, 9be399440ded4e2ebe6b6e0bf486cbb1, faadabfb099b4abf90a466395be585b5
W20240502 21:27:17.009924 22567 leader_election.cc:334] T 693a8666bbe54ae1b774f7f4da5ba67e P bc20559563fe44efa0a2b7b084aafbce [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer faadabfb099b4abf90a466395be585b5 (127.15.255.129:32983): Network error: Client connection negotiation failed: client connection to 127.15.255.129:32983: connect: Connection refused (error 111)
W20240502 21:27:17.010097 22567 leader_election.cc:334] T 40d4f61d0f6d4d31a5626c7743c88133 P bc20559563fe44efa0a2b7b084aafbce [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9be399440ded4e2ebe6b6e0bf486cbb1 (127.15.255.133:34217): Network error: Client connection negotiation failed: client connection to 127.15.255.133:34217: connect: Connection refused (error 111)
I20240502 21:27:17.010156 22631 raft_consensus.cc:2454] T 693a8666bbe54ae1b774f7f4da5ba67e P bc20559563fe44efa0a2b7b084aafbce [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90e329b457c24caf93a9f880345eb288 in term 1.
I20240502 21:27:17.010210 22567 leader_election.cc:302] T 40d4f61d0f6d4d31a5626c7743c88133 P bc20559563fe44efa0a2b7b084aafbce [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 5 responses out of 5 voters: 2 yes votes; 3 no votes. yes voters: 90e329b457c24caf93a9f880345eb288, bc20559563fe44efa0a2b7b084aafbce; no voters: 82f50f4b66f3450aa9f58a0b317f5cea, 9be399440ded4e2ebe6b6e0bf486cbb1, faadabfb099b4abf90a466395be585b5
I20240502 21:27:17.010289 23690 raft_consensus.cc:2726] T 693a8666bbe54ae1b774f7f4da5ba67e P 90e329b457c24caf93a9f880345eb288 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 21:27:17.010411 23712 raft_consensus.cc:2726] T 40d4f61d0f6d4d31a5626c7743c88133 P bc20559563fe44efa0a2b7b084aafbce [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 21:27:17.010702 22567 leader_election.cc:334] T 693a8666bbe54ae1b774f7f4da5ba67e P bc20559563fe44efa0a2b7b084aafbce [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9be399440ded4e2ebe6b6e0bf486cbb1 (127.15.255.133:34217): Network error: Client connection negotiation failed: client connection to 127.15.255.133:34217: connect: Connection refused (error 111)
I20240502 21:27:17.010795 22567 leader_election.cc:302] T 693a8666bbe54ae1b774f7f4da5ba67e P bc20559563fe44efa0a2b7b084aafbce [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 5 responses out of 5 voters: 2 yes votes; 3 no votes. yes voters: 90e329b457c24caf93a9f880345eb288, bc20559563fe44efa0a2b7b084aafbce; no voters: 501e3c0d44fe4d70af9ff926d9622417, 9be399440ded4e2ebe6b6e0bf486cbb1, faadabfb099b4abf90a466395be585b5
I20240502 21:27:17.011062 23712 raft_consensus.cc:2726] T 693a8666bbe54ae1b774f7f4da5ba67e P bc20559563fe44efa0a2b7b084aafbce [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 21:27:17.064007 23690 raft_consensus.cc:482] T a7a9256f2b784abcb780b190e2ff3983 P 90e329b457c24caf93a9f880345eb288 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 82f50f4b66f3450aa9f58a0b317f5cea)
I20240502 21:27:17.064179 23690 raft_consensus.cc:504] T a7a9256f2b784abcb780b190e2ff3983 P 90e329b457c24caf93a9f880345eb288 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 513 OBSOLETE_local: false peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: false } }
I20240502 21:27:17.065001 23690 leader_election.cc:288] T a7a9256f2b784abcb780b190e2ff3983 P 90e329b457c24caf93a9f880345eb288 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 501e3c0d44fe4d70af9ff926d9622417 (127.15.255.131:42501), faadabfb099b4abf90a466395be585b5 (127.15.255.129:32983), 82f50f4b66f3450aa9f58a0b317f5cea (127.15.255.130:45139), bc20559563fe44efa0a2b7b084aafbce (127.15.255.136:34965)
W20240502 21:27:17.066444 20094 leader_election.cc:334] T a7a9256f2b784abcb780b190e2ff3983 P 90e329b457c24caf93a9f880345eb288 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 501e3c0d44fe4d70af9ff926d9622417 (127.15.255.131:42501): Network error: Client connection negotiation failed: client connection to 127.15.255.131:42501: connect: Connection refused (error 111)
I20240502 21:27:17.066434 22631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7a9256f2b784abcb780b190e2ff3983" candidate_uuid: "90e329b457c24caf93a9f880345eb288" candidate_term: 2 candidate_status { last_received { term: 1 index: 513 } } ignore_live_leader: false dest_uuid: "bc20559563fe44efa0a2b7b084aafbce" is_pre_election: true
I20240502 21:27:17.066715 22631 raft_consensus.cc:2454] T a7a9256f2b784abcb780b190e2ff3983 P bc20559563fe44efa0a2b7b084aafbce [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90e329b457c24caf93a9f880345eb288 in term 1.
W20240502 21:27:17.066834 20098 leader_election.cc:334] T a7a9256f2b784abcb780b190e2ff3983 P 90e329b457c24caf93a9f880345eb288 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 82f50f4b66f3450aa9f58a0b317f5cea (127.15.255.130:45139): Network error: Client connection negotiation failed: client connection to 127.15.255.130:45139: connect: Connection refused (error 111)
W20240502 21:27:17.066970 20096 leader_election.cc:334] T a7a9256f2b784abcb780b190e2ff3983 P 90e329b457c24caf93a9f880345eb288 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer faadabfb099b4abf90a466395be585b5 (127.15.255.129:32983): Network error: Client connection negotiation failed: client connection to 127.15.255.129:32983: connect: Connection refused (error 111)
I20240502 21:27:17.067080 20096 leader_election.cc:302] T a7a9256f2b784abcb780b190e2ff3983 P 90e329b457c24caf93a9f880345eb288 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 5 voters: 1 yes votes; 3 no votes. yes voters: 90e329b457c24caf93a9f880345eb288; no voters: 501e3c0d44fe4d70af9ff926d9622417, 82f50f4b66f3450aa9f58a0b317f5cea, faadabfb099b4abf90a466395be585b5
I20240502 21:27:17.067381 23690 raft_consensus.cc:2726] T a7a9256f2b784abcb780b190e2ff3983 P 90e329b457c24caf93a9f880345eb288 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 21:27:17.186698 22567 consensus_peers.cc:485] T ed78d40318fe4ab5bee4a01b182a34b6 P bc20559563fe44efa0a2b7b084aafbce -> Peer 9be399440ded4e2ebe6b6e0bf486cbb1 (127.15.255.133:34217): Couldn't send request to peer 9be399440ded4e2ebe6b6e0bf486cbb1. Status: Network error: Client connection negotiation failed: client connection to 127.15.255.133:34217: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:17.197088 23707 raft_consensus.cc:482] T 84b235fd89634168938c0dddb534462a P 2877108756304e05bd60ba1186b870da [term 1 FOLLOWER]: Starting pre-election (detected failure of leader faadabfb099b4abf90a466395be585b5)
I20240502 21:27:17.197446 23707 raft_consensus.cc:504] T 84b235fd89634168938c0dddb534462a P 2877108756304e05bd60ba1186b870da [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 134 OBSOLETE_local: false peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "82f50f4b66f3450aa9f58a0b317f5cea" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 45139 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } }
I20240502 21:27:17.198693 23707 leader_election.cc:288] T 84b235fd89634168938c0dddb534462a P 2877108756304e05bd60ba1186b870da [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9be399440ded4e2ebe6b6e0bf486cbb1 (127.15.255.133:34217), faadabfb099b4abf90a466395be585b5 (127.15.255.129:32983), 82f50f4b66f3450aa9f58a0b317f5cea (127.15.255.130:45139), 90e329b457c24caf93a9f880345eb288 (127.15.255.135:43395)
I20240502 21:27:17.199537 20167 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84b235fd89634168938c0dddb534462a" candidate_uuid: "2877108756304e05bd60ba1186b870da" candidate_term: 2 candidate_status { last_received { term: 1 index: 571 } } ignore_live_leader: false dest_uuid: "90e329b457c24caf93a9f880345eb288" is_pre_election: true
I20240502 21:27:17.199790 20167 raft_consensus.cc:2454] T 84b235fd89634168938c0dddb534462a P 90e329b457c24caf93a9f880345eb288 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2877108756304e05bd60ba1186b870da in term 1.
I20240502 21:27:17.206019 16382 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvNciQZ/build/asan/bin/kudu with pid 19061
I20240502 21:27:17.252039 16382 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvNciQZ/build/asan/bin/kudu pid 20077
I20240502 21:27:17.257354 20209 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 21:27:17.342164 23712 raft_consensus.cc:482] T 37e9dcc6191c42d3a5c1503dedc5511e P bc20559563fe44efa0a2b7b084aafbce [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c75831edb3b44dd59c00ef151adca49d)
I20240502 21:27:17.342428 23712 raft_consensus.cc:504] T 37e9dcc6191c42d3a5c1503dedc5511e P bc20559563fe44efa0a2b7b084aafbce [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 534 OBSOLETE_local: false peers { permanent_uuid: "9be399440ded4e2ebe6b6e0bf486cbb1" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 34217 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } attrs { promote: false } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: false } }
I20240502 21:27:17.343443 23712 leader_election.cc:288] T 37e9dcc6191c42d3a5c1503dedc5511e P bc20559563fe44efa0a2b7b084aafbce [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9be399440ded4e2ebe6b6e0bf486cbb1 (127.15.255.133:34217), c75831edb3b44dd59c00ef151adca49d (127.15.255.132:43607), 2877108756304e05bd60ba1186b870da (127.15.255.134:43635), faadabfb099b4abf90a466395be585b5 (127.15.255.129:32983)
W20240502 21:27:17.344941 22567 leader_election.cc:334] T 37e9dcc6191c42d3a5c1503dedc5511e P bc20559563fe44efa0a2b7b084aafbce [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 9be399440ded4e2ebe6b6e0bf486cbb1 (127.15.255.133:34217): Network error: Client connection negotiation failed: client connection to 127.15.255.133:34217: connect: Connection refused (error 111)
W20240502 21:27:17.345218 22567 leader_election.cc:334] T 37e9dcc6191c42d3a5c1503dedc5511e P bc20559563fe44efa0a2b7b084aafbce [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer faadabfb099b4abf90a466395be585b5 (127.15.255.129:32983): Network error: Client connection negotiation failed: client connection to 127.15.255.129:32983: connect: Connection refused (error 111)
W20240502 21:27:17.345448 22569 leader_election.cc:334] T 37e9dcc6191c42d3a5c1503dedc5511e P bc20559563fe44efa0a2b7b084aafbce [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2877108756304e05bd60ba1186b870da (127.15.255.134:43635): Network error: Client connection negotiation failed: client connection to 127.15.255.134:43635: connect: Connection refused (error 111)
I20240502 21:27:17.345590 22569 leader_election.cc:302] T 37e9dcc6191c42d3a5c1503dedc5511e P bc20559563fe44efa0a2b7b084aafbce [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 5 voters: 1 yes votes; 3 no votes. yes voters: bc20559563fe44efa0a2b7b084aafbce; no voters: 2877108756304e05bd60ba1186b870da, 9be399440ded4e2ebe6b6e0bf486cbb1, faadabfb099b4abf90a466395be585b5
W20240502 21:27:17.345782 22568 leader_election.cc:334] T 37e9dcc6191c42d3a5c1503dedc5511e P bc20559563fe44efa0a2b7b084aafbce [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c75831edb3b44dd59c00ef151adca49d (127.15.255.132:43607): Network error: Client connection negotiation failed: client connection to 127.15.255.132:43607: connect: Connection refused (error 111)
I20240502 21:27:17.345865 23712 raft_consensus.cc:2726] T 37e9dcc6191c42d3a5c1503dedc5511e P bc20559563fe44efa0a2b7b084aafbce [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 21:27:17.380148 23712 raft_consensus.cc:482] T 1df9d3ad5ddc438798543900b6c67b46 P bc20559563fe44efa0a2b7b084aafbce [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 501e3c0d44fe4d70af9ff926d9622417)
I20240502 21:27:17.380373 23712 raft_consensus.cc:504] T 1df9d3ad5ddc438798543900b6c67b46 P bc20559563fe44efa0a2b7b084aafbce [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 504 OBSOLETE_local: false peers { permanent_uuid: "faadabfb099b4abf90a466395be585b5" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 32983 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: false } }
I20240502 21:27:17.381994 23712 leader_election.cc:288] T 1df9d3ad5ddc438798543900b6c67b46 P bc20559563fe44efa0a2b7b084aafbce [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers faadabfb099b4abf90a466395be585b5 (127.15.255.129:32983), 2877108756304e05bd60ba1186b870da (127.15.255.134:43635), 501e3c0d44fe4d70af9ff926d9622417 (127.15.255.131:42501), c75831edb3b44dd59c00ef151adca49d (127.15.255.132:43607)
W20240502 21:27:17.383386 22566 leader_election.cc:334] T 1df9d3ad5ddc438798543900b6c67b46 P bc20559563fe44efa0a2b7b084aafbce [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 501e3c0d44fe4d70af9ff926d9622417 (127.15.255.131:42501): Network error: Client connection negotiation failed: client connection to 127.15.255.131:42501: connect: Connection refused (error 111)
W20240502 21:27:17.383646 22569 leader_election.cc:334] T 1df9d3ad5ddc438798543900b6c67b46 P bc20559563fe44efa0a2b7b084aafbce [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2877108756304e05bd60ba1186b870da (127.15.255.134:43635): Network error: Client connection negotiation failed: client connection to 127.15.255.134:43635: connect: Connection refused (error 111)
W20240502 21:27:17.383952 22568 leader_election.cc:334] T 1df9d3ad5ddc438798543900b6c67b46 P bc20559563fe44efa0a2b7b084aafbce [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c75831edb3b44dd59c00ef151adca49d (127.15.255.132:43607): Network error: Client connection negotiation failed: client connection to 127.15.255.132:43607: connect: Connection refused (error 111)
I20240502 21:27:17.384093 22568 leader_election.cc:302] T 1df9d3ad5ddc438798543900b6c67b46 P bc20559563fe44efa0a2b7b084aafbce [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 5 voters: 1 yes votes; 3 no votes. yes voters: bc20559563fe44efa0a2b7b084aafbce; no voters: 2877108756304e05bd60ba1186b870da, 501e3c0d44fe4d70af9ff926d9622417, c75831edb3b44dd59c00ef151adca49d
I20240502 21:27:17.384518 23712 raft_consensus.cc:2726] T 1df9d3ad5ddc438798543900b6c67b46 P bc20559563fe44efa0a2b7b084aafbce [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 21:27:17.384867 22567 leader_election.cc:334] T 1df9d3ad5ddc438798543900b6c67b46 P bc20559563fe44efa0a2b7b084aafbce [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer faadabfb099b4abf90a466395be585b5 (127.15.255.129:32983): Network error: Client connection negotiation failed: client connection to 127.15.255.129:32983: connect: Connection refused (error 111)
I20240502 21:27:17.516743 16382 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvNciQZ/build/asan/bin/kudu with pid 20077
I20240502 21:27:17.560053 16382 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvNciQZ/build/asan/bin/kudu pid 22551
I20240502 21:27:17.565356 22672 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 21:27:17.740371 23712 raft_consensus.cc:482] T ebf7371ff0ef4de99ced8fec0468af28 P bc20559563fe44efa0a2b7b084aafbce [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c75831edb3b44dd59c00ef151adca49d)
I20240502 21:27:17.741044 23712 raft_consensus.cc:504] T ebf7371ff0ef4de99ced8fec0468af28 P bc20559563fe44efa0a2b7b084aafbce [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 389 OBSOLETE_local: false peers { permanent_uuid: "c75831edb3b44dd59c00ef151adca49d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 43607 } } peers { permanent_uuid: "501e3c0d44fe4d70af9ff926d9622417" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 42501 } } peers { permanent_uuid: "2877108756304e05bd60ba1186b870da" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 43635 } } peers { permanent_uuid: "90e329b457c24caf93a9f880345eb288" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 43395 } attrs { promote: false } } peers { permanent_uuid: "bc20559563fe44efa0a2b7b084aafbce" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 34965 } attrs { promote: false } }
I20240502 21:27:17.741297 16382 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvNciQZ/build/asan/bin/kudu with pid 22551
I20240502 21:27:17.766817 16382 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvNciQZ/build/asan/bin/kudu pid 18297
I20240502 21:27:17.767812 18360 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 21:27:17.959622 16382 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvNciQZ/build/asan/bin/kudu with pid 18297
2024-05-02T21:27:17Z chronyd exiting
[ OK ] RebalanceParamTest.Rebalance/7 (102140 ms)
[----------] 1 test from RebalanceParamTest (102140 ms total)
[----------] 1 test from TserverAddedDuringRebalancingTest
[ RUN ] TserverAddedDuringRebalancingTest.TserverStarts/1
I20240502 21:27:18.106509 16382 test_util.cc:268] Using random seed: 1149557124
I20240502 21:27:18.107990 16382 ts_itest-base.cc:112] Starting cluster with:
I20240502 21:27:18.108036 16382 ts_itest-base.cc:113] --------------
I20240502 21:27:18.108052 16382 ts_itest-base.cc:114] 8 tablet servers
I20240502 21:27:18.108088 16382 ts_itest-base.cc:115] 3 replicas per TS
I20240502 21:27:18.108124 16382 ts_itest-base.cc:116] --------------
2024-05-02T21:27:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-02T21:27:18Z Disabled control of system clock
I20240502 21:27:18.157616 16382 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvNciQZ/build/asan/bin/kudu
/tmp/dist-test-taskvNciQZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.255.190:33301
--webserver_interface=127.15.255.190
--webserver_port=0
--builtin_ntp_servers=127.15.255.148:46015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.255.190:33301
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=true
--enable_range_replica_placement=false with env {}
W20240502 21:27:18.477913 23738 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:27:18.478176 23738 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:27:18.478264 23738 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:27:18.485591 23738 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 21:27:18.485697 23738 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240502 21:27:18.485749 23738 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:27:18.485803 23738 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240502 21:27:18.485848 23738 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240502 21:27:18.494491 23738 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.255.148:46015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.15.255.190:33301
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.255.190:33301
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.15.255.190
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:27:18.495620 23738 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:27:18.497351 23738 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:27:18.507182 23744 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:27:18.507697 23747 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:27:18.508010 23745 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:27:18.508298 23738 server_base.cc:1034] running on GCE node
I20240502 21:27:18.509023 23738 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:27:18.510288 23738 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:27:18.511482 23738 hybrid_clock.cc:648] HybridClock initialized: now 1714685238511434 us; error 38 us; skew 500 ppm
I20240502 21:27:18.511806 23738 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:27:18.513856 23738 webserver.cc:460] Webserver started at http://127.15.255.190:44375/ using document root <none> and password file <none>
I20240502 21:27:18.514499 23738 fs_manager.cc:359] Metadata directory not provided
I20240502 21:27:18.514621 23738 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:27:18.514916 23738 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:27:18.517414 23738 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "953efe8866e34dff88dabafc97c90783"
format_stamp: "Formatted at 2024-05-02 21:27:18 on dist-test-slave-jdfx"
I20240502 21:27:18.518152 23738 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "953efe8866e34dff88dabafc97c90783"
format_stamp: "Formatted at 2024-05-02 21:27:18 on dist-test-slave-jdfx"
I20240502 21:27:18.522887 23738 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20240502 21:27:18.526036 23753 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:27:18.527201 23738 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240502 21:27:18.527364 23738 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/master-0/wal
uuid: "953efe8866e34dff88dabafc97c90783"
format_stamp: "Formatted at 2024-05-02 21:27:18 on dist-test-slave-jdfx"
I20240502 21:27:18.527554 23738 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:27:18.549567 23738 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:27:18.550357 23738 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:27:18.550637 23738 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:27:18.567239 23738 rpc_server.cc:307] RPC server started. Bound to: 127.15.255.190:33301
I20240502 21:27:18.567247 23806 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.255.190:33301 every 8 connection(s)
I20240502 21:27:18.568506 23738 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240502 21:27:18.572088 23807 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:18.572139 16382 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvNciQZ/build/asan/bin/kudu as pid 23738
I20240502 21:27:18.572352 16382 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240502 21:27:18.583228 23807 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 953efe8866e34dff88dabafc97c90783: Bootstrap starting.
I20240502 21:27:18.586580 23807 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 953efe8866e34dff88dabafc97c90783: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:18.587491 23807 log.cc:824] T 00000000000000000000000000000000 P 953efe8866e34dff88dabafc97c90783: Log is configured to *not* fsync() on all Append() calls
I20240502 21:27:18.589736 23807 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 953efe8866e34dff88dabafc97c90783: No bootstrap required, opened a new log
I20240502 21:27:18.595523 23807 raft_consensus.cc:348] T 00000000000000000000000000000000 P 953efe8866e34dff88dabafc97c90783 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "953efe8866e34dff88dabafc97c90783" member_type: VOTER last_known_addr { host: "127.15.255.190" port: 33301 } }
I20240502 21:27:18.595854 23807 raft_consensus.cc:374] T 00000000000000000000000000000000 P 953efe8866e34dff88dabafc97c90783 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:18.595955 23807 raft_consensus.cc:729] T 00000000000000000000000000000000 P 953efe8866e34dff88dabafc97c90783 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 953efe8866e34dff88dabafc97c90783, State: Initialized, Role: FOLLOWER
I20240502 21:27:18.596562 23807 consensus_queue.cc:260] T 00000000000000000000000000000000 P 953efe8866e34dff88dabafc97c90783 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "953efe8866e34dff88dabafc97c90783" member_type: VOTER last_known_addr { host: "127.15.255.190" port: 33301 } }
I20240502 21:27:18.596771 23807 raft_consensus.cc:388] T 00000000000000000000000000000000 P 953efe8866e34dff88dabafc97c90783 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 21:27:18.596944 23807 raft_consensus.cc:482] T 00000000000000000000000000000000 P 953efe8866e34dff88dabafc97c90783 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 21:27:18.597136 23807 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 953efe8866e34dff88dabafc97c90783 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:18.599130 23807 raft_consensus.cc:504] T 00000000000000000000000000000000 P 953efe8866e34dff88dabafc97c90783 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "953efe8866e34dff88dabafc97c90783" member_type: VOTER last_known_addr { host: "127.15.255.190" port: 33301 } }
I20240502 21:27:18.599576 23807 leader_election.cc:302] T 00000000000000000000000000000000 P 953efe8866e34dff88dabafc97c90783 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 953efe8866e34dff88dabafc97c90783; no voters:
I20240502 21:27:18.600129 23807 leader_election.cc:288] T 00000000000000000000000000000000 P 953efe8866e34dff88dabafc97c90783 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 21:27:18.600255 23812 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 953efe8866e34dff88dabafc97c90783 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:18.600891 23812 raft_consensus.cc:686] T 00000000000000000000000000000000 P 953efe8866e34dff88dabafc97c90783 [term 1 LEADER]: Becoming Leader. State: Replica: 953efe8866e34dff88dabafc97c90783, State: Running, Role: LEADER
I20240502 21:27:18.601400 23812 consensus_queue.cc:237] T 00000000000000000000000000000000 P 953efe8866e34dff88dabafc97c90783 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "953efe8866e34dff88dabafc97c90783" member_type: VOTER last_known_addr { host: "127.15.255.190" port: 33301 } }
I20240502 21:27:18.602059 23807 sys_catalog.cc:564] T 00000000000000000000000000000000 P 953efe8866e34dff88dabafc97c90783 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 21:27:18.604163 23814 sys_catalog.cc:455] T 00000000000000000000000000000000 P 953efe8866e34dff88dabafc97c90783 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 953efe8866e34dff88dabafc97c90783. Latest consensus state: current_term: 1 leader_uuid: "953efe8866e34dff88dabafc97c90783" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "953efe8866e34dff88dabafc97c90783" member_type: VOTER last_known_addr { host: "127.15.255.190" port: 33301 } } }
I20240502 21:27:18.604590 23814 sys_catalog.cc:458] T 00000000000000000000000000000000 P 953efe8866e34dff88dabafc97c90783 [sys.catalog]: This master's current role is: LEADER
I20240502 21:27:18.604140 23813 sys_catalog.cc:455] T 00000000000000000000000000000000 P 953efe8866e34dff88dabafc97c90783 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "953efe8866e34dff88dabafc97c90783" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "953efe8866e34dff88dabafc97c90783" member_type: VOTER last_known_addr { host: "127.15.255.190" port: 33301 } } }
I20240502 21:27:18.605262 23813 sys_catalog.cc:458] T 00000000000000000000000000000000 P 953efe8866e34dff88dabafc97c90783 [sys.catalog]: This master's current role is: LEADER
I20240502 21:27:18.605832 23818 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 21:27:18.609970 23818 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 21:27:18.618153 23818 catalog_manager.cc:1347] Generated new cluster ID: 28b0d11475f14ea494edf713bc1bd313
I20240502 21:27:18.618264 23818 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 21:27:18.629737 23818 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 21:27:18.630853 23818 catalog_manager.cc:1504] Loading token signing keys...
I20240502 21:27:18.641405 23818 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 953efe8866e34dff88dabafc97c90783: Generated new TSK 0
I20240502 21:27:18.642122 23818 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 21:27:18.649120 16382 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvNciQZ/build/asan/bin/kudu
/tmp/dist-test-taskvNciQZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.255.129:0
--local_ip_for_outbound_sockets=127.15.255.129
--webserver_interface=127.15.255.129
--webserver_port=0
--tserver_master_addrs=127.15.255.190:33301
--builtin_ntp_servers=127.15.255.148:46015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
W20240502 21:27:18.975793 23831 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:27:18.976182 23831 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:27:18.976351 23831 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:27:18.983824 23831 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 21:27:18.983952 23831 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:27:18.984122 23831 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.255.129
I20240502 21:27:18.993104 23831 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.255.148:46015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.255.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.15.255.129
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.15.255.190:33301
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.255.129
--log_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:27:18.994354 23831 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:27:18.996058 23831 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:27:19.007448 23836 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:27:19.007920 23840 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:27:19.008378 23837 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:27:19.008677 23831 server_base.cc:1034] running on GCE node
I20240502 21:27:19.009222 23831 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:27:19.010097 23831 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:27:19.011279 23831 hybrid_clock.cc:648] HybridClock initialized: now 1714685239011223 us; error 49 us; skew 500 ppm
I20240502 21:27:19.011574 23831 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:27:19.013702 23831 webserver.cc:460] Webserver started at http://127.15.255.129:46553/ using document root <none> and password file <none>
I20240502 21:27:19.014418 23831 fs_manager.cc:359] Metadata directory not provided
I20240502 21:27:19.014519 23831 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:27:19.014820 23831 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:27:19.017308 23831 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "6810a866decb4f5089f7d35c870e4204"
format_stamp: "Formatted at 2024-05-02 21:27:19 on dist-test-slave-jdfx"
I20240502 21:27:19.018007 23831 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "6810a866decb4f5089f7d35c870e4204"
format_stamp: "Formatted at 2024-05-02 21:27:19 on dist-test-slave-jdfx"
I20240502 21:27:19.022725 23831 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20240502 21:27:19.025882 23846 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:27:19.027047 23831 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240502 21:27:19.027194 23831 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "6810a866decb4f5089f7d35c870e4204"
format_stamp: "Formatted at 2024-05-02 21:27:19 on dist-test-slave-jdfx"
I20240502 21:27:19.027376 23831 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:27:19.042733 23831 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:27:19.043450 23831 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:27:19.043673 23831 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:27:19.044520 23831 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:27:19.045855 23831 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:27:19.045938 23831 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:27:19.046016 23831 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:27:19.046068 23831 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 21:27:19.074265 23831 rpc_server.cc:307] RPC server started. Bound to: 127.15.255.129:40435
I20240502 21:27:19.074319 23961 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.255.129:40435 every 8 connection(s)
I20240502 21:27:19.075498 23831 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240502 21:27:19.078559 16382 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvNciQZ/build/asan/bin/kudu as pid 23831
I20240502 21:27:19.078756 16382 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240502 21:27:19.081888 16382 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvNciQZ/build/asan/bin/kudu
/tmp/dist-test-taskvNciQZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.255.130:0
--local_ip_for_outbound_sockets=127.15.255.130
--webserver_interface=127.15.255.130
--webserver_port=0
--tserver_master_addrs=127.15.255.190:33301
--builtin_ntp_servers=127.15.255.148:46015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240502 21:27:19.087471 23962 heartbeater.cc:347] Connected to a master server at 127.15.255.190:33301
I20240502 21:27:19.087739 23962 heartbeater.cc:464] Registering TS with master...
I20240502 21:27:19.088555 23962 heartbeater.cc:511] Master 127.15.255.190:33301 requested a full tablet report, sending...
I20240502 21:27:19.090409 23772 ts_manager.cc:194] Registered new tserver with Master: 6810a866decb4f5089f7d35c870e4204 (127.15.255.129:40435)
I20240502 21:27:19.091897 23772 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.255.129:48941
W20240502 21:27:19.401346 23965 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:27:19.401590 23965 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:27:19.401737 23965 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:27:19.409046 23965 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 21:27:19.409148 23965 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:27:19.409291 23965 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.255.130
I20240502 21:27:19.417723 23965 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.255.148:46015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.255.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.15.255.130
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.15.255.190:33301
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.255.130
--log_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:27:19.418802 23965 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:27:19.420367 23965 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:27:19.431295 23973 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:27:19.431591 23971 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:27:19.431958 23970 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:27:19.432205 23965 server_base.cc:1034] running on GCE node
I20240502 21:27:19.432760 23965 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:27:19.433543 23965 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:27:19.434741 23965 hybrid_clock.cc:648] HybridClock initialized: now 1714685239434695 us; error 35 us; skew 500 ppm
I20240502 21:27:19.435025 23965 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:27:19.441944 23965 webserver.cc:460] Webserver started at http://127.15.255.130:44855/ using document root <none> and password file <none>
I20240502 21:27:19.442567 23965 fs_manager.cc:359] Metadata directory not provided
I20240502 21:27:19.442673 23965 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:27:19.442956 23965 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:27:19.447782 23965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "6c194a6db6c145acafc0f3068b8ce1fd"
format_stamp: "Formatted at 2024-05-02 21:27:19 on dist-test-slave-jdfx"
I20240502 21:27:19.448482 23965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "6c194a6db6c145acafc0f3068b8ce1fd"
format_stamp: "Formatted at 2024-05-02 21:27:19 on dist-test-slave-jdfx"
I20240502 21:27:19.452991 23965 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.002s
I20240502 21:27:19.455920 23979 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:27:19.457032 23965 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240502 21:27:19.457173 23965 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "6c194a6db6c145acafc0f3068b8ce1fd"
format_stamp: "Formatted at 2024-05-02 21:27:19 on dist-test-slave-jdfx"
I20240502 21:27:19.457345 23965 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:27:19.483283 23965 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:27:19.483958 23965 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:27:19.484197 23965 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:27:19.485074 23965 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:27:19.486382 23965 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:27:19.486472 23965 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:27:19.486569 23965 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:27:19.486635 23965 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 21:27:19.516170 23965 rpc_server.cc:307] RPC server started. Bound to: 127.15.255.130:41235
I20240502 21:27:19.516265 24096 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.255.130:41235 every 8 connection(s)
I20240502 21:27:19.517323 23965 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240502 21:27:19.526959 16382 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvNciQZ/build/asan/bin/kudu as pid 23965
I20240502 21:27:19.527112 16382 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240502 21:27:19.529078 24097 heartbeater.cc:347] Connected to a master server at 127.15.255.190:33301
I20240502 21:27:19.529363 24097 heartbeater.cc:464] Registering TS with master...
I20240502 21:27:19.530196 24097 heartbeater.cc:511] Master 127.15.255.190:33301 requested a full tablet report, sending...
I20240502 21:27:19.530412 16382 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvNciQZ/build/asan/bin/kudu
/tmp/dist-test-taskvNciQZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.255.131:0
--local_ip_for_outbound_sockets=127.15.255.131
--webserver_interface=127.15.255.131
--webserver_port=0
--tserver_master_addrs=127.15.255.190:33301
--builtin_ntp_servers=127.15.255.148:46015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240502 21:27:19.531702 23771 ts_manager.cc:194] Registered new tserver with Master: 6c194a6db6c145acafc0f3068b8ce1fd (127.15.255.130:41235)
I20240502 21:27:19.532755 23771 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.255.130:36111
W20240502 21:27:19.861892 24100 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:27:19.862167 24100 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:27:19.862278 24100 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:27:19.869278 24100 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 21:27:19.869369 24100 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:27:19.869482 24100 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.255.131
I20240502 21:27:19.877873 24100 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.255.148:46015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.255.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.15.255.131
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.15.255.190:33301
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.255.131
--log_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:27:19.878929 24100 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:27:19.880523 24100 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:27:19.891587 24106 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:27:19.891850 24108 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:27:19.892066 24105 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:27:19.892648 24100 server_base.cc:1034] running on GCE node
I20240502 21:27:19.893147 24100 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:27:19.893932 24100 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:27:19.895117 24100 hybrid_clock.cc:648] HybridClock initialized: now 1714685239895074 us; error 38 us; skew 500 ppm
I20240502 21:27:19.895393 24100 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:27:19.897337 24100 webserver.cc:460] Webserver started at http://127.15.255.131:42699/ using document root <none> and password file <none>
I20240502 21:27:19.897961 24100 fs_manager.cc:359] Metadata directory not provided
I20240502 21:27:19.898070 24100 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:27:19.898360 24100 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:27:19.900971 24100 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "009d3e7e8a8b4e2497de3a73d01ced49"
format_stamp: "Formatted at 2024-05-02 21:27:19 on dist-test-slave-jdfx"
I20240502 21:27:19.901731 24100 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "009d3e7e8a8b4e2497de3a73d01ced49"
format_stamp: "Formatted at 2024-05-02 21:27:19 on dist-test-slave-jdfx"
I20240502 21:27:19.906675 24100 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20240502 21:27:19.909937 24114 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:27:19.911083 24100 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240502 21:27:19.911239 24100 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "009d3e7e8a8b4e2497de3a73d01ced49"
format_stamp: "Formatted at 2024-05-02 21:27:19 on dist-test-slave-jdfx"
I20240502 21:27:19.911448 24100 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:27:19.925554 24100 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:27:19.926266 24100 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:27:19.926546 24100 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:27:19.927434 24100 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:27:19.928907 24100 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:27:19.929011 24100 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:27:19.929121 24100 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:27:19.929196 24100 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 21:27:19.960366 24100 rpc_server.cc:307] RPC server started. Bound to: 127.15.255.131:37895
I20240502 21:27:19.960459 24230 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.255.131:37895 every 8 connection(s)
I20240502 21:27:19.961597 24100 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240502 21:27:19.968978 16382 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvNciQZ/build/asan/bin/kudu as pid 24100
I20240502 21:27:19.969146 16382 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240502 21:27:19.972246 16382 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvNciQZ/build/asan/bin/kudu
/tmp/dist-test-taskvNciQZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.255.132:0
--local_ip_for_outbound_sockets=127.15.255.132
--webserver_interface=127.15.255.132
--webserver_port=0
--tserver_master_addrs=127.15.255.190:33301
--builtin_ntp_servers=127.15.255.148:46015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240502 21:27:19.973292 24231 heartbeater.cc:347] Connected to a master server at 127.15.255.190:33301
I20240502 21:27:19.973551 24231 heartbeater.cc:464] Registering TS with master...
I20240502 21:27:19.974257 24231 heartbeater.cc:511] Master 127.15.255.190:33301 requested a full tablet report, sending...
I20240502 21:27:19.975595 23771 ts_manager.cc:194] Registered new tserver with Master: 009d3e7e8a8b4e2497de3a73d01ced49 (127.15.255.131:37895)
I20240502 21:27:19.976541 23771 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.255.131:49359
I20240502 21:27:20.094341 23962 heartbeater.cc:503] Master 127.15.255.190:33301 was elected leader, sending a full tablet report...
W20240502 21:27:20.286671 24234 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:27:20.286926 24234 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:27:20.287067 24234 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:27:20.294113 24234 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 21:27:20.294226 24234 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:27:20.294371 24234 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.255.132
I20240502 21:27:20.302747 24234 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.255.148:46015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.255.132:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.15.255.132
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.15.255.190:33301
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.255.132
--log_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:27:20.303857 24234 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:27:20.305399 24234 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:27:20.316213 24239 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:27:20.316742 24243 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:27:20.317003 24240 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:27:20.317230 24234 server_base.cc:1034] running on GCE node
I20240502 21:27:20.317746 24234 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:27:20.318593 24234 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:27:20.319788 24234 hybrid_clock.cc:648] HybridClock initialized: now 1714685240319739 us; error 43 us; skew 500 ppm
I20240502 21:27:20.320062 24234 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:27:20.326412 24234 webserver.cc:460] Webserver started at http://127.15.255.132:38581/ using document root <none> and password file <none>
I20240502 21:27:20.327046 24234 fs_manager.cc:359] Metadata directory not provided
I20240502 21:27:20.327150 24234 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:27:20.327433 24234 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:27:20.329828 24234 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6"
format_stamp: "Formatted at 2024-05-02 21:27:20 on dist-test-slave-jdfx"
I20240502 21:27:20.330508 24234 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6"
format_stamp: "Formatted at 2024-05-02 21:27:20 on dist-test-slave-jdfx"
I20240502 21:27:20.335132 24234 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.005s
I20240502 21:27:20.338315 24249 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:27:20.339430 24234 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240502 21:27:20.339570 24234 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6"
format_stamp: "Formatted at 2024-05-02 21:27:20 on dist-test-slave-jdfx"
I20240502 21:27:20.339751 24234 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:27:20.361928 24234 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:27:20.362643 24234 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:27:20.362886 24234 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:27:20.363716 24234 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:27:20.365023 24234 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:27:20.365119 24234 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:27:20.365221 24234 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:27:20.365288 24234 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 21:27:20.393234 24234 rpc_server.cc:307] RPC server started. Bound to: 127.15.255.132:46813
I20240502 21:27:20.393298 24362 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.255.132:46813 every 8 connection(s)
I20240502 21:27:20.394311 24234 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240502 21:27:20.399574 16382 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvNciQZ/build/asan/bin/kudu as pid 24234
I20240502 21:27:20.399773 16382 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240502 21:27:20.402678 16382 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvNciQZ/build/asan/bin/kudu
/tmp/dist-test-taskvNciQZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.255.133:0
--local_ip_for_outbound_sockets=127.15.255.133
--webserver_interface=127.15.255.133
--webserver_port=0
--tserver_master_addrs=127.15.255.190:33301
--builtin_ntp_servers=127.15.255.148:46015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240502 21:27:20.405710 24363 heartbeater.cc:347] Connected to a master server at 127.15.255.190:33301
I20240502 21:27:20.405948 24363 heartbeater.cc:464] Registering TS with master...
I20240502 21:27:20.406652 24363 heartbeater.cc:511] Master 127.15.255.190:33301 requested a full tablet report, sending...
I20240502 21:27:20.409658 23771 ts_manager.cc:194] Registered new tserver with Master: 038c5d9d1d7a49c981c4c34fbc00d9e6 (127.15.255.132:46813)
I20240502 21:27:20.410627 23771 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.255.132:44807
I20240502 21:27:20.534876 24097 heartbeater.cc:503] Master 127.15.255.190:33301 was elected leader, sending a full tablet report...
W20240502 21:27:20.721464 24366 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:27:20.721729 24366 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:27:20.721868 24366 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:27:20.728899 24366 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 21:27:20.729008 24366 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:27:20.729148 24366 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.255.133
I20240502 21:27:20.737679 24366 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.255.148:46015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.255.133:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.15.255.133
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.15.255.190:33301
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.255.133
--log_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:27:20.738785 24366 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:27:20.740355 24366 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:27:20.751329 24371 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:27:20.751717 24372 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:27:20.752228 24374 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:27:20.752480 24366 server_base.cc:1034] running on GCE node
I20240502 21:27:20.752959 24366 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:27:20.753719 24366 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:27:20.754902 24366 hybrid_clock.cc:648] HybridClock initialized: now 1714685240754859 us; error 32 us; skew 500 ppm
I20240502 21:27:20.755178 24366 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:27:20.757115 24366 webserver.cc:460] Webserver started at http://127.15.255.133:43285/ using document root <none> and password file <none>
I20240502 21:27:20.757747 24366 fs_manager.cc:359] Metadata directory not provided
I20240502 21:27:20.757848 24366 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:27:20.758128 24366 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:27:20.760478 24366 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "ee27c3f9e0ec46caa742ac386d6b7c20"
format_stamp: "Formatted at 2024-05-02 21:27:20 on dist-test-slave-jdfx"
I20240502 21:27:20.761217 24366 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "ee27c3f9e0ec46caa742ac386d6b7c20"
format_stamp: "Formatted at 2024-05-02 21:27:20 on dist-test-slave-jdfx"
I20240502 21:27:20.765839 24366 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20240502 21:27:20.768770 24380 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:27:20.769923 24366 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240502 21:27:20.770058 24366 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "ee27c3f9e0ec46caa742ac386d6b7c20"
format_stamp: "Formatted at 2024-05-02 21:27:20 on dist-test-slave-jdfx"
I20240502 21:27:20.770224 24366 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:27:20.783850 24366 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:27:20.784557 24366 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:27:20.784785 24366 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:27:20.785575 24366 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:27:20.786868 24366 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:27:20.786953 24366 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:27:20.787050 24366 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:27:20.787113 24366 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 21:27:20.814806 24366 rpc_server.cc:307] RPC server started. Bound to: 127.15.255.133:40841
I20240502 21:27:20.814867 24495 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.255.133:40841 every 8 connection(s)
I20240502 21:27:20.815999 24366 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240502 21:27:20.821584 16382 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvNciQZ/build/asan/bin/kudu as pid 24366
I20240502 21:27:20.821748 16382 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240502 21:27:20.824707 16382 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvNciQZ/build/asan/bin/kudu
/tmp/dist-test-taskvNciQZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.255.134:0
--local_ip_for_outbound_sockets=127.15.255.134
--webserver_interface=127.15.255.134
--webserver_port=0
--tserver_master_addrs=127.15.255.190:33301
--builtin_ntp_servers=127.15.255.148:46015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240502 21:27:20.827589 24496 heartbeater.cc:347] Connected to a master server at 127.15.255.190:33301
I20240502 21:27:20.827836 24496 heartbeater.cc:464] Registering TS with master...
I20240502 21:27:20.828599 24496 heartbeater.cc:511] Master 127.15.255.190:33301 requested a full tablet report, sending...
I20240502 21:27:20.829850 23771 ts_manager.cc:194] Registered new tserver with Master: ee27c3f9e0ec46caa742ac386d6b7c20 (127.15.255.133:40841)
I20240502 21:27:20.830607 23771 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.255.133:33221
I20240502 21:27:20.978760 24231 heartbeater.cc:503] Master 127.15.255.190:33301 was elected leader, sending a full tablet report...
W20240502 21:27:21.141011 24499 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:27:21.141260 24499 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:27:21.141402 24499 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:27:21.148643 24499 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 21:27:21.148742 24499 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:27:21.148891 24499 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.255.134
I20240502 21:27:21.157306 24499 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.255.148:46015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.255.134:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.15.255.134
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.15.255.190:33301
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.255.134
--log_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:27:21.158488 24499 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:27:21.160076 24499 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:27:21.171016 24504 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:27:21.171437 24507 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:27:21.171949 24505 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:27:21.172103 24499 server_base.cc:1034] running on GCE node
I20240502 21:27:21.172683 24499 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:27:21.173502 24499 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:27:21.174685 24499 hybrid_clock.cc:648] HybridClock initialized: now 1714685241174648 us; error 35 us; skew 500 ppm
I20240502 21:27:21.174988 24499 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:27:21.177017 24499 webserver.cc:460] Webserver started at http://127.15.255.134:43073/ using document root <none> and password file <none>
I20240502 21:27:21.177670 24499 fs_manager.cc:359] Metadata directory not provided
I20240502 21:27:21.177778 24499 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:27:21.178061 24499 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:27:21.180522 24499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "fb6004b8640a4906aa0cfe63e383d40e"
format_stamp: "Formatted at 2024-05-02 21:27:21 on dist-test-slave-jdfx"
I20240502 21:27:21.181248 24499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "fb6004b8640a4906aa0cfe63e383d40e"
format_stamp: "Formatted at 2024-05-02 21:27:21 on dist-test-slave-jdfx"
I20240502 21:27:21.185930 24499 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20240502 21:27:21.188966 24514 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:27:21.190114 24499 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240502 21:27:21.190258 24499 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "fb6004b8640a4906aa0cfe63e383d40e"
format_stamp: "Formatted at 2024-05-02 21:27:21 on dist-test-slave-jdfx"
I20240502 21:27:21.190435 24499 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:27:21.203668 24499 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:27:21.204357 24499 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:27:21.204654 24499 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:27:21.205461 24499 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:27:21.206810 24499 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:27:21.206902 24499 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:27:21.207003 24499 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:27:21.207069 24499 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 21:27:21.235121 24499 rpc_server.cc:307] RPC server started. Bound to: 127.15.255.134:32825
I20240502 21:27:21.235175 24628 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.255.134:32825 every 8 connection(s)
I20240502 21:27:21.236281 24499 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240502 21:27:21.242699 16382 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvNciQZ/build/asan/bin/kudu as pid 24499
I20240502 21:27:21.242890 16382 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240502 21:27:21.246277 16382 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvNciQZ/build/asan/bin/kudu
/tmp/dist-test-taskvNciQZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.255.135:0
--local_ip_for_outbound_sockets=127.15.255.135
--webserver_interface=127.15.255.135
--webserver_port=0
--tserver_master_addrs=127.15.255.190:33301
--builtin_ntp_servers=127.15.255.148:46015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240502 21:27:21.248045 24629 heartbeater.cc:347] Connected to a master server at 127.15.255.190:33301
I20240502 21:27:21.248351 24629 heartbeater.cc:464] Registering TS with master...
I20240502 21:27:21.249258 24629 heartbeater.cc:511] Master 127.15.255.190:33301 requested a full tablet report, sending...
I20240502 21:27:21.250613 23771 ts_manager.cc:194] Registered new tserver with Master: fb6004b8640a4906aa0cfe63e383d40e (127.15.255.134:32825)
I20240502 21:27:21.251379 23771 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.255.134:53861
I20240502 21:27:21.413051 24363 heartbeater.cc:503] Master 127.15.255.190:33301 was elected leader, sending a full tablet report...
W20240502 21:27:21.566713 24632 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:27:21.566957 24632 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:27:21.567085 24632 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:27:21.574256 24632 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 21:27:21.574373 24632 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:27:21.574533 24632 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.255.135
I20240502 21:27:21.583058 24632 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.255.148:46015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.255.135:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.15.255.135
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.15.255.190:33301
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.255.135
--log_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:27:21.584200 24632 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:27:21.585827 24632 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:27:21.596630 24637 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:27:21.597108 24638 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:27:21.597509 24640 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:27:21.597749 24632 server_base.cc:1034] running on GCE node
I20240502 21:27:21.598240 24632 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:27:21.599061 24632 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:27:21.600248 24632 hybrid_clock.cc:648] HybridClock initialized: now 1714685241600203 us; error 39 us; skew 500 ppm
I20240502 21:27:21.600577 24632 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:27:21.602559 24632 webserver.cc:460] Webserver started at http://127.15.255.135:34771/ using document root <none> and password file <none>
I20240502 21:27:21.603179 24632 fs_manager.cc:359] Metadata directory not provided
I20240502 21:27:21.603282 24632 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:27:21.603592 24632 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:27:21.605950 24632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-6/data/instance:
uuid: "e117afbf8819424e8a98476453dfde2a"
format_stamp: "Formatted at 2024-05-02 21:27:21 on dist-test-slave-jdfx"
I20240502 21:27:21.606628 24632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-6/wal/instance:
uuid: "e117afbf8819424e8a98476453dfde2a"
format_stamp: "Formatted at 2024-05-02 21:27:21 on dist-test-slave-jdfx"
I20240502 21:27:21.611322 24632 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20240502 21:27:21.614358 24646 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:27:21.615527 24632 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 21:27:21.615664 24632 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "e117afbf8819424e8a98476453dfde2a"
format_stamp: "Formatted at 2024-05-02 21:27:21 on dist-test-slave-jdfx"
I20240502 21:27:21.615836 24632 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:27:21.635370 24632 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:27:21.636022 24632 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:27:21.636250 24632 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:27:21.637087 24632 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:27:21.638371 24632 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:27:21.638460 24632 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:27:21.638556 24632 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:27:21.638622 24632 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 21:27:21.667124 24632 rpc_server.cc:307] RPC server started. Bound to: 127.15.255.135:32775
I20240502 21:27:21.667182 24760 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.255.135:32775 every 8 connection(s)
I20240502 21:27:21.668319 24632 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240502 21:27:21.674677 16382 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvNciQZ/build/asan/bin/kudu as pid 24632
I20240502 21:27:21.674870 16382 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-6/wal/instance
I20240502 21:27:21.678382 16382 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvNciQZ/build/asan/bin/kudu
/tmp/dist-test-taskvNciQZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.255.136:0
--local_ip_for_outbound_sockets=127.15.255.136
--webserver_interface=127.15.255.136
--webserver_port=0
--tserver_master_addrs=127.15.255.190:33301
--builtin_ntp_servers=127.15.255.148:46015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240502 21:27:21.680838 24761 heartbeater.cc:347] Connected to a master server at 127.15.255.190:33301
I20240502 21:27:21.681139 24761 heartbeater.cc:464] Registering TS with master...
I20240502 21:27:21.682046 24761 heartbeater.cc:511] Master 127.15.255.190:33301 requested a full tablet report, sending...
I20240502 21:27:21.683414 23771 ts_manager.cc:194] Registered new tserver with Master: e117afbf8819424e8a98476453dfde2a (127.15.255.135:32775)
I20240502 21:27:21.684360 23771 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.255.135:44011
I20240502 21:27:21.832953 24496 heartbeater.cc:503] Master 127.15.255.190:33301 was elected leader, sending a full tablet report...
W20240502 21:27:22.001459 24764 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:27:22.001690 24764 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:27:22.001793 24764 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:27:22.009099 24764 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 21:27:22.009195 24764 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:27:22.009315 24764 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.255.136
I20240502 21:27:22.017928 24764 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.255.148:46015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.255.136:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.15.255.136
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.15.255.190:33301
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.255.136
--log_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:27:22.019002 24764 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:27:22.020707 24764 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:27:22.031950 24772 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:27:22.031909 24769 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:27:22.032488 24770 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:27:22.032917 24764 server_base.cc:1034] running on GCE node
I20240502 21:27:22.033403 24764 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:27:22.034130 24764 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:27:22.035317 24764 hybrid_clock.cc:648] HybridClock initialized: now 1714685242035264 us; error 39 us; skew 500 ppm
I20240502 21:27:22.035593 24764 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:27:22.037580 24764 webserver.cc:460] Webserver started at http://127.15.255.136:41673/ using document root <none> and password file <none>
I20240502 21:27:22.038199 24764 fs_manager.cc:359] Metadata directory not provided
I20240502 21:27:22.038316 24764 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:27:22.038599 24764 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:27:22.040948 24764 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-7/data/instance:
uuid: "a3e500d7077a48acb5edb33f7e178c81"
format_stamp: "Formatted at 2024-05-02 21:27:22 on dist-test-slave-jdfx"
I20240502 21:27:22.041635 24764 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-7/wal/instance:
uuid: "a3e500d7077a48acb5edb33f7e178c81"
format_stamp: "Formatted at 2024-05-02 21:27:22 on dist-test-slave-jdfx"
I20240502 21:27:22.046157 24764 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20240502 21:27:22.049108 24778 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:27:22.050199 24764 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240502 21:27:22.050352 24764 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "a3e500d7077a48acb5edb33f7e178c81"
format_stamp: "Formatted at 2024-05-02 21:27:22 on dist-test-slave-jdfx"
I20240502 21:27:22.050529 24764 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:27:22.087545 24764 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:27:22.088310 24764 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:27:22.088593 24764 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:27:22.089452 24764 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:27:22.090867 24764 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:27:22.090965 24764 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:27:22.091065 24764 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:27:22.091137 24764 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 21:27:22.120234 24764 rpc_server.cc:307] RPC server started. Bound to: 127.15.255.136:43595
I20240502 21:27:22.120293 24891 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.255.136:43595 every 8 connection(s)
I20240502 21:27:22.121477 24764 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240502 21:27:22.128343 16382 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvNciQZ/build/asan/bin/kudu as pid 24764
I20240502 21:27:22.128541 16382 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-7/wal/instance
I20240502 21:27:22.133002 24892 heartbeater.cc:347] Connected to a master server at 127.15.255.190:33301
I20240502 21:27:22.133253 24892 heartbeater.cc:464] Registering TS with master...
I20240502 21:27:22.133982 24892 heartbeater.cc:511] Master 127.15.255.190:33301 requested a full tablet report, sending...
I20240502 21:27:22.135169 23771 ts_manager.cc:194] Registered new tserver with Master: a3e500d7077a48acb5edb33f7e178c81 (127.15.255.136:43595)
I20240502 21:27:22.135917 23771 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.255.136:51195
I20240502 21:27:22.144526 16382 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240502 21:27:22.160135 16382 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240502 21:27:22.160264 16382 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvNciQZ/build/asan/bin/kudu pid 24366
I20240502 21:27:22.167824 24491 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 21:27:22.246614 16382 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvNciQZ/build/asan/bin/kudu with pid 24366
I20240502 21:27:22.253952 24629 heartbeater.cc:503] Master 127.15.255.190:33301 was elected leader, sending a full tablet report...
I20240502 21:27:22.261045 16382 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvNciQZ/build/asan/bin/kudu pid 24499
I20240502 21:27:22.268235 24624 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 21:27:22.345505 16382 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvNciQZ/build/asan/bin/kudu with pid 24499
I20240502 21:27:22.359383 16382 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvNciQZ/build/asan/bin/kudu pid 24632
I20240502 21:27:22.366936 24756 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 21:27:22.444025 16382 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvNciQZ/build/asan/bin/kudu with pid 24632
I20240502 21:27:22.457993 16382 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvNciQZ/build/asan/bin/kudu pid 24764
I20240502 21:27:22.465725 24887 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 21:27:22.544344 16382 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvNciQZ/build/asan/bin/kudu with pid 24764
I20240502 21:27:26.310134 23771 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41618:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 21:27:26.352016 24164 tablet_service.cc:1467] Processing CreateTablet for tablet d79eec4f59634df480161815d888bcac (DEFAULT_TABLE table=rebalance_test_table_0 [id=1e673d0be4694336bbf65d9c114c614f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:26.353456 24164 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d79eec4f59634df480161815d888bcac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:26.352937 24029 tablet_service.cc:1467] Processing CreateTablet for tablet d79eec4f59634df480161815d888bcac (DEFAULT_TABLE table=rebalance_test_table_0 [id=1e673d0be4694336bbf65d9c114c614f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:26.354240 24029 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d79eec4f59634df480161815d888bcac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:26.354944 24027 tablet_service.cc:1467] Processing CreateTablet for tablet adf80b9d71314010a7a4cf052fd8a511 (DEFAULT_TABLE table=rebalance_test_table_0 [id=1e673d0be4694336bbf65d9c114c614f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:26.355957 24163 tablet_service.cc:1467] Processing CreateTablet for tablet dc0e070513cc46ca82c1946423ad91c5 (DEFAULT_TABLE table=rebalance_test_table_0 [id=1e673d0be4694336bbf65d9c114c614f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:26.356413 24163 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc0e070513cc46ca82c1946423ad91c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:26.357379 24028 tablet_service.cc:1467] Processing CreateTablet for tablet dc0e070513cc46ca82c1946423ad91c5 (DEFAULT_TABLE table=rebalance_test_table_0 [id=1e673d0be4694336bbf65d9c114c614f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:26.358108 24027 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet adf80b9d71314010a7a4cf052fd8a511. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:26.358450 24296 tablet_service.cc:1467] Processing CreateTablet for tablet adf80b9d71314010a7a4cf052fd8a511 (DEFAULT_TABLE table=rebalance_test_table_0 [id=1e673d0be4694336bbf65d9c114c614f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:26.358992 23895 tablet_service.cc:1467] Processing CreateTablet for tablet adf80b9d71314010a7a4cf052fd8a511 (DEFAULT_TABLE table=rebalance_test_table_0 [id=1e673d0be4694336bbf65d9c114c614f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:26.359673 24296 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet adf80b9d71314010a7a4cf052fd8a511. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:26.360216 23895 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet adf80b9d71314010a7a4cf052fd8a511. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:26.360816 24028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc0e070513cc46ca82c1946423ad91c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:26.361066 23896 tablet_service.cc:1467] Processing CreateTablet for tablet dc0e070513cc46ca82c1946423ad91c5 (DEFAULT_TABLE table=rebalance_test_table_0 [id=1e673d0be4694336bbf65d9c114c614f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:26.361492 23896 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc0e070513cc46ca82c1946423ad91c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:26.362427 24297 tablet_service.cc:1467] Processing CreateTablet for tablet d79eec4f59634df480161815d888bcac (DEFAULT_TABLE table=rebalance_test_table_0 [id=1e673d0be4694336bbf65d9c114c614f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:26.362915 24297 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d79eec4f59634df480161815d888bcac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:26.374439 24921 tablet_bootstrap.cc:492] T dc0e070513cc46ca82c1946423ad91c5 P 009d3e7e8a8b4e2497de3a73d01ced49: Bootstrap starting.
I20240502 21:27:26.377714 24921 tablet_bootstrap.cc:654] T dc0e070513cc46ca82c1946423ad91c5 P 009d3e7e8a8b4e2497de3a73d01ced49: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:26.379015 24921 log.cc:824] T dc0e070513cc46ca82c1946423ad91c5 P 009d3e7e8a8b4e2497de3a73d01ced49: Log is configured to *not* fsync() on all Append() calls
I20240502 21:27:26.382364 24921 tablet_bootstrap.cc:492] T dc0e070513cc46ca82c1946423ad91c5 P 009d3e7e8a8b4e2497de3a73d01ced49: No bootstrap required, opened a new log
I20240502 21:27:26.382637 24921 ts_tablet_manager.cc:1367] T dc0e070513cc46ca82c1946423ad91c5 P 009d3e7e8a8b4e2497de3a73d01ced49: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.005s
I20240502 21:27:26.391542 24921 raft_consensus.cc:348] T dc0e070513cc46ca82c1946423ad91c5 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } }
I20240502 21:27:26.391942 24921 raft_consensus.cc:374] T dc0e070513cc46ca82c1946423ad91c5 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:26.392050 24921 raft_consensus.cc:729] T dc0e070513cc46ca82c1946423ad91c5 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 009d3e7e8a8b4e2497de3a73d01ced49, State: Initialized, Role: FOLLOWER
I20240502 21:27:26.392736 24921 consensus_queue.cc:260] T dc0e070513cc46ca82c1946423ad91c5 P 009d3e7e8a8b4e2497de3a73d01ced49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } }
I20240502 21:27:26.397317 24925 tablet_bootstrap.cc:492] T adf80b9d71314010a7a4cf052fd8a511 P 6810a866decb4f5089f7d35c870e4204: Bootstrap starting.
I20240502 21:27:26.400688 24921 ts_tablet_manager.cc:1398] T dc0e070513cc46ca82c1946423ad91c5 P 009d3e7e8a8b4e2497de3a73d01ced49: Time spent starting tablet: real 0.018s user 0.006s sys 0.012s
I20240502 21:27:26.401051 24921 tablet_bootstrap.cc:492] T d79eec4f59634df480161815d888bcac P 009d3e7e8a8b4e2497de3a73d01ced49: Bootstrap starting.
I20240502 21:27:26.401034 24923 tablet_bootstrap.cc:492] T dc0e070513cc46ca82c1946423ad91c5 P 6c194a6db6c145acafc0f3068b8ce1fd: Bootstrap starting.
I20240502 21:27:26.402581 24921 tablet_bootstrap.cc:654] T d79eec4f59634df480161815d888bcac P 009d3e7e8a8b4e2497de3a73d01ced49: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:26.404178 24925 tablet_bootstrap.cc:654] T adf80b9d71314010a7a4cf052fd8a511 P 6810a866decb4f5089f7d35c870e4204: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:26.405025 24922 tablet_bootstrap.cc:492] T adf80b9d71314010a7a4cf052fd8a511 P 038c5d9d1d7a49c981c4c34fbc00d9e6: Bootstrap starting.
I20240502 21:27:26.406896 24921 tablet_bootstrap.cc:492] T d79eec4f59634df480161815d888bcac P 009d3e7e8a8b4e2497de3a73d01ced49: No bootstrap required, opened a new log
I20240502 21:27:26.406877 24923 tablet_bootstrap.cc:654] T dc0e070513cc46ca82c1946423ad91c5 P 6c194a6db6c145acafc0f3068b8ce1fd: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:26.407033 24921 ts_tablet_manager.cc:1367] T d79eec4f59634df480161815d888bcac P 009d3e7e8a8b4e2497de3a73d01ced49: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.001s
I20240502 21:27:26.406885 24925 log.cc:824] T adf80b9d71314010a7a4cf052fd8a511 P 6810a866decb4f5089f7d35c870e4204: Log is configured to *not* fsync() on all Append() calls
I20240502 21:27:26.408002 24922 tablet_bootstrap.cc:654] T adf80b9d71314010a7a4cf052fd8a511 P 038c5d9d1d7a49c981c4c34fbc00d9e6: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:26.408071 24923 log.cc:824] T dc0e070513cc46ca82c1946423ad91c5 P 6c194a6db6c145acafc0f3068b8ce1fd: Log is configured to *not* fsync() on all Append() calls
I20240502 21:27:26.408023 24921 raft_consensus.cc:348] T d79eec4f59634df480161815d888bcac P 009d3e7e8a8b4e2497de3a73d01ced49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } }
I20240502 21:27:26.408207 24921 raft_consensus.cc:374] T d79eec4f59634df480161815d888bcac P 009d3e7e8a8b4e2497de3a73d01ced49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:26.408285 24921 raft_consensus.cc:729] T d79eec4f59634df480161815d888bcac P 009d3e7e8a8b4e2497de3a73d01ced49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 009d3e7e8a8b4e2497de3a73d01ced49, State: Initialized, Role: FOLLOWER
I20240502 21:27:26.408619 24921 consensus_queue.cc:260] T d79eec4f59634df480161815d888bcac P 009d3e7e8a8b4e2497de3a73d01ced49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } }
I20240502 21:27:26.409224 24922 log.cc:824] T adf80b9d71314010a7a4cf052fd8a511 P 038c5d9d1d7a49c981c4c34fbc00d9e6: Log is configured to *not* fsync() on all Append() calls
I20240502 21:27:26.411206 24921 ts_tablet_manager.cc:1398] T d79eec4f59634df480161815d888bcac P 009d3e7e8a8b4e2497de3a73d01ced49: Time spent starting tablet: real 0.004s user 0.001s sys 0.002s
I20240502 21:27:26.414481 24923 tablet_bootstrap.cc:492] T dc0e070513cc46ca82c1946423ad91c5 P 6c194a6db6c145acafc0f3068b8ce1fd: No bootstrap required, opened a new log
I20240502 21:27:26.414742 24923 ts_tablet_manager.cc:1367] T dc0e070513cc46ca82c1946423ad91c5 P 6c194a6db6c145acafc0f3068b8ce1fd: Time spent bootstrapping tablet: real 0.014s user 0.009s sys 0.000s
I20240502 21:27:26.415266 24925 tablet_bootstrap.cc:492] T adf80b9d71314010a7a4cf052fd8a511 P 6810a866decb4f5089f7d35c870e4204: No bootstrap required, opened a new log
I20240502 21:27:26.415495 24925 ts_tablet_manager.cc:1367] T adf80b9d71314010a7a4cf052fd8a511 P 6810a866decb4f5089f7d35c870e4204: Time spent bootstrapping tablet: real 0.018s user 0.005s sys 0.006s
I20240502 21:27:26.415827 24922 tablet_bootstrap.cc:492] T adf80b9d71314010a7a4cf052fd8a511 P 038c5d9d1d7a49c981c4c34fbc00d9e6: No bootstrap required, opened a new log
I20240502 21:27:26.416085 24922 ts_tablet_manager.cc:1367] T adf80b9d71314010a7a4cf052fd8a511 P 038c5d9d1d7a49c981c4c34fbc00d9e6: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.005s
I20240502 21:27:26.421267 24925 raft_consensus.cc:348] T adf80b9d71314010a7a4cf052fd8a511 P 6810a866decb4f5089f7d35c870e4204 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } }
I20240502 21:27:26.421584 24925 raft_consensus.cc:374] T adf80b9d71314010a7a4cf052fd8a511 P 6810a866decb4f5089f7d35c870e4204 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:26.421679 24925 raft_consensus.cc:729] T adf80b9d71314010a7a4cf052fd8a511 P 6810a866decb4f5089f7d35c870e4204 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6810a866decb4f5089f7d35c870e4204, State: Initialized, Role: FOLLOWER
I20240502 21:27:26.422986 24925 consensus_queue.cc:260] T adf80b9d71314010a7a4cf052fd8a511 P 6810a866decb4f5089f7d35c870e4204 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } }
I20240502 21:27:26.423156 24923 raft_consensus.cc:348] T dc0e070513cc46ca82c1946423ad91c5 P 6c194a6db6c145acafc0f3068b8ce1fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } }
I20240502 21:27:26.423550 24923 raft_consensus.cc:374] T dc0e070513cc46ca82c1946423ad91c5 P 6c194a6db6c145acafc0f3068b8ce1fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:26.423640 24923 raft_consensus.cc:729] T dc0e070513cc46ca82c1946423ad91c5 P 6c194a6db6c145acafc0f3068b8ce1fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c194a6db6c145acafc0f3068b8ce1fd, State: Initialized, Role: FOLLOWER
I20240502 21:27:26.424232 24923 consensus_queue.cc:260] T dc0e070513cc46ca82c1946423ad91c5 P 6c194a6db6c145acafc0f3068b8ce1fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } }
I20240502 21:27:26.425171 24922 raft_consensus.cc:348] T adf80b9d71314010a7a4cf052fd8a511 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } }
I20240502 21:27:26.425544 24922 raft_consensus.cc:374] T adf80b9d71314010a7a4cf052fd8a511 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:26.425660 24922 raft_consensus.cc:729] T adf80b9d71314010a7a4cf052fd8a511 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 038c5d9d1d7a49c981c4c34fbc00d9e6, State: Initialized, Role: FOLLOWER
I20240502 21:27:26.425897 24923 ts_tablet_manager.cc:1398] T dc0e070513cc46ca82c1946423ad91c5 P 6c194a6db6c145acafc0f3068b8ce1fd: Time spent starting tablet: real 0.011s user 0.012s sys 0.000s
I20240502 21:27:26.426283 24923 tablet_bootstrap.cc:492] T adf80b9d71314010a7a4cf052fd8a511 P 6c194a6db6c145acafc0f3068b8ce1fd: Bootstrap starting.
I20240502 21:27:26.426263 24922 consensus_queue.cc:260] T adf80b9d71314010a7a4cf052fd8a511 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } }
I20240502 21:27:26.427894 24923 tablet_bootstrap.cc:654] T adf80b9d71314010a7a4cf052fd8a511 P 6c194a6db6c145acafc0f3068b8ce1fd: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:26.430708 24925 ts_tablet_manager.cc:1398] T adf80b9d71314010a7a4cf052fd8a511 P 6810a866decb4f5089f7d35c870e4204: Time spent starting tablet: real 0.015s user 0.006s sys 0.006s
I20240502 21:27:26.431154 24925 tablet_bootstrap.cc:492] T dc0e070513cc46ca82c1946423ad91c5 P 6810a866decb4f5089f7d35c870e4204: Bootstrap starting.
I20240502 21:27:26.432770 24925 tablet_bootstrap.cc:654] T dc0e070513cc46ca82c1946423ad91c5 P 6810a866decb4f5089f7d35c870e4204: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:26.433993 24923 tablet_bootstrap.cc:492] T adf80b9d71314010a7a4cf052fd8a511 P 6c194a6db6c145acafc0f3068b8ce1fd: No bootstrap required, opened a new log
I20240502 21:27:26.434132 24923 ts_tablet_manager.cc:1367] T adf80b9d71314010a7a4cf052fd8a511 P 6c194a6db6c145acafc0f3068b8ce1fd: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20240502 21:27:26.434137 24922 ts_tablet_manager.cc:1398] T adf80b9d71314010a7a4cf052fd8a511 P 038c5d9d1d7a49c981c4c34fbc00d9e6: Time spent starting tablet: real 0.018s user 0.009s sys 0.008s
I20240502 21:27:26.434487 24922 tablet_bootstrap.cc:492] T d79eec4f59634df480161815d888bcac P 038c5d9d1d7a49c981c4c34fbc00d9e6: Bootstrap starting.
I20240502 21:27:26.434998 24923 raft_consensus.cc:348] T adf80b9d71314010a7a4cf052fd8a511 P 6c194a6db6c145acafc0f3068b8ce1fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } }
I20240502 21:27:26.435472 24923 raft_consensus.cc:374] T adf80b9d71314010a7a4cf052fd8a511 P 6c194a6db6c145acafc0f3068b8ce1fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:26.435567 24923 raft_consensus.cc:729] T adf80b9d71314010a7a4cf052fd8a511 P 6c194a6db6c145acafc0f3068b8ce1fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c194a6db6c145acafc0f3068b8ce1fd, State: Initialized, Role: FOLLOWER
I20240502 21:27:26.435779 24923 consensus_queue.cc:260] T adf80b9d71314010a7a4cf052fd8a511 P 6c194a6db6c145acafc0f3068b8ce1fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } }
I20240502 21:27:26.435943 24922 tablet_bootstrap.cc:654] T d79eec4f59634df480161815d888bcac P 038c5d9d1d7a49c981c4c34fbc00d9e6: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:26.436251 24923 ts_tablet_manager.cc:1398] T adf80b9d71314010a7a4cf052fd8a511 P 6c194a6db6c145acafc0f3068b8ce1fd: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:27:26.436558 24923 tablet_bootstrap.cc:492] T d79eec4f59634df480161815d888bcac P 6c194a6db6c145acafc0f3068b8ce1fd: Bootstrap starting.
I20240502 21:27:26.437991 24923 tablet_bootstrap.cc:654] T d79eec4f59634df480161815d888bcac P 6c194a6db6c145acafc0f3068b8ce1fd: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:26.439997 24923 tablet_bootstrap.cc:492] T d79eec4f59634df480161815d888bcac P 6c194a6db6c145acafc0f3068b8ce1fd: No bootstrap required, opened a new log
I20240502 21:27:26.440146 24923 ts_tablet_manager.cc:1367] T d79eec4f59634df480161815d888bcac P 6c194a6db6c145acafc0f3068b8ce1fd: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 21:27:26.440831 24923 raft_consensus.cc:348] T d79eec4f59634df480161815d888bcac P 6c194a6db6c145acafc0f3068b8ce1fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } }
I20240502 21:27:26.441056 24923 raft_consensus.cc:374] T d79eec4f59634df480161815d888bcac P 6c194a6db6c145acafc0f3068b8ce1fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:26.441128 24923 raft_consensus.cc:729] T d79eec4f59634df480161815d888bcac P 6c194a6db6c145acafc0f3068b8ce1fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c194a6db6c145acafc0f3068b8ce1fd, State: Initialized, Role: FOLLOWER
I20240502 21:27:26.441356 24923 consensus_queue.cc:260] T d79eec4f59634df480161815d888bcac P 6c194a6db6c145acafc0f3068b8ce1fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } }
I20240502 21:27:26.441870 24923 ts_tablet_manager.cc:1398] T d79eec4f59634df480161815d888bcac P 6c194a6db6c145acafc0f3068b8ce1fd: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 21:27:26.442443 24925 tablet_bootstrap.cc:492] T dc0e070513cc46ca82c1946423ad91c5 P 6810a866decb4f5089f7d35c870e4204: No bootstrap required, opened a new log
I20240502 21:27:26.442576 24925 ts_tablet_manager.cc:1367] T dc0e070513cc46ca82c1946423ad91c5 P 6810a866decb4f5089f7d35c870e4204: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.004s
I20240502 21:27:26.443142 24922 tablet_bootstrap.cc:492] T d79eec4f59634df480161815d888bcac P 038c5d9d1d7a49c981c4c34fbc00d9e6: No bootstrap required, opened a new log
I20240502 21:27:26.443282 24922 ts_tablet_manager.cc:1367] T d79eec4f59634df480161815d888bcac P 038c5d9d1d7a49c981c4c34fbc00d9e6: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.007s
I20240502 21:27:26.443332 24925 raft_consensus.cc:348] T dc0e070513cc46ca82c1946423ad91c5 P 6810a866decb4f5089f7d35c870e4204 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } }
I20240502 21:27:26.443506 24925 raft_consensus.cc:374] T dc0e070513cc46ca82c1946423ad91c5 P 6810a866decb4f5089f7d35c870e4204 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:26.443601 24925 raft_consensus.cc:729] T dc0e070513cc46ca82c1946423ad91c5 P 6810a866decb4f5089f7d35c870e4204 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6810a866decb4f5089f7d35c870e4204, State: Initialized, Role: FOLLOWER
I20240502 21:27:26.443814 24925 consensus_queue.cc:260] T dc0e070513cc46ca82c1946423ad91c5 P 6810a866decb4f5089f7d35c870e4204 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } }
I20240502 21:27:26.444331 24925 ts_tablet_manager.cc:1398] T dc0e070513cc46ca82c1946423ad91c5 P 6810a866decb4f5089f7d35c870e4204: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 21:27:26.444329 24922 raft_consensus.cc:348] T d79eec4f59634df480161815d888bcac P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } }
I20240502 21:27:26.444571 24922 raft_consensus.cc:374] T d79eec4f59634df480161815d888bcac P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:26.444646 24922 raft_consensus.cc:729] T d79eec4f59634df480161815d888bcac P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 038c5d9d1d7a49c981c4c34fbc00d9e6, State: Initialized, Role: FOLLOWER
I20240502 21:27:26.444824 24922 consensus_queue.cc:260] T d79eec4f59634df480161815d888bcac P 038c5d9d1d7a49c981c4c34fbc00d9e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } }
I20240502 21:27:26.445241 24922 ts_tablet_manager.cc:1398] T d79eec4f59634df480161815d888bcac P 038c5d9d1d7a49c981c4c34fbc00d9e6: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
W20240502 21:27:26.467123 24232 tablet.cc:2369] T d79eec4f59634df480161815d888bcac P 009d3e7e8a8b4e2497de3a73d01ced49: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240502 21:27:26.523267 24098 tablet.cc:2369] T adf80b9d71314010a7a4cf052fd8a511 P 6c194a6db6c145acafc0f3068b8ce1fd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 21:27:26.526859 24931 raft_consensus.cc:482] T adf80b9d71314010a7a4cf052fd8a511 P 6810a866decb4f5089f7d35c870e4204 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:26.527097 24931 raft_consensus.cc:504] T adf80b9d71314010a7a4cf052fd8a511 P 6810a866decb4f5089f7d35c870e4204 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } }
I20240502 21:27:26.528746 24931 leader_election.cc:288] T adf80b9d71314010a7a4cf052fd8a511 P 6810a866decb4f5089f7d35c870e4204 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 038c5d9d1d7a49c981c4c34fbc00d9e6 (127.15.255.132:46813), 6c194a6db6c145acafc0f3068b8ce1fd (127.15.255.130:41235)
I20240502 21:27:26.530551 24932 raft_consensus.cc:482] T dc0e070513cc46ca82c1946423ad91c5 P 6c194a6db6c145acafc0f3068b8ce1fd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:26.530753 24932 raft_consensus.cc:504] T dc0e070513cc46ca82c1946423ad91c5 P 6c194a6db6c145acafc0f3068b8ce1fd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } }
I20240502 21:27:26.532702 24932 leader_election.cc:288] T dc0e070513cc46ca82c1946423ad91c5 P 6c194a6db6c145acafc0f3068b8ce1fd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 009d3e7e8a8b4e2497de3a73d01ced49 (127.15.255.131:37895), 6810a866decb4f5089f7d35c870e4204 (127.15.255.129:40435)
I20240502 21:27:26.537187 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adf80b9d71314010a7a4cf052fd8a511" candidate_uuid: "6810a866decb4f5089f7d35c870e4204" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" is_pre_election: true
I20240502 21:27:26.537741 24317 raft_consensus.cc:2454] T adf80b9d71314010a7a4cf052fd8a511 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6810a866decb4f5089f7d35c870e4204 in term 0.
I20240502 21:27:26.537798 24049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adf80b9d71314010a7a4cf052fd8a511" candidate_uuid: "6810a866decb4f5089f7d35c870e4204" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" is_pre_election: true
I20240502 21:27:26.538259 24049 raft_consensus.cc:2454] T adf80b9d71314010a7a4cf052fd8a511 P 6c194a6db6c145acafc0f3068b8ce1fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6810a866decb4f5089f7d35c870e4204 in term 0.
I20240502 21:27:26.538424 23849 leader_election.cc:302] T adf80b9d71314010a7a4cf052fd8a511 P 6810a866decb4f5089f7d35c870e4204 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 038c5d9d1d7a49c981c4c34fbc00d9e6, 6810a866decb4f5089f7d35c870e4204; no voters:
I20240502 21:27:26.538867 24931 raft_consensus.cc:2781] T adf80b9d71314010a7a4cf052fd8a511 P 6810a866decb4f5089f7d35c870e4204 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:26.539016 24931 raft_consensus.cc:482] T adf80b9d71314010a7a4cf052fd8a511 P 6810a866decb4f5089f7d35c870e4204 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:26.539126 24931 raft_consensus.cc:3037] T adf80b9d71314010a7a4cf052fd8a511 P 6810a866decb4f5089f7d35c870e4204 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:26.541810 24931 raft_consensus.cc:504] T adf80b9d71314010a7a4cf052fd8a511 P 6810a866decb4f5089f7d35c870e4204 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } }
I20240502 21:27:26.542016 23916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc0e070513cc46ca82c1946423ad91c5" candidate_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6810a866decb4f5089f7d35c870e4204" is_pre_election: true
I20240502 21:27:26.542106 24185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc0e070513cc46ca82c1946423ad91c5" candidate_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" is_pre_election: true
I20240502 21:27:26.542402 23916 raft_consensus.cc:2454] T dc0e070513cc46ca82c1946423ad91c5 P 6810a866decb4f5089f7d35c870e4204 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6c194a6db6c145acafc0f3068b8ce1fd in term 0.
I20240502 21:27:26.542559 24931 leader_election.cc:288] T adf80b9d71314010a7a4cf052fd8a511 P 6810a866decb4f5089f7d35c870e4204 [CANDIDATE]: Term 1 election: Requested vote from peers 038c5d9d1d7a49c981c4c34fbc00d9e6 (127.15.255.132:46813), 6c194a6db6c145acafc0f3068b8ce1fd (127.15.255.130:41235)
I20240502 21:27:26.542640 24185 raft_consensus.cc:2454] T dc0e070513cc46ca82c1946423ad91c5 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6c194a6db6c145acafc0f3068b8ce1fd in term 0.
I20240502 21:27:26.542992 24049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adf80b9d71314010a7a4cf052fd8a511" candidate_uuid: "6810a866decb4f5089f7d35c870e4204" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c194a6db6c145acafc0f3068b8ce1fd"
I20240502 21:27:26.543046 23984 leader_election.cc:302] T dc0e070513cc46ca82c1946423ad91c5 P 6c194a6db6c145acafc0f3068b8ce1fd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6810a866decb4f5089f7d35c870e4204, 6c194a6db6c145acafc0f3068b8ce1fd; no voters:
I20240502 21:27:26.543193 24049 raft_consensus.cc:3037] T adf80b9d71314010a7a4cf052fd8a511 P 6c194a6db6c145acafc0f3068b8ce1fd [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:26.543242 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adf80b9d71314010a7a4cf052fd8a511" candidate_uuid: "6810a866decb4f5089f7d35c870e4204" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6"
I20240502 21:27:26.543418 24317 raft_consensus.cc:3037] T adf80b9d71314010a7a4cf052fd8a511 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:26.544565 24932 raft_consensus.cc:2781] T dc0e070513cc46ca82c1946423ad91c5 P 6c194a6db6c145acafc0f3068b8ce1fd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:26.544711 24932 raft_consensus.cc:482] T dc0e070513cc46ca82c1946423ad91c5 P 6c194a6db6c145acafc0f3068b8ce1fd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:26.544807 24932 raft_consensus.cc:3037] T dc0e070513cc46ca82c1946423ad91c5 P 6c194a6db6c145acafc0f3068b8ce1fd [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:26.545485 24317 raft_consensus.cc:2454] T adf80b9d71314010a7a4cf052fd8a511 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6810a866decb4f5089f7d35c870e4204 in term 1.
I20240502 21:27:26.545812 24049 raft_consensus.cc:2454] T adf80b9d71314010a7a4cf052fd8a511 P 6c194a6db6c145acafc0f3068b8ce1fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6810a866decb4f5089f7d35c870e4204 in term 1.
I20240502 21:27:26.545855 23849 leader_election.cc:302] T adf80b9d71314010a7a4cf052fd8a511 P 6810a866decb4f5089f7d35c870e4204 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 038c5d9d1d7a49c981c4c34fbc00d9e6, 6810a866decb4f5089f7d35c870e4204; no voters:
I20240502 21:27:26.546150 24931 raft_consensus.cc:2781] T adf80b9d71314010a7a4cf052fd8a511 P 6810a866decb4f5089f7d35c870e4204 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:26.546638 24931 raft_consensus.cc:686] T adf80b9d71314010a7a4cf052fd8a511 P 6810a866decb4f5089f7d35c870e4204 [term 1 LEADER]: Becoming Leader. State: Replica: 6810a866decb4f5089f7d35c870e4204, State: Running, Role: LEADER
I20240502 21:27:26.547075 24931 consensus_queue.cc:237] T adf80b9d71314010a7a4cf052fd8a511 P 6810a866decb4f5089f7d35c870e4204 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } }
I20240502 21:27:26.547242 24932 raft_consensus.cc:504] T dc0e070513cc46ca82c1946423ad91c5 P 6c194a6db6c145acafc0f3068b8ce1fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } }
I20240502 21:27:26.547849 24932 leader_election.cc:288] T dc0e070513cc46ca82c1946423ad91c5 P 6c194a6db6c145acafc0f3068b8ce1fd [CANDIDATE]: Term 1 election: Requested vote from peers 009d3e7e8a8b4e2497de3a73d01ced49 (127.15.255.131:37895), 6810a866decb4f5089f7d35c870e4204 (127.15.255.129:40435)
I20240502 21:27:26.548178 24185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc0e070513cc46ca82c1946423ad91c5" candidate_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "009d3e7e8a8b4e2497de3a73d01ced49"
I20240502 21:27:26.548336 24185 raft_consensus.cc:3037] T dc0e070513cc46ca82c1946423ad91c5 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:26.548321 23916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc0e070513cc46ca82c1946423ad91c5" candidate_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6810a866decb4f5089f7d35c870e4204"
I20240502 21:27:26.548514 23916 raft_consensus.cc:3037] T dc0e070513cc46ca82c1946423ad91c5 P 6810a866decb4f5089f7d35c870e4204 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:26.550336 24185 raft_consensus.cc:2454] T dc0e070513cc46ca82c1946423ad91c5 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c194a6db6c145acafc0f3068b8ce1fd in term 1.
I20240502 21:27:26.550451 23916 raft_consensus.cc:2454] T dc0e070513cc46ca82c1946423ad91c5 P 6810a866decb4f5089f7d35c870e4204 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c194a6db6c145acafc0f3068b8ce1fd in term 1.
I20240502 21:27:26.550762 23982 leader_election.cc:302] T dc0e070513cc46ca82c1946423ad91c5 P 6c194a6db6c145acafc0f3068b8ce1fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 009d3e7e8a8b4e2497de3a73d01ced49, 6c194a6db6c145acafc0f3068b8ce1fd; no voters:
I20240502 21:27:26.551056 24932 raft_consensus.cc:2781] T dc0e070513cc46ca82c1946423ad91c5 P 6c194a6db6c145acafc0f3068b8ce1fd [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:26.551466 24932 raft_consensus.cc:686] T dc0e070513cc46ca82c1946423ad91c5 P 6c194a6db6c145acafc0f3068b8ce1fd [term 1 LEADER]: Becoming Leader. State: Replica: 6c194a6db6c145acafc0f3068b8ce1fd, State: Running, Role: LEADER
I20240502 21:27:26.551923 24932 consensus_queue.cc:237] T dc0e070513cc46ca82c1946423ad91c5 P 6c194a6db6c145acafc0f3068b8ce1fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } }
I20240502 21:27:26.555270 23771 catalog_manager.cc:5484] T adf80b9d71314010a7a4cf052fd8a511 P 6810a866decb4f5089f7d35c870e4204 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6810a866decb4f5089f7d35c870e4204 (127.15.255.129). New cstate: current_term: 1 leader_uuid: "6810a866decb4f5089f7d35c870e4204" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:26.556249 23772 catalog_manager.cc:5484] T dc0e070513cc46ca82c1946423ad91c5 P 6c194a6db6c145acafc0f3068b8ce1fd reported cstate change: term changed from 0 to 1, leader changed from <none> to 6c194a6db6c145acafc0f3068b8ce1fd (127.15.255.130). New cstate: current_term: 1 leader_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } health_report { overall_health: HEALTHY } } }
W20240502 21:27:26.581609 23963 tablet.cc:2369] T adf80b9d71314010a7a4cf052fd8a511 P 6810a866decb4f5089f7d35c870e4204: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240502 21:27:26.649427 24364 tablet.cc:2369] T adf80b9d71314010a7a4cf052fd8a511 P 038c5d9d1d7a49c981c4c34fbc00d9e6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 21:27:26.650516 24926 raft_consensus.cc:482] T d79eec4f59634df480161815d888bcac P 009d3e7e8a8b4e2497de3a73d01ced49 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:26.650696 24926 raft_consensus.cc:504] T d79eec4f59634df480161815d888bcac P 009d3e7e8a8b4e2497de3a73d01ced49 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } }
I20240502 21:27:26.652221 24926 leader_election.cc:288] T d79eec4f59634df480161815d888bcac P 009d3e7e8a8b4e2497de3a73d01ced49 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6c194a6db6c145acafc0f3068b8ce1fd (127.15.255.130:41235), 038c5d9d1d7a49c981c4c34fbc00d9e6 (127.15.255.132:46813)
I20240502 21:27:26.658465 24049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d79eec4f59634df480161815d888bcac" candidate_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" is_pre_election: true
I20240502 21:27:26.658578 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d79eec4f59634df480161815d888bcac" candidate_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" is_pre_election: true
I20240502 21:27:26.658735 24049 raft_consensus.cc:2454] T d79eec4f59634df480161815d888bcac P 6c194a6db6c145acafc0f3068b8ce1fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 009d3e7e8a8b4e2497de3a73d01ced49 in term 0.
I20240502 21:27:26.658812 24317 raft_consensus.cc:2454] T d79eec4f59634df480161815d888bcac P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 009d3e7e8a8b4e2497de3a73d01ced49 in term 0.
I20240502 21:27:26.659207 24118 leader_election.cc:302] T d79eec4f59634df480161815d888bcac P 009d3e7e8a8b4e2497de3a73d01ced49 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 009d3e7e8a8b4e2497de3a73d01ced49, 6c194a6db6c145acafc0f3068b8ce1fd; no voters:
I20240502 21:27:26.659577 24926 raft_consensus.cc:2781] T d79eec4f59634df480161815d888bcac P 009d3e7e8a8b4e2497de3a73d01ced49 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:26.659688 24926 raft_consensus.cc:482] T d79eec4f59634df480161815d888bcac P 009d3e7e8a8b4e2497de3a73d01ced49 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:26.659775 24926 raft_consensus.cc:3037] T d79eec4f59634df480161815d888bcac P 009d3e7e8a8b4e2497de3a73d01ced49 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:26.661759 24926 raft_consensus.cc:504] T d79eec4f59634df480161815d888bcac P 009d3e7e8a8b4e2497de3a73d01ced49 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } }
I20240502 21:27:26.662290 24926 leader_election.cc:288] T d79eec4f59634df480161815d888bcac P 009d3e7e8a8b4e2497de3a73d01ced49 [CANDIDATE]: Term 1 election: Requested vote from peers 6c194a6db6c145acafc0f3068b8ce1fd (127.15.255.130:41235), 038c5d9d1d7a49c981c4c34fbc00d9e6 (127.15.255.132:46813)
I20240502 21:27:26.662657 24049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d79eec4f59634df480161815d888bcac" candidate_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c194a6db6c145acafc0f3068b8ce1fd"
I20240502 21:27:26.662710 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d79eec4f59634df480161815d888bcac" candidate_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6"
I20240502 21:27:26.662840 24049 raft_consensus.cc:3037] T d79eec4f59634df480161815d888bcac P 6c194a6db6c145acafc0f3068b8ce1fd [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:26.662921 24317 raft_consensus.cc:3037] T d79eec4f59634df480161815d888bcac P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:26.664819 24049 raft_consensus.cc:2454] T d79eec4f59634df480161815d888bcac P 6c194a6db6c145acafc0f3068b8ce1fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 009d3e7e8a8b4e2497de3a73d01ced49 in term 1.
I20240502 21:27:26.664989 24317 raft_consensus.cc:2454] T d79eec4f59634df480161815d888bcac P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 009d3e7e8a8b4e2497de3a73d01ced49 in term 1.
I20240502 21:27:26.665314 24118 leader_election.cc:302] T d79eec4f59634df480161815d888bcac P 009d3e7e8a8b4e2497de3a73d01ced49 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 009d3e7e8a8b4e2497de3a73d01ced49, 6c194a6db6c145acafc0f3068b8ce1fd; no voters:
I20240502 21:27:26.665592 24926 raft_consensus.cc:2781] T d79eec4f59634df480161815d888bcac P 009d3e7e8a8b4e2497de3a73d01ced49 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:26.665962 24926 raft_consensus.cc:686] T d79eec4f59634df480161815d888bcac P 009d3e7e8a8b4e2497de3a73d01ced49 [term 1 LEADER]: Becoming Leader. State: Replica: 009d3e7e8a8b4e2497de3a73d01ced49, State: Running, Role: LEADER
I20240502 21:27:26.666368 24926 consensus_queue.cc:237] T d79eec4f59634df480161815d888bcac P 009d3e7e8a8b4e2497de3a73d01ced49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } }
I20240502 21:27:26.670539 23772 catalog_manager.cc:5484] T d79eec4f59634df480161815d888bcac P 009d3e7e8a8b4e2497de3a73d01ced49 reported cstate change: term changed from 0 to 1, leader changed from <none> to 009d3e7e8a8b4e2497de3a73d01ced49 (127.15.255.131). New cstate: current_term: 1 leader_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:27.031002 24932 consensus_queue.cc:1035] T dc0e070513cc46ca82c1946423ad91c5 P 6c194a6db6c145acafc0f3068b8ce1fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:27.033048 24931 consensus_queue.cc:1035] T adf80b9d71314010a7a4cf052fd8a511 P 6810a866decb4f5089f7d35c870e4204 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:27.041895 24931 consensus_queue.cc:1035] T adf80b9d71314010a7a4cf052fd8a511 P 6810a866decb4f5089f7d35c870e4204 [LEADER]: Connected to new peer: Peer: permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:27.061623 24943 consensus_queue.cc:1035] T dc0e070513cc46ca82c1946423ad91c5 P 6c194a6db6c145acafc0f3068b8ce1fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 21:27:27.076025 24951 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:27:27.076314 24951 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:27:27.083518 24951 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_0'
I20240502 21:27:27.239886 24317 raft_consensus.cc:1261] T d79eec4f59634df480161815d888bcac P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 1 FOLLOWER]: Refusing update from remote peer 009d3e7e8a8b4e2497de3a73d01ced49: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:27:27.240957 24926 consensus_queue.cc:1035] T d79eec4f59634df480161815d888bcac P 009d3e7e8a8b4e2497de3a73d01ced49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:27.266479 24048 raft_consensus.cc:1261] T d79eec4f59634df480161815d888bcac P 6c194a6db6c145acafc0f3068b8ce1fd [term 1 FOLLOWER]: Refusing update from remote peer 009d3e7e8a8b4e2497de3a73d01ced49: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:27:27.267102 24926 consensus_queue.cc:1035] T d79eec4f59634df480161815d888bcac P 009d3e7e8a8b4e2497de3a73d01ced49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:27.316511 24999 mvcc.cc:205] Tried to move back new op lower bound from 7023350772686159872 to 7023350770348687360. Current Snapshot: MvccSnapshot[applied={T|T < 7023350772686159872}]
INSERT report
rows total: 2000
time total: 208.341 ms
time per row: 0.104171 ms
I20240502 21:27:27.430859 23766 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41618:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 21:27:27.439252 24164 tablet_service.cc:1467] Processing CreateTablet for tablet 0b3c514213bb40ca98955d6409a3b372 (DEFAULT_TABLE table=rebalance_test_table_1 [id=e962de13fbc44d1986b1e1ee191d22e2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:27.439584 23895 tablet_service.cc:1467] Processing CreateTablet for tablet 0b3c514213bb40ca98955d6409a3b372 (DEFAULT_TABLE table=rebalance_test_table_1 [id=e962de13fbc44d1986b1e1ee191d22e2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:27.439735 24164 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b3c514213bb40ca98955d6409a3b372. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:27.439863 24296 tablet_service.cc:1467] Processing CreateTablet for tablet 0b3c514213bb40ca98955d6409a3b372 (DEFAULT_TABLE table=rebalance_test_table_1 [id=e962de13fbc44d1986b1e1ee191d22e2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:27.439992 23895 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b3c514213bb40ca98955d6409a3b372. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:27.440238 24296 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b3c514213bb40ca98955d6409a3b372. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:27.441574 24163 tablet_service.cc:1467] Processing CreateTablet for tablet 4eebf4508766421eb6c547a0fbbc28b1 (DEFAULT_TABLE table=rebalance_test_table_1 [id=e962de13fbc44d1986b1e1ee191d22e2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:27.441936 24163 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4eebf4508766421eb6c547a0fbbc28b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:27.444888 24162 tablet_service.cc:1467] Processing CreateTablet for tablet 85e77d5faa8346cd8a99dcbcdf75f250 (DEFAULT_TABLE table=rebalance_test_table_1 [id=e962de13fbc44d1986b1e1ee191d22e2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:27.445147 24028 tablet_service.cc:1467] Processing CreateTablet for tablet 85e77d5faa8346cd8a99dcbcdf75f250 (DEFAULT_TABLE table=rebalance_test_table_1 [id=e962de13fbc44d1986b1e1ee191d22e2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:27.445305 24162 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85e77d5faa8346cd8a99dcbcdf75f250. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:27.445571 24028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85e77d5faa8346cd8a99dcbcdf75f250. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:27.446177 24297 tablet_service.cc:1467] Processing CreateTablet for tablet 4eebf4508766421eb6c547a0fbbc28b1 (DEFAULT_TABLE table=rebalance_test_table_1 [id=e962de13fbc44d1986b1e1ee191d22e2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:27.446548 24297 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4eebf4508766421eb6c547a0fbbc28b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:27.451930 23896 tablet_service.cc:1467] Processing CreateTablet for tablet 4eebf4508766421eb6c547a0fbbc28b1 (DEFAULT_TABLE table=rebalance_test_table_1 [id=e962de13fbc44d1986b1e1ee191d22e2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:27.452025 23894 tablet_service.cc:1467] Processing CreateTablet for tablet 85e77d5faa8346cd8a99dcbcdf75f250 (DEFAULT_TABLE table=rebalance_test_table_1 [id=e962de13fbc44d1986b1e1ee191d22e2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:27.452378 23896 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4eebf4508766421eb6c547a0fbbc28b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:27.452616 23894 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85e77d5faa8346cd8a99dcbcdf75f250. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:27.461076 25024 tablet_bootstrap.cc:492] T 0b3c514213bb40ca98955d6409a3b372 P 038c5d9d1d7a49c981c4c34fbc00d9e6: Bootstrap starting.
I20240502 21:27:27.462810 25024 tablet_bootstrap.cc:654] T 0b3c514213bb40ca98955d6409a3b372 P 038c5d9d1d7a49c981c4c34fbc00d9e6: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:27.463173 25023 tablet_bootstrap.cc:492] T 0b3c514213bb40ca98955d6409a3b372 P 009d3e7e8a8b4e2497de3a73d01ced49: Bootstrap starting.
I20240502 21:27:27.464211 25025 tablet_bootstrap.cc:492] T 4eebf4508766421eb6c547a0fbbc28b1 P 6810a866decb4f5089f7d35c870e4204: Bootstrap starting.
I20240502 21:27:27.465458 25023 tablet_bootstrap.cc:654] T 0b3c514213bb40ca98955d6409a3b372 P 009d3e7e8a8b4e2497de3a73d01ced49: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:27.466044 25025 tablet_bootstrap.cc:654] T 4eebf4508766421eb6c547a0fbbc28b1 P 6810a866decb4f5089f7d35c870e4204: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:27.466920 25026 tablet_bootstrap.cc:492] T 85e77d5faa8346cd8a99dcbcdf75f250 P 6c194a6db6c145acafc0f3068b8ce1fd: Bootstrap starting.
I20240502 21:27:27.468777 25026 tablet_bootstrap.cc:654] T 85e77d5faa8346cd8a99dcbcdf75f250 P 6c194a6db6c145acafc0f3068b8ce1fd: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:27.469267 25025 tablet_bootstrap.cc:492] T 4eebf4508766421eb6c547a0fbbc28b1 P 6810a866decb4f5089f7d35c870e4204: No bootstrap required, opened a new log
I20240502 21:27:27.469316 25023 tablet_bootstrap.cc:492] T 0b3c514213bb40ca98955d6409a3b372 P 009d3e7e8a8b4e2497de3a73d01ced49: No bootstrap required, opened a new log
I20240502 21:27:27.469417 25025 ts_tablet_manager.cc:1367] T 4eebf4508766421eb6c547a0fbbc28b1 P 6810a866decb4f5089f7d35c870e4204: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20240502 21:27:27.469442 25023 ts_tablet_manager.cc:1367] T 0b3c514213bb40ca98955d6409a3b372 P 009d3e7e8a8b4e2497de3a73d01ced49: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.003s
I20240502 21:27:27.470050 25024 tablet_bootstrap.cc:492] T 0b3c514213bb40ca98955d6409a3b372 P 038c5d9d1d7a49c981c4c34fbc00d9e6: No bootstrap required, opened a new log
I20240502 21:27:27.470203 25023 raft_consensus.cc:348] T 0b3c514213bb40ca98955d6409a3b372 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } }
I20240502 21:27:27.470232 25025 raft_consensus.cc:348] T 4eebf4508766421eb6c547a0fbbc28b1 P 6810a866decb4f5089f7d35c870e4204 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } }
I20240502 21:27:27.470397 25023 raft_consensus.cc:374] T 0b3c514213bb40ca98955d6409a3b372 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:27.470427 25025 raft_consensus.cc:374] T 4eebf4508766421eb6c547a0fbbc28b1 P 6810a866decb4f5089f7d35c870e4204 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:27.470490 25023 raft_consensus.cc:729] T 0b3c514213bb40ca98955d6409a3b372 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 009d3e7e8a8b4e2497de3a73d01ced49, State: Initialized, Role: FOLLOWER
I20240502 21:27:27.470504 25025 raft_consensus.cc:729] T 4eebf4508766421eb6c547a0fbbc28b1 P 6810a866decb4f5089f7d35c870e4204 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6810a866decb4f5089f7d35c870e4204, State: Initialized, Role: FOLLOWER
I20240502 21:27:27.470732 25025 consensus_queue.cc:260] T 4eebf4508766421eb6c547a0fbbc28b1 P 6810a866decb4f5089f7d35c870e4204 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } }
I20240502 21:27:27.470734 25023 consensus_queue.cc:260] T 0b3c514213bb40ca98955d6409a3b372 P 009d3e7e8a8b4e2497de3a73d01ced49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } }
I20240502 21:27:27.471269 25023 ts_tablet_manager.cc:1398] T 0b3c514213bb40ca98955d6409a3b372 P 009d3e7e8a8b4e2497de3a73d01ced49: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 21:27:27.471392 25025 ts_tablet_manager.cc:1398] T 4eebf4508766421eb6c547a0fbbc28b1 P 6810a866decb4f5089f7d35c870e4204: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 21:27:27.471522 25023 tablet_bootstrap.cc:492] T 85e77d5faa8346cd8a99dcbcdf75f250 P 009d3e7e8a8b4e2497de3a73d01ced49: Bootstrap starting.
I20240502 21:27:27.471740 25025 tablet_bootstrap.cc:492] T 85e77d5faa8346cd8a99dcbcdf75f250 P 6810a866decb4f5089f7d35c870e4204: Bootstrap starting.
I20240502 21:27:27.472975 25023 tablet_bootstrap.cc:654] T 85e77d5faa8346cd8a99dcbcdf75f250 P 009d3e7e8a8b4e2497de3a73d01ced49: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:27.473186 25025 tablet_bootstrap.cc:654] T 85e77d5faa8346cd8a99dcbcdf75f250 P 6810a866decb4f5089f7d35c870e4204: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:27.474141 25024 ts_tablet_manager.cc:1367] T 0b3c514213bb40ca98955d6409a3b372 P 038c5d9d1d7a49c981c4c34fbc00d9e6: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.004s
I20240502 21:27:27.475209 25024 raft_consensus.cc:348] T 0b3c514213bb40ca98955d6409a3b372 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } }
I20240502 21:27:27.476501 25024 raft_consensus.cc:374] T 0b3c514213bb40ca98955d6409a3b372 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:27.476653 25024 raft_consensus.cc:729] T 0b3c514213bb40ca98955d6409a3b372 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 038c5d9d1d7a49c981c4c34fbc00d9e6, State: Initialized, Role: FOLLOWER
I20240502 21:27:27.478467 25024 consensus_queue.cc:260] T 0b3c514213bb40ca98955d6409a3b372 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } }
I20240502 21:27:27.479246 25024 ts_tablet_manager.cc:1398] T 0b3c514213bb40ca98955d6409a3b372 P 038c5d9d1d7a49c981c4c34fbc00d9e6: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20240502 21:27:27.479593 25024 tablet_bootstrap.cc:492] T 4eebf4508766421eb6c547a0fbbc28b1 P 038c5d9d1d7a49c981c4c34fbc00d9e6: Bootstrap starting.
I20240502 21:27:27.482417 25026 tablet_bootstrap.cc:492] T 85e77d5faa8346cd8a99dcbcdf75f250 P 6c194a6db6c145acafc0f3068b8ce1fd: No bootstrap required, opened a new log
I20240502 21:27:27.482564 25026 ts_tablet_manager.cc:1367] T 85e77d5faa8346cd8a99dcbcdf75f250 P 6c194a6db6c145acafc0f3068b8ce1fd: Time spent bootstrapping tablet: real 0.016s user 0.009s sys 0.000s
I20240502 21:27:27.483769 25026 raft_consensus.cc:348] T 85e77d5faa8346cd8a99dcbcdf75f250 P 6c194a6db6c145acafc0f3068b8ce1fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } }
I20240502 21:27:27.483985 25026 raft_consensus.cc:374] T 85e77d5faa8346cd8a99dcbcdf75f250 P 6c194a6db6c145acafc0f3068b8ce1fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:27.484071 25026 raft_consensus.cc:729] T 85e77d5faa8346cd8a99dcbcdf75f250 P 6c194a6db6c145acafc0f3068b8ce1fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c194a6db6c145acafc0f3068b8ce1fd, State: Initialized, Role: FOLLOWER
I20240502 21:27:27.484510 25024 tablet_bootstrap.cc:654] T 4eebf4508766421eb6c547a0fbbc28b1 P 038c5d9d1d7a49c981c4c34fbc00d9e6: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:27.484323 25026 consensus_queue.cc:260] T 85e77d5faa8346cd8a99dcbcdf75f250 P 6c194a6db6c145acafc0f3068b8ce1fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } }
I20240502 21:27:27.485349 25026 ts_tablet_manager.cc:1398] T 85e77d5faa8346cd8a99dcbcdf75f250 P 6c194a6db6c145acafc0f3068b8ce1fd: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 21:27:27.488044 25025 tablet_bootstrap.cc:492] T 85e77d5faa8346cd8a99dcbcdf75f250 P 6810a866decb4f5089f7d35c870e4204: No bootstrap required, opened a new log
I20240502 21:27:27.488210 25025 ts_tablet_manager.cc:1367] T 85e77d5faa8346cd8a99dcbcdf75f250 P 6810a866decb4f5089f7d35c870e4204: Time spent bootstrapping tablet: real 0.017s user 0.009s sys 0.007s
I20240502 21:27:27.488344 25024 tablet_bootstrap.cc:492] T 4eebf4508766421eb6c547a0fbbc28b1 P 038c5d9d1d7a49c981c4c34fbc00d9e6: No bootstrap required, opened a new log
I20240502 21:27:27.488523 25024 ts_tablet_manager.cc:1367] T 4eebf4508766421eb6c547a0fbbc28b1 P 038c5d9d1d7a49c981c4c34fbc00d9e6: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.003s
I20240502 21:27:27.488626 25023 tablet_bootstrap.cc:492] T 85e77d5faa8346cd8a99dcbcdf75f250 P 009d3e7e8a8b4e2497de3a73d01ced49: No bootstrap required, opened a new log
I20240502 21:27:27.488816 25023 ts_tablet_manager.cc:1367] T 85e77d5faa8346cd8a99dcbcdf75f250 P 009d3e7e8a8b4e2497de3a73d01ced49: Time spent bootstrapping tablet: real 0.017s user 0.013s sys 0.003s
I20240502 21:27:27.488993 25025 raft_consensus.cc:348] T 85e77d5faa8346cd8a99dcbcdf75f250 P 6810a866decb4f5089f7d35c870e4204 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } }
I20240502 21:27:27.489221 25025 raft_consensus.cc:374] T 85e77d5faa8346cd8a99dcbcdf75f250 P 6810a866decb4f5089f7d35c870e4204 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:27.489288 25025 raft_consensus.cc:729] T 85e77d5faa8346cd8a99dcbcdf75f250 P 6810a866decb4f5089f7d35c870e4204 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6810a866decb4f5089f7d35c870e4204, State: Initialized, Role: FOLLOWER
I20240502 21:27:27.489234 25024 raft_consensus.cc:348] T 4eebf4508766421eb6c547a0fbbc28b1 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } }
I20240502 21:27:27.489444 25024 raft_consensus.cc:374] T 4eebf4508766421eb6c547a0fbbc28b1 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:27.489527 25024 raft_consensus.cc:729] T 4eebf4508766421eb6c547a0fbbc28b1 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 038c5d9d1d7a49c981c4c34fbc00d9e6, State: Initialized, Role: FOLLOWER
I20240502 21:27:27.489522 25025 consensus_queue.cc:260] T 85e77d5faa8346cd8a99dcbcdf75f250 P 6810a866decb4f5089f7d35c870e4204 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } }
I20240502 21:27:27.489770 25024 consensus_queue.cc:260] T 4eebf4508766421eb6c547a0fbbc28b1 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } }
I20240502 21:27:27.490046 25025 ts_tablet_manager.cc:1398] T 85e77d5faa8346cd8a99dcbcdf75f250 P 6810a866decb4f5089f7d35c870e4204: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 21:27:27.490226 25024 ts_tablet_manager.cc:1398] T 4eebf4508766421eb6c547a0fbbc28b1 P 038c5d9d1d7a49c981c4c34fbc00d9e6: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 21:27:27.489761 25023 raft_consensus.cc:348] T 85e77d5faa8346cd8a99dcbcdf75f250 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } }
I20240502 21:27:27.490576 25023 raft_consensus.cc:374] T 85e77d5faa8346cd8a99dcbcdf75f250 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:27.490653 25023 raft_consensus.cc:729] T 85e77d5faa8346cd8a99dcbcdf75f250 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 009d3e7e8a8b4e2497de3a73d01ced49, State: Initialized, Role: FOLLOWER
I20240502 21:27:27.491079 25025 tablet_bootstrap.cc:492] T 0b3c514213bb40ca98955d6409a3b372 P 6810a866decb4f5089f7d35c870e4204: Bootstrap starting.
I20240502 21:27:27.491045 25023 consensus_queue.cc:260] T 85e77d5faa8346cd8a99dcbcdf75f250 P 009d3e7e8a8b4e2497de3a73d01ced49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } }
I20240502 21:27:27.491536 25023 ts_tablet_manager.cc:1398] T 85e77d5faa8346cd8a99dcbcdf75f250 P 009d3e7e8a8b4e2497de3a73d01ced49: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 21:27:27.491829 25023 tablet_bootstrap.cc:492] T 4eebf4508766421eb6c547a0fbbc28b1 P 009d3e7e8a8b4e2497de3a73d01ced49: Bootstrap starting.
I20240502 21:27:27.493218 25023 tablet_bootstrap.cc:654] T 4eebf4508766421eb6c547a0fbbc28b1 P 009d3e7e8a8b4e2497de3a73d01ced49: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:27.494912 25025 tablet_bootstrap.cc:654] T 0b3c514213bb40ca98955d6409a3b372 P 6810a866decb4f5089f7d35c870e4204: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:27.495234 25023 tablet_bootstrap.cc:492] T 4eebf4508766421eb6c547a0fbbc28b1 P 009d3e7e8a8b4e2497de3a73d01ced49: No bootstrap required, opened a new log
I20240502 21:27:27.495378 25023 ts_tablet_manager.cc:1367] T 4eebf4508766421eb6c547a0fbbc28b1 P 009d3e7e8a8b4e2497de3a73d01ced49: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240502 21:27:27.496124 25023 raft_consensus.cc:348] T 4eebf4508766421eb6c547a0fbbc28b1 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } }
I20240502 21:27:27.496309 25023 raft_consensus.cc:374] T 4eebf4508766421eb6c547a0fbbc28b1 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:27.496438 25023 raft_consensus.cc:729] T 4eebf4508766421eb6c547a0fbbc28b1 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 009d3e7e8a8b4e2497de3a73d01ced49, State: Initialized, Role: FOLLOWER
I20240502 21:27:27.496719 25025 tablet_bootstrap.cc:492] T 0b3c514213bb40ca98955d6409a3b372 P 6810a866decb4f5089f7d35c870e4204: No bootstrap required, opened a new log
I20240502 21:27:27.496661 25023 consensus_queue.cc:260] T 4eebf4508766421eb6c547a0fbbc28b1 P 009d3e7e8a8b4e2497de3a73d01ced49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } }
I20240502 21:27:27.496850 25025 ts_tablet_manager.cc:1367] T 0b3c514213bb40ca98955d6409a3b372 P 6810a866decb4f5089f7d35c870e4204: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.003s
I20240502 21:27:27.497603 25023 ts_tablet_manager.cc:1398] T 4eebf4508766421eb6c547a0fbbc28b1 P 009d3e7e8a8b4e2497de3a73d01ced49: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 21:27:27.497551 25025 raft_consensus.cc:348] T 0b3c514213bb40ca98955d6409a3b372 P 6810a866decb4f5089f7d35c870e4204 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } }
I20240502 21:27:27.497747 25025 raft_consensus.cc:374] T 0b3c514213bb40ca98955d6409a3b372 P 6810a866decb4f5089f7d35c870e4204 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:27.497835 25025 raft_consensus.cc:729] T 0b3c514213bb40ca98955d6409a3b372 P 6810a866decb4f5089f7d35c870e4204 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6810a866decb4f5089f7d35c870e4204, State: Initialized, Role: FOLLOWER
I20240502 21:27:27.498035 25025 consensus_queue.cc:260] T 0b3c514213bb40ca98955d6409a3b372 P 6810a866decb4f5089f7d35c870e4204 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } }
I20240502 21:27:27.498466 25025 ts_tablet_manager.cc:1398] T 0b3c514213bb40ca98955d6409a3b372 P 6810a866decb4f5089f7d35c870e4204: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 21:27:27.506934 25012 raft_consensus.cc:482] T 0b3c514213bb40ca98955d6409a3b372 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:27.507054 25012 raft_consensus.cc:504] T 0b3c514213bb40ca98955d6409a3b372 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } }
I20240502 21:27:27.507732 25012 leader_election.cc:288] T 0b3c514213bb40ca98955d6409a3b372 P 009d3e7e8a8b4e2497de3a73d01ced49 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6810a866decb4f5089f7d35c870e4204 (127.15.255.129:40435), 038c5d9d1d7a49c981c4c34fbc00d9e6 (127.15.255.132:46813)
I20240502 21:27:27.508206 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b3c514213bb40ca98955d6409a3b372" candidate_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" is_pre_election: true
I20240502 21:27:27.508462 24317 raft_consensus.cc:2454] T 0b3c514213bb40ca98955d6409a3b372 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 009d3e7e8a8b4e2497de3a73d01ced49 in term 0.
I20240502 21:27:27.508884 24116 leader_election.cc:302] T 0b3c514213bb40ca98955d6409a3b372 P 009d3e7e8a8b4e2497de3a73d01ced49 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 009d3e7e8a8b4e2497de3a73d01ced49, 038c5d9d1d7a49c981c4c34fbc00d9e6; no voters:
I20240502 21:27:27.509147 25012 raft_consensus.cc:2781] T 0b3c514213bb40ca98955d6409a3b372 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:27.509261 25012 raft_consensus.cc:482] T 0b3c514213bb40ca98955d6409a3b372 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:27.509353 25012 raft_consensus.cc:3037] T 0b3c514213bb40ca98955d6409a3b372 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:27.511456 25012 raft_consensus.cc:504] T 0b3c514213bb40ca98955d6409a3b372 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } }
I20240502 21:27:27.512014 25012 leader_election.cc:288] T 0b3c514213bb40ca98955d6409a3b372 P 009d3e7e8a8b4e2497de3a73d01ced49 [CANDIDATE]: Term 1 election: Requested vote from peers 6810a866decb4f5089f7d35c870e4204 (127.15.255.129:40435), 038c5d9d1d7a49c981c4c34fbc00d9e6 (127.15.255.132:46813)
I20240502 21:27:27.512403 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b3c514213bb40ca98955d6409a3b372" candidate_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6"
I20240502 21:27:27.512620 24317 raft_consensus.cc:3037] T 0b3c514213bb40ca98955d6409a3b372 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:27.513955 23916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b3c514213bb40ca98955d6409a3b372" candidate_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6810a866decb4f5089f7d35c870e4204" is_pre_election: true
I20240502 21:27:27.514036 23915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b3c514213bb40ca98955d6409a3b372" candidate_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6810a866decb4f5089f7d35c870e4204"
I20240502 21:27:27.514222 23916 raft_consensus.cc:2454] T 0b3c514213bb40ca98955d6409a3b372 P 6810a866decb4f5089f7d35c870e4204 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 009d3e7e8a8b4e2497de3a73d01ced49 in term 0.
I20240502 21:27:27.514792 24317 raft_consensus.cc:2454] T 0b3c514213bb40ca98955d6409a3b372 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 009d3e7e8a8b4e2497de3a73d01ced49 in term 1.
I20240502 21:27:27.515125 24116 leader_election.cc:302] T 0b3c514213bb40ca98955d6409a3b372 P 009d3e7e8a8b4e2497de3a73d01ced49 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 009d3e7e8a8b4e2497de3a73d01ced49, 038c5d9d1d7a49c981c4c34fbc00d9e6; no voters: 6810a866decb4f5089f7d35c870e4204
I20240502 21:27:27.515398 25012 raft_consensus.cc:2781] T 0b3c514213bb40ca98955d6409a3b372 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:27.515519 25012 raft_consensus.cc:686] T 0b3c514213bb40ca98955d6409a3b372 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 1 LEADER]: Becoming Leader. State: Replica: 009d3e7e8a8b4e2497de3a73d01ced49, State: Running, Role: LEADER
I20240502 21:27:27.515803 25012 consensus_queue.cc:237] T 0b3c514213bb40ca98955d6409a3b372 P 009d3e7e8a8b4e2497de3a73d01ced49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } }
I20240502 21:27:27.523264 23769 catalog_manager.cc:5484] T 0b3c514213bb40ca98955d6409a3b372 P 009d3e7e8a8b4e2497de3a73d01ced49 reported cstate change: term changed from 0 to 1, leader changed from <none> to 009d3e7e8a8b4e2497de3a73d01ced49 (127.15.255.131). New cstate: current_term: 1 leader_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:27.699013 25012 raft_consensus.cc:482] T 85e77d5faa8346cd8a99dcbcdf75f250 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:27.699218 25012 raft_consensus.cc:504] T 85e77d5faa8346cd8a99dcbcdf75f250 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } }
I20240502 21:27:27.699793 25012 leader_election.cc:288] T 85e77d5faa8346cd8a99dcbcdf75f250 P 009d3e7e8a8b4e2497de3a73d01ced49 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6c194a6db6c145acafc0f3068b8ce1fd (127.15.255.130:41235), 6810a866decb4f5089f7d35c870e4204 (127.15.255.129:40435)
I20240502 21:27:27.700352 24049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85e77d5faa8346cd8a99dcbcdf75f250" candidate_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" is_pre_election: true
I20240502 21:27:27.700393 23915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85e77d5faa8346cd8a99dcbcdf75f250" candidate_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6810a866decb4f5089f7d35c870e4204" is_pre_election: true
I20240502 21:27:27.700596 24049 raft_consensus.cc:2454] T 85e77d5faa8346cd8a99dcbcdf75f250 P 6c194a6db6c145acafc0f3068b8ce1fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 009d3e7e8a8b4e2497de3a73d01ced49 in term 0.
I20240502 21:27:27.700685 23915 raft_consensus.cc:2454] T 85e77d5faa8346cd8a99dcbcdf75f250 P 6810a866decb4f5089f7d35c870e4204 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 009d3e7e8a8b4e2497de3a73d01ced49 in term 0.
I20240502 21:27:27.701040 24118 leader_election.cc:302] T 85e77d5faa8346cd8a99dcbcdf75f250 P 009d3e7e8a8b4e2497de3a73d01ced49 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 009d3e7e8a8b4e2497de3a73d01ced49, 6c194a6db6c145acafc0f3068b8ce1fd; no voters:
I20240502 21:27:27.701333 25012 raft_consensus.cc:2781] T 85e77d5faa8346cd8a99dcbcdf75f250 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:27.701432 25012 raft_consensus.cc:482] T 85e77d5faa8346cd8a99dcbcdf75f250 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:27.701532 25012 raft_consensus.cc:3037] T 85e77d5faa8346cd8a99dcbcdf75f250 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:27.703390 25012 raft_consensus.cc:504] T 85e77d5faa8346cd8a99dcbcdf75f250 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } }
I20240502 21:27:27.703938 25012 leader_election.cc:288] T 85e77d5faa8346cd8a99dcbcdf75f250 P 009d3e7e8a8b4e2497de3a73d01ced49 [CANDIDATE]: Term 1 election: Requested vote from peers 6c194a6db6c145acafc0f3068b8ce1fd (127.15.255.130:41235), 6810a866decb4f5089f7d35c870e4204 (127.15.255.129:40435)
I20240502 21:27:27.704279 24049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85e77d5faa8346cd8a99dcbcdf75f250" candidate_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c194a6db6c145acafc0f3068b8ce1fd"
I20240502 21:27:27.704468 24049 raft_consensus.cc:3037] T 85e77d5faa8346cd8a99dcbcdf75f250 P 6c194a6db6c145acafc0f3068b8ce1fd [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:27.704401 23915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85e77d5faa8346cd8a99dcbcdf75f250" candidate_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6810a866decb4f5089f7d35c870e4204"
I20240502 21:27:27.704625 23915 raft_consensus.cc:3037] T 85e77d5faa8346cd8a99dcbcdf75f250 P 6810a866decb4f5089f7d35c870e4204 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:27.706408 24049 raft_consensus.cc:2454] T 85e77d5faa8346cd8a99dcbcdf75f250 P 6c194a6db6c145acafc0f3068b8ce1fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 009d3e7e8a8b4e2497de3a73d01ced49 in term 1.
I20240502 21:27:27.706647 23915 raft_consensus.cc:2454] T 85e77d5faa8346cd8a99dcbcdf75f250 P 6810a866decb4f5089f7d35c870e4204 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 009d3e7e8a8b4e2497de3a73d01ced49 in term 1.
I20240502 21:27:27.706732 24118 leader_election.cc:302] T 85e77d5faa8346cd8a99dcbcdf75f250 P 009d3e7e8a8b4e2497de3a73d01ced49 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 009d3e7e8a8b4e2497de3a73d01ced49, 6c194a6db6c145acafc0f3068b8ce1fd; no voters:
I20240502 21:27:27.707012 25012 raft_consensus.cc:2781] T 85e77d5faa8346cd8a99dcbcdf75f250 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:27.707190 25012 raft_consensus.cc:686] T 85e77d5faa8346cd8a99dcbcdf75f250 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 1 LEADER]: Becoming Leader. State: Replica: 009d3e7e8a8b4e2497de3a73d01ced49, State: Running, Role: LEADER
I20240502 21:27:27.707442 25012 consensus_queue.cc:237] T 85e77d5faa8346cd8a99dcbcdf75f250 P 009d3e7e8a8b4e2497de3a73d01ced49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } }
I20240502 21:27:27.709817 23769 catalog_manager.cc:5484] T 85e77d5faa8346cd8a99dcbcdf75f250 P 009d3e7e8a8b4e2497de3a73d01ced49 reported cstate change: term changed from 0 to 1, leader changed from <none> to 009d3e7e8a8b4e2497de3a73d01ced49 (127.15.255.131). New cstate: current_term: 1 leader_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } health_report { overall_health: HEALTHY } } }
I20240502 21:27:27.749578 24931 raft_consensus.cc:482] T 4eebf4508766421eb6c547a0fbbc28b1 P 6810a866decb4f5089f7d35c870e4204 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:27.749750 24931 raft_consensus.cc:504] T 4eebf4508766421eb6c547a0fbbc28b1 P 6810a866decb4f5089f7d35c870e4204 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } }
I20240502 21:27:27.750483 24931 leader_election.cc:288] T 4eebf4508766421eb6c547a0fbbc28b1 P 6810a866decb4f5089f7d35c870e4204 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 009d3e7e8a8b4e2497de3a73d01ced49 (127.15.255.131:37895), 038c5d9d1d7a49c981c4c34fbc00d9e6 (127.15.255.132:46813)
I20240502 21:27:27.751163 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4eebf4508766421eb6c547a0fbbc28b1" candidate_uuid: "6810a866decb4f5089f7d35c870e4204" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" is_pre_election: true
I20240502 21:27:27.751370 24317 raft_consensus.cc:2454] T 4eebf4508766421eb6c547a0fbbc28b1 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6810a866decb4f5089f7d35c870e4204 in term 0.
I20240502 21:27:27.751796 23849 leader_election.cc:302] T 4eebf4508766421eb6c547a0fbbc28b1 P 6810a866decb4f5089f7d35c870e4204 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 038c5d9d1d7a49c981c4c34fbc00d9e6, 6810a866decb4f5089f7d35c870e4204; no voters:
I20240502 21:27:27.752067 24931 raft_consensus.cc:2781] T 4eebf4508766421eb6c547a0fbbc28b1 P 6810a866decb4f5089f7d35c870e4204 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:27.752207 24931 raft_consensus.cc:482] T 4eebf4508766421eb6c547a0fbbc28b1 P 6810a866decb4f5089f7d35c870e4204 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:27.752343 24931 raft_consensus.cc:3037] T 4eebf4508766421eb6c547a0fbbc28b1 P 6810a866decb4f5089f7d35c870e4204 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:27.754434 24931 raft_consensus.cc:504] T 4eebf4508766421eb6c547a0fbbc28b1 P 6810a866decb4f5089f7d35c870e4204 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } }
I20240502 21:27:27.755002 24931 leader_election.cc:288] T 4eebf4508766421eb6c547a0fbbc28b1 P 6810a866decb4f5089f7d35c870e4204 [CANDIDATE]: Term 1 election: Requested vote from peers 009d3e7e8a8b4e2497de3a73d01ced49 (127.15.255.131:37895), 038c5d9d1d7a49c981c4c34fbc00d9e6 (127.15.255.132:46813)
I20240502 21:27:27.755523 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4eebf4508766421eb6c547a0fbbc28b1" candidate_uuid: "6810a866decb4f5089f7d35c870e4204" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6"
I20240502 21:27:27.755690 24317 raft_consensus.cc:3037] T 4eebf4508766421eb6c547a0fbbc28b1 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:27.756860 24185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4eebf4508766421eb6c547a0fbbc28b1" candidate_uuid: "6810a866decb4f5089f7d35c870e4204" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "009d3e7e8a8b4e2497de3a73d01ced49"
I20240502 21:27:27.756834 24184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4eebf4508766421eb6c547a0fbbc28b1" candidate_uuid: "6810a866decb4f5089f7d35c870e4204" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" is_pre_election: true
I20240502 21:27:27.757066 24185 raft_consensus.cc:3037] T 4eebf4508766421eb6c547a0fbbc28b1 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:27.757902 24317 raft_consensus.cc:2454] T 4eebf4508766421eb6c547a0fbbc28b1 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6810a866decb4f5089f7d35c870e4204 in term 1.
I20240502 21:27:27.758203 23849 leader_election.cc:302] T 4eebf4508766421eb6c547a0fbbc28b1 P 6810a866decb4f5089f7d35c870e4204 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 038c5d9d1d7a49c981c4c34fbc00d9e6, 6810a866decb4f5089f7d35c870e4204; no voters:
I20240502 21:27:27.758451 24931 raft_consensus.cc:2781] T 4eebf4508766421eb6c547a0fbbc28b1 P 6810a866decb4f5089f7d35c870e4204 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:27.758620 24931 raft_consensus.cc:686] T 4eebf4508766421eb6c547a0fbbc28b1 P 6810a866decb4f5089f7d35c870e4204 [term 1 LEADER]: Becoming Leader. State: Replica: 6810a866decb4f5089f7d35c870e4204, State: Running, Role: LEADER
I20240502 21:27:27.758890 24931 consensus_queue.cc:237] T 4eebf4508766421eb6c547a0fbbc28b1 P 6810a866decb4f5089f7d35c870e4204 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } }
I20240502 21:27:27.759220 24185 raft_consensus.cc:2454] T 4eebf4508766421eb6c547a0fbbc28b1 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6810a866decb4f5089f7d35c870e4204 in term 1.
I20240502 21:27:27.761878 23769 catalog_manager.cc:5484] T 4eebf4508766421eb6c547a0fbbc28b1 P 6810a866decb4f5089f7d35c870e4204 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6810a866decb4f5089f7d35c870e4204 (127.15.255.129). New cstate: current_term: 1 leader_uuid: "6810a866decb4f5089f7d35c870e4204" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:27.945272 23915 raft_consensus.cc:3037] T 0b3c514213bb40ca98955d6409a3b372 P 6810a866decb4f5089f7d35c870e4204 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:27.947853 25012 consensus_queue.cc:1035] T 0b3c514213bb40ca98955d6409a3b372 P 009d3e7e8a8b4e2497de3a73d01ced49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:27.953533 25012 consensus_queue.cc:1035] T 0b3c514213bb40ca98955d6409a3b372 P 009d3e7e8a8b4e2497de3a73d01ced49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 21:27:28.159874 25042 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:27:28.160185 25042 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:27:28.167464 25042 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 21:27:28.192024 25012 consensus_queue.cc:1035] T 85e77d5faa8346cd8a99dcbcdf75f250 P 009d3e7e8a8b4e2497de3a73d01ced49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_1'
I20240502 21:27:28.196856 25013 consensus_queue.cc:1035] T 85e77d5faa8346cd8a99dcbcdf75f250 P 009d3e7e8a8b4e2497de3a73d01ced49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:28.284862 24185 raft_consensus.cc:1261] T 4eebf4508766421eb6c547a0fbbc28b1 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 1 FOLLOWER]: Refusing update from remote peer 6810a866decb4f5089f7d35c870e4204: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:27:28.285630 24316 raft_consensus.cc:1261] T 4eebf4508766421eb6c547a0fbbc28b1 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 1 FOLLOWER]: Refusing update from remote peer 6810a866decb4f5089f7d35c870e4204: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:27:28.285743 24931 consensus_queue.cc:1035] T 4eebf4508766421eb6c547a0fbbc28b1 P 6810a866decb4f5089f7d35c870e4204 [LEADER]: Connected to new peer: Peer: permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:28.293352 24931 consensus_queue.cc:1035] T 4eebf4508766421eb6c547a0fbbc28b1 P 6810a866decb4f5089f7d35c870e4204 [LEADER]: Connected to new peer: Peer: permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:28.355122 25044 mvcc.cc:205] Tried to move back new op lower bound from 7023350776953794560 to 7023350774823129088. Current Snapshot: MvccSnapshot[applied={T|T < 7023350776953794560}]
W20240502 21:27:28.400848 23849 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.132:46813, user_credentials={real_user=slave}} blocked reactor thread for 57405.3us
INSERT report
rows total: 2000
time total: 213.33 ms
time per row: 0.106665 ms
I20240502 21:27:28.520381 23766 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41618:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 21:27:28.529446 24028 tablet_service.cc:1467] Processing CreateTablet for tablet 906cfe7530754271bb1ee57c470109bd (DEFAULT_TABLE table=rebalance_test_table_2 [id=85ca987b38d84a7db5119dfc88b4feaf]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:28.529775 24296 tablet_service.cc:1467] Processing CreateTablet for tablet 906cfe7530754271bb1ee57c470109bd (DEFAULT_TABLE table=rebalance_test_table_2 [id=85ca987b38d84a7db5119dfc88b4feaf]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:28.529902 24028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 906cfe7530754271bb1ee57c470109bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:28.530195 24296 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 906cfe7530754271bb1ee57c470109bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:28.531258 24297 tablet_service.cc:1467] Processing CreateTablet for tablet cca45c89b9fe40e7800d95c1598a9e52 (DEFAULT_TABLE table=rebalance_test_table_2 [id=85ca987b38d84a7db5119dfc88b4feaf]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:28.531251 23896 tablet_service.cc:1467] Processing CreateTablet for tablet 906cfe7530754271bb1ee57c470109bd (DEFAULT_TABLE table=rebalance_test_table_2 [id=85ca987b38d84a7db5119dfc88b4feaf]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:28.531646 24297 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cca45c89b9fe40e7800d95c1598a9e52. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:28.531689 23896 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 906cfe7530754271bb1ee57c470109bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:28.536244 24164 tablet_service.cc:1467] Processing CreateTablet for tablet cca45c89b9fe40e7800d95c1598a9e52 (DEFAULT_TABLE table=rebalance_test_table_2 [id=85ca987b38d84a7db5119dfc88b4feaf]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:28.536744 24164 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cca45c89b9fe40e7800d95c1598a9e52. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:28.541522 24295 tablet_service.cc:1467] Processing CreateTablet for tablet c9b5be01b5e941b1848df79c7a9db348 (DEFAULT_TABLE table=rebalance_test_table_2 [id=85ca987b38d84a7db5119dfc88b4feaf]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:28.542016 24295 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9b5be01b5e941b1848df79c7a9db348. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:28.545523 24027 tablet_service.cc:1467] Processing CreateTablet for tablet cca45c89b9fe40e7800d95c1598a9e52 (DEFAULT_TABLE table=rebalance_test_table_2 [id=85ca987b38d84a7db5119dfc88b4feaf]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:28.545953 24027 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cca45c89b9fe40e7800d95c1598a9e52. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:28.545877 23895 tablet_service.cc:1467] Processing CreateTablet for tablet c9b5be01b5e941b1848df79c7a9db348 (DEFAULT_TABLE table=rebalance_test_table_2 [id=85ca987b38d84a7db5119dfc88b4feaf]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:28.546284 23895 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9b5be01b5e941b1848df79c7a9db348. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:28.547935 25107 tablet_bootstrap.cc:492] T 906cfe7530754271bb1ee57c470109bd P 6c194a6db6c145acafc0f3068b8ce1fd: Bootstrap starting.
I20240502 21:27:28.549975 25107 tablet_bootstrap.cc:654] T 906cfe7530754271bb1ee57c470109bd P 6c194a6db6c145acafc0f3068b8ce1fd: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:28.552585 24029 tablet_service.cc:1467] Processing CreateTablet for tablet c9b5be01b5e941b1848df79c7a9db348 (DEFAULT_TABLE table=rebalance_test_table_2 [id=85ca987b38d84a7db5119dfc88b4feaf]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:28.552996 24029 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9b5be01b5e941b1848df79c7a9db348. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:28.565173 25111 tablet_bootstrap.cc:492] T cca45c89b9fe40e7800d95c1598a9e52 P 009d3e7e8a8b4e2497de3a73d01ced49: Bootstrap starting.
I20240502 21:27:28.568293 25111 tablet_bootstrap.cc:654] T cca45c89b9fe40e7800d95c1598a9e52 P 009d3e7e8a8b4e2497de3a73d01ced49: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:28.569782 25109 tablet_bootstrap.cc:492] T 906cfe7530754271bb1ee57c470109bd P 6810a866decb4f5089f7d35c870e4204: Bootstrap starting.
I20240502 21:27:28.571628 25109 tablet_bootstrap.cc:654] T 906cfe7530754271bb1ee57c470109bd P 6810a866decb4f5089f7d35c870e4204: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:28.573168 25108 tablet_bootstrap.cc:492] T 906cfe7530754271bb1ee57c470109bd P 038c5d9d1d7a49c981c4c34fbc00d9e6: Bootstrap starting.
I20240502 21:27:28.574489 25111 tablet_bootstrap.cc:492] T cca45c89b9fe40e7800d95c1598a9e52 P 009d3e7e8a8b4e2497de3a73d01ced49: No bootstrap required, opened a new log
I20240502 21:27:28.574697 25111 ts_tablet_manager.cc:1367] T cca45c89b9fe40e7800d95c1598a9e52 P 009d3e7e8a8b4e2497de3a73d01ced49: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.000s
I20240502 21:27:28.574854 25108 tablet_bootstrap.cc:654] T 906cfe7530754271bb1ee57c470109bd P 038c5d9d1d7a49c981c4c34fbc00d9e6: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:28.575644 25111 raft_consensus.cc:348] T cca45c89b9fe40e7800d95c1598a9e52 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } }
I20240502 21:27:28.575968 25111 raft_consensus.cc:374] T cca45c89b9fe40e7800d95c1598a9e52 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:28.576104 25111 raft_consensus.cc:729] T cca45c89b9fe40e7800d95c1598a9e52 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 009d3e7e8a8b4e2497de3a73d01ced49, State: Initialized, Role: FOLLOWER
I20240502 21:27:28.576504 25111 consensus_queue.cc:260] T cca45c89b9fe40e7800d95c1598a9e52 P 009d3e7e8a8b4e2497de3a73d01ced49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } }
I20240502 21:27:28.576874 25107 tablet_bootstrap.cc:492] T 906cfe7530754271bb1ee57c470109bd P 6c194a6db6c145acafc0f3068b8ce1fd: No bootstrap required, opened a new log
I20240502 21:27:28.577042 25107 ts_tablet_manager.cc:1367] T 906cfe7530754271bb1ee57c470109bd P 6c194a6db6c145acafc0f3068b8ce1fd: Time spent bootstrapping tablet: real 0.029s user 0.012s sys 0.016s
I20240502 21:27:28.577281 25111 ts_tablet_manager.cc:1398] T cca45c89b9fe40e7800d95c1598a9e52 P 009d3e7e8a8b4e2497de3a73d01ced49: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:27:28.577836 25107 raft_consensus.cc:348] T 906cfe7530754271bb1ee57c470109bd P 6c194a6db6c145acafc0f3068b8ce1fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } }
I20240502 21:27:28.578034 25107 raft_consensus.cc:374] T 906cfe7530754271bb1ee57c470109bd P 6c194a6db6c145acafc0f3068b8ce1fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:28.578135 25107 raft_consensus.cc:729] T 906cfe7530754271bb1ee57c470109bd P 6c194a6db6c145acafc0f3068b8ce1fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c194a6db6c145acafc0f3068b8ce1fd, State: Initialized, Role: FOLLOWER
I20240502 21:27:28.578361 25107 consensus_queue.cc:260] T 906cfe7530754271bb1ee57c470109bd P 6c194a6db6c145acafc0f3068b8ce1fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } }
I20240502 21:27:28.578908 25107 ts_tablet_manager.cc:1398] T 906cfe7530754271bb1ee57c470109bd P 6c194a6db6c145acafc0f3068b8ce1fd: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 21:27:28.579198 25107 tablet_bootstrap.cc:492] T cca45c89b9fe40e7800d95c1598a9e52 P 6c194a6db6c145acafc0f3068b8ce1fd: Bootstrap starting.
I20240502 21:27:28.580194 25108 tablet_bootstrap.cc:492] T 906cfe7530754271bb1ee57c470109bd P 038c5d9d1d7a49c981c4c34fbc00d9e6: No bootstrap required, opened a new log
I20240502 21:27:28.580339 25108 ts_tablet_manager.cc:1367] T 906cfe7530754271bb1ee57c470109bd P 038c5d9d1d7a49c981c4c34fbc00d9e6: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20240502 21:27:28.580719 25107 tablet_bootstrap.cc:654] T cca45c89b9fe40e7800d95c1598a9e52 P 6c194a6db6c145acafc0f3068b8ce1fd: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:28.581152 25108 raft_consensus.cc:348] T 906cfe7530754271bb1ee57c470109bd P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } }
I20240502 21:27:28.581354 25108 raft_consensus.cc:374] T 906cfe7530754271bb1ee57c470109bd P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:28.581445 25108 raft_consensus.cc:729] T 906cfe7530754271bb1ee57c470109bd P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 038c5d9d1d7a49c981c4c34fbc00d9e6, State: Initialized, Role: FOLLOWER
I20240502 21:27:28.581662 25108 consensus_queue.cc:260] T 906cfe7530754271bb1ee57c470109bd P 038c5d9d1d7a49c981c4c34fbc00d9e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } }
I20240502 21:27:28.582597 25108 ts_tablet_manager.cc:1398] T 906cfe7530754271bb1ee57c470109bd P 038c5d9d1d7a49c981c4c34fbc00d9e6: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:27:28.582890 25108 tablet_bootstrap.cc:492] T cca45c89b9fe40e7800d95c1598a9e52 P 038c5d9d1d7a49c981c4c34fbc00d9e6: Bootstrap starting.
I20240502 21:27:28.584151 25109 tablet_bootstrap.cc:492] T 906cfe7530754271bb1ee57c470109bd P 6810a866decb4f5089f7d35c870e4204: No bootstrap required, opened a new log
I20240502 21:27:28.584338 25108 tablet_bootstrap.cc:654] T cca45c89b9fe40e7800d95c1598a9e52 P 038c5d9d1d7a49c981c4c34fbc00d9e6: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:28.584371 25109 ts_tablet_manager.cc:1367] T 906cfe7530754271bb1ee57c470109bd P 6810a866decb4f5089f7d35c870e4204: Time spent bootstrapping tablet: real 0.015s user 0.000s sys 0.007s
I20240502 21:27:28.585317 25109 raft_consensus.cc:348] T 906cfe7530754271bb1ee57c470109bd P 6810a866decb4f5089f7d35c870e4204 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } }
I20240502 21:27:28.585623 25109 raft_consensus.cc:374] T 906cfe7530754271bb1ee57c470109bd P 6810a866decb4f5089f7d35c870e4204 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:28.585762 25109 raft_consensus.cc:729] T 906cfe7530754271bb1ee57c470109bd P 6810a866decb4f5089f7d35c870e4204 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6810a866decb4f5089f7d35c870e4204, State: Initialized, Role: FOLLOWER
I20240502 21:27:28.586046 25109 consensus_queue.cc:260] T 906cfe7530754271bb1ee57c470109bd P 6810a866decb4f5089f7d35c870e4204 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } }
I20240502 21:27:28.586671 25109 ts_tablet_manager.cc:1398] T 906cfe7530754271bb1ee57c470109bd P 6810a866decb4f5089f7d35c870e4204: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 21:27:28.587081 25109 tablet_bootstrap.cc:492] T c9b5be01b5e941b1848df79c7a9db348 P 6810a866decb4f5089f7d35c870e4204: Bootstrap starting.
I20240502 21:27:28.588722 25109 tablet_bootstrap.cc:654] T c9b5be01b5e941b1848df79c7a9db348 P 6810a866decb4f5089f7d35c870e4204: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:28.593369 25108 tablet_bootstrap.cc:492] T cca45c89b9fe40e7800d95c1598a9e52 P 038c5d9d1d7a49c981c4c34fbc00d9e6: No bootstrap required, opened a new log
I20240502 21:27:28.593540 25108 ts_tablet_manager.cc:1367] T cca45c89b9fe40e7800d95c1598a9e52 P 038c5d9d1d7a49c981c4c34fbc00d9e6: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20240502 21:27:28.594347 25108 raft_consensus.cc:348] T cca45c89b9fe40e7800d95c1598a9e52 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } }
I20240502 21:27:28.594590 25108 raft_consensus.cc:374] T cca45c89b9fe40e7800d95c1598a9e52 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:28.594666 25108 raft_consensus.cc:729] T cca45c89b9fe40e7800d95c1598a9e52 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 038c5d9d1d7a49c981c4c34fbc00d9e6, State: Initialized, Role: FOLLOWER
I20240502 21:27:28.594946 25108 consensus_queue.cc:260] T cca45c89b9fe40e7800d95c1598a9e52 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } }
I20240502 21:27:28.595590 25108 ts_tablet_manager.cc:1398] T cca45c89b9fe40e7800d95c1598a9e52 P 038c5d9d1d7a49c981c4c34fbc00d9e6: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:27:28.595890 25108 tablet_bootstrap.cc:492] T c9b5be01b5e941b1848df79c7a9db348 P 038c5d9d1d7a49c981c4c34fbc00d9e6: Bootstrap starting.
I20240502 21:27:28.597568 25108 tablet_bootstrap.cc:654] T c9b5be01b5e941b1848df79c7a9db348 P 038c5d9d1d7a49c981c4c34fbc00d9e6: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:28.599550 25109 tablet_bootstrap.cc:492] T c9b5be01b5e941b1848df79c7a9db348 P 6810a866decb4f5089f7d35c870e4204: No bootstrap required, opened a new log
I20240502 21:27:28.599767 25109 ts_tablet_manager.cc:1367] T c9b5be01b5e941b1848df79c7a9db348 P 6810a866decb4f5089f7d35c870e4204: Time spent bootstrapping tablet: real 0.013s user 0.008s sys 0.004s
I20240502 21:27:28.599947 25108 tablet_bootstrap.cc:492] T c9b5be01b5e941b1848df79c7a9db348 P 038c5d9d1d7a49c981c4c34fbc00d9e6: No bootstrap required, opened a new log
I20240502 21:27:28.600080 25108 ts_tablet_manager.cc:1367] T c9b5be01b5e941b1848df79c7a9db348 P 038c5d9d1d7a49c981c4c34fbc00d9e6: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240502 21:27:28.600744 25109 raft_consensus.cc:348] T c9b5be01b5e941b1848df79c7a9db348 P 6810a866decb4f5089f7d35c870e4204 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } }
I20240502 21:27:28.600792 25108 raft_consensus.cc:348] T c9b5be01b5e941b1848df79c7a9db348 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } }
I20240502 21:27:28.600947 25108 raft_consensus.cc:374] T c9b5be01b5e941b1848df79c7a9db348 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:28.601024 25108 raft_consensus.cc:729] T c9b5be01b5e941b1848df79c7a9db348 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 038c5d9d1d7a49c981c4c34fbc00d9e6, State: Initialized, Role: FOLLOWER
I20240502 21:27:28.601157 25109 raft_consensus.cc:374] T c9b5be01b5e941b1848df79c7a9db348 P 6810a866decb4f5089f7d35c870e4204 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:28.601284 25109 raft_consensus.cc:729] T c9b5be01b5e941b1848df79c7a9db348 P 6810a866decb4f5089f7d35c870e4204 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6810a866decb4f5089f7d35c870e4204, State: Initialized, Role: FOLLOWER
I20240502 21:27:28.601274 25108 consensus_queue.cc:260] T c9b5be01b5e941b1848df79c7a9db348 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } }
I20240502 21:27:28.601665 25109 consensus_queue.cc:260] T c9b5be01b5e941b1848df79c7a9db348 P 6810a866decb4f5089f7d35c870e4204 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } }
I20240502 21:27:28.601918 25108 ts_tablet_manager.cc:1398] T c9b5be01b5e941b1848df79c7a9db348 P 038c5d9d1d7a49c981c4c34fbc00d9e6: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:27:28.602231 25109 ts_tablet_manager.cc:1398] T c9b5be01b5e941b1848df79c7a9db348 P 6810a866decb4f5089f7d35c870e4204: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:27:28.609467 25107 tablet_bootstrap.cc:492] T cca45c89b9fe40e7800d95c1598a9e52 P 6c194a6db6c145acafc0f3068b8ce1fd: No bootstrap required, opened a new log
I20240502 21:27:28.609649 25107 ts_tablet_manager.cc:1367] T cca45c89b9fe40e7800d95c1598a9e52 P 6c194a6db6c145acafc0f3068b8ce1fd: Time spent bootstrapping tablet: real 0.031s user 0.014s sys 0.015s
I20240502 21:27:28.610244 25107 raft_consensus.cc:348] T cca45c89b9fe40e7800d95c1598a9e52 P 6c194a6db6c145acafc0f3068b8ce1fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } }
I20240502 21:27:28.610430 25107 raft_consensus.cc:374] T cca45c89b9fe40e7800d95c1598a9e52 P 6c194a6db6c145acafc0f3068b8ce1fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:28.610508 25107 raft_consensus.cc:729] T cca45c89b9fe40e7800d95c1598a9e52 P 6c194a6db6c145acafc0f3068b8ce1fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c194a6db6c145acafc0f3068b8ce1fd, State: Initialized, Role: FOLLOWER
I20240502 21:27:28.610757 25107 consensus_queue.cc:260] T cca45c89b9fe40e7800d95c1598a9e52 P 6c194a6db6c145acafc0f3068b8ce1fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } }
I20240502 21:27:28.611256 25107 ts_tablet_manager.cc:1398] T cca45c89b9fe40e7800d95c1598a9e52 P 6c194a6db6c145acafc0f3068b8ce1fd: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 21:27:28.611584 25107 tablet_bootstrap.cc:492] T c9b5be01b5e941b1848df79c7a9db348 P 6c194a6db6c145acafc0f3068b8ce1fd: Bootstrap starting.
I20240502 21:27:28.613185 25107 tablet_bootstrap.cc:654] T c9b5be01b5e941b1848df79c7a9db348 P 6c194a6db6c145acafc0f3068b8ce1fd: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:28.615525 25107 tablet_bootstrap.cc:492] T c9b5be01b5e941b1848df79c7a9db348 P 6c194a6db6c145acafc0f3068b8ce1fd: No bootstrap required, opened a new log
I20240502 21:27:28.615655 25107 ts_tablet_manager.cc:1367] T c9b5be01b5e941b1848df79c7a9db348 P 6c194a6db6c145acafc0f3068b8ce1fd: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240502 21:27:28.616312 25107 raft_consensus.cc:348] T c9b5be01b5e941b1848df79c7a9db348 P 6c194a6db6c145acafc0f3068b8ce1fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } }
I20240502 21:27:28.616487 25107 raft_consensus.cc:374] T c9b5be01b5e941b1848df79c7a9db348 P 6c194a6db6c145acafc0f3068b8ce1fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:28.616581 25107 raft_consensus.cc:729] T c9b5be01b5e941b1848df79c7a9db348 P 6c194a6db6c145acafc0f3068b8ce1fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c194a6db6c145acafc0f3068b8ce1fd, State: Initialized, Role: FOLLOWER
I20240502 21:27:28.616796 25107 consensus_queue.cc:260] T c9b5be01b5e941b1848df79c7a9db348 P 6c194a6db6c145acafc0f3068b8ce1fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } }
I20240502 21:27:28.617285 25107 ts_tablet_manager.cc:1398] T c9b5be01b5e941b1848df79c7a9db348 P 6c194a6db6c145acafc0f3068b8ce1fd: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 21:27:28.635850 25088 raft_consensus.cc:482] T 906cfe7530754271bb1ee57c470109bd P 6810a866decb4f5089f7d35c870e4204 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:28.636010 25088 raft_consensus.cc:504] T 906cfe7530754271bb1ee57c470109bd P 6810a866decb4f5089f7d35c870e4204 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } }
I20240502 21:27:28.636701 25088 leader_election.cc:288] T 906cfe7530754271bb1ee57c470109bd P 6810a866decb4f5089f7d35c870e4204 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6c194a6db6c145acafc0f3068b8ce1fd (127.15.255.130:41235), 038c5d9d1d7a49c981c4c34fbc00d9e6 (127.15.255.132:46813)
I20240502 21:27:28.637153 24049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "906cfe7530754271bb1ee57c470109bd" candidate_uuid: "6810a866decb4f5089f7d35c870e4204" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" is_pre_election: true
I20240502 21:27:28.637192 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "906cfe7530754271bb1ee57c470109bd" candidate_uuid: "6810a866decb4f5089f7d35c870e4204" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" is_pre_election: true
I20240502 21:27:28.637377 24049 raft_consensus.cc:2454] T 906cfe7530754271bb1ee57c470109bd P 6c194a6db6c145acafc0f3068b8ce1fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6810a866decb4f5089f7d35c870e4204 in term 0.
I20240502 21:27:28.637420 24316 raft_consensus.cc:2454] T 906cfe7530754271bb1ee57c470109bd P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6810a866decb4f5089f7d35c870e4204 in term 0.
I20240502 21:27:28.637773 23851 leader_election.cc:302] T 906cfe7530754271bb1ee57c470109bd P 6810a866decb4f5089f7d35c870e4204 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6810a866decb4f5089f7d35c870e4204, 6c194a6db6c145acafc0f3068b8ce1fd; no voters:
I20240502 21:27:28.638057 25088 raft_consensus.cc:2781] T 906cfe7530754271bb1ee57c470109bd P 6810a866decb4f5089f7d35c870e4204 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:28.638173 25088 raft_consensus.cc:482] T 906cfe7530754271bb1ee57c470109bd P 6810a866decb4f5089f7d35c870e4204 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:28.638263 25088 raft_consensus.cc:3037] T 906cfe7530754271bb1ee57c470109bd P 6810a866decb4f5089f7d35c870e4204 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:28.640126 25088 raft_consensus.cc:504] T 906cfe7530754271bb1ee57c470109bd P 6810a866decb4f5089f7d35c870e4204 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } }
I20240502 21:27:28.640695 25088 leader_election.cc:288] T 906cfe7530754271bb1ee57c470109bd P 6810a866decb4f5089f7d35c870e4204 [CANDIDATE]: Term 1 election: Requested vote from peers 6c194a6db6c145acafc0f3068b8ce1fd (127.15.255.130:41235), 038c5d9d1d7a49c981c4c34fbc00d9e6 (127.15.255.132:46813)
I20240502 21:27:28.641083 24049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "906cfe7530754271bb1ee57c470109bd" candidate_uuid: "6810a866decb4f5089f7d35c870e4204" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c194a6db6c145acafc0f3068b8ce1fd"
I20240502 21:27:28.641112 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "906cfe7530754271bb1ee57c470109bd" candidate_uuid: "6810a866decb4f5089f7d35c870e4204" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6"
I20240502 21:27:28.641274 24049 raft_consensus.cc:3037] T 906cfe7530754271bb1ee57c470109bd P 6c194a6db6c145acafc0f3068b8ce1fd [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:28.641286 24316 raft_consensus.cc:3037] T 906cfe7530754271bb1ee57c470109bd P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:28.643049 24049 raft_consensus.cc:2454] T 906cfe7530754271bb1ee57c470109bd P 6c194a6db6c145acafc0f3068b8ce1fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6810a866decb4f5089f7d35c870e4204 in term 1.
I20240502 21:27:28.643160 24316 raft_consensus.cc:2454] T 906cfe7530754271bb1ee57c470109bd P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6810a866decb4f5089f7d35c870e4204 in term 1.
I20240502 21:27:28.643438 23851 leader_election.cc:302] T 906cfe7530754271bb1ee57c470109bd P 6810a866decb4f5089f7d35c870e4204 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6810a866decb4f5089f7d35c870e4204, 6c194a6db6c145acafc0f3068b8ce1fd; no voters:
I20240502 21:27:28.643748 25088 raft_consensus.cc:2781] T 906cfe7530754271bb1ee57c470109bd P 6810a866decb4f5089f7d35c870e4204 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:28.643873 25088 raft_consensus.cc:686] T 906cfe7530754271bb1ee57c470109bd P 6810a866decb4f5089f7d35c870e4204 [term 1 LEADER]: Becoming Leader. State: Replica: 6810a866decb4f5089f7d35c870e4204, State: Running, Role: LEADER
I20240502 21:27:28.644125 25088 consensus_queue.cc:237] T 906cfe7530754271bb1ee57c470109bd P 6810a866decb4f5089f7d35c870e4204 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } }
I20240502 21:27:28.647037 23764 catalog_manager.cc:5484] T 906cfe7530754271bb1ee57c470109bd P 6810a866decb4f5089f7d35c870e4204 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6810a866decb4f5089f7d35c870e4204 (127.15.255.129). New cstate: current_term: 1 leader_uuid: "6810a866decb4f5089f7d35c870e4204" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } health_report { overall_health: HEALTHY } } }
I20240502 21:27:28.683964 25013 raft_consensus.cc:482] T cca45c89b9fe40e7800d95c1598a9e52 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:28.684165 25013 raft_consensus.cc:504] T cca45c89b9fe40e7800d95c1598a9e52 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } }
I20240502 21:27:28.684870 25013 leader_election.cc:288] T cca45c89b9fe40e7800d95c1598a9e52 P 009d3e7e8a8b4e2497de3a73d01ced49 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 038c5d9d1d7a49c981c4c34fbc00d9e6 (127.15.255.132:46813), 6c194a6db6c145acafc0f3068b8ce1fd (127.15.255.130:41235)
I20240502 21:27:28.685397 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cca45c89b9fe40e7800d95c1598a9e52" candidate_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" is_pre_election: true
I20240502 21:27:28.685424 24049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cca45c89b9fe40e7800d95c1598a9e52" candidate_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" is_pre_election: true
I20240502 21:27:28.685599 24316 raft_consensus.cc:2454] T cca45c89b9fe40e7800d95c1598a9e52 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 009d3e7e8a8b4e2497de3a73d01ced49 in term 0.
I20240502 21:27:28.685633 24049 raft_consensus.cc:2454] T cca45c89b9fe40e7800d95c1598a9e52 P 6c194a6db6c145acafc0f3068b8ce1fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 009d3e7e8a8b4e2497de3a73d01ced49 in term 0.
I20240502 21:27:28.686012 24118 leader_election.cc:302] T cca45c89b9fe40e7800d95c1598a9e52 P 009d3e7e8a8b4e2497de3a73d01ced49 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 009d3e7e8a8b4e2497de3a73d01ced49, 6c194a6db6c145acafc0f3068b8ce1fd; no voters:
I20240502 21:27:28.686281 25013 raft_consensus.cc:2781] T cca45c89b9fe40e7800d95c1598a9e52 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:28.686368 25013 raft_consensus.cc:482] T cca45c89b9fe40e7800d95c1598a9e52 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:28.686442 25013 raft_consensus.cc:3037] T cca45c89b9fe40e7800d95c1598a9e52 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:28.688236 25013 raft_consensus.cc:504] T cca45c89b9fe40e7800d95c1598a9e52 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } }
I20240502 21:27:28.688868 25013 leader_election.cc:288] T cca45c89b9fe40e7800d95c1598a9e52 P 009d3e7e8a8b4e2497de3a73d01ced49 [CANDIDATE]: Term 1 election: Requested vote from peers 038c5d9d1d7a49c981c4c34fbc00d9e6 (127.15.255.132:46813), 6c194a6db6c145acafc0f3068b8ce1fd (127.15.255.130:41235)
I20240502 21:27:28.689198 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cca45c89b9fe40e7800d95c1598a9e52" candidate_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6"
I20240502 21:27:28.689375 24316 raft_consensus.cc:3037] T cca45c89b9fe40e7800d95c1598a9e52 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:28.689359 24049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cca45c89b9fe40e7800d95c1598a9e52" candidate_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c194a6db6c145acafc0f3068b8ce1fd"
I20240502 21:27:28.689548 24049 raft_consensus.cc:3037] T cca45c89b9fe40e7800d95c1598a9e52 P 6c194a6db6c145acafc0f3068b8ce1fd [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:28.691227 24316 raft_consensus.cc:2454] T cca45c89b9fe40e7800d95c1598a9e52 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 009d3e7e8a8b4e2497de3a73d01ced49 in term 1.
I20240502 21:27:28.691375 24049 raft_consensus.cc:2454] T cca45c89b9fe40e7800d95c1598a9e52 P 6c194a6db6c145acafc0f3068b8ce1fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 009d3e7e8a8b4e2497de3a73d01ced49 in term 1.
I20240502 21:27:28.691594 24116 leader_election.cc:302] T cca45c89b9fe40e7800d95c1598a9e52 P 009d3e7e8a8b4e2497de3a73d01ced49 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 009d3e7e8a8b4e2497de3a73d01ced49, 038c5d9d1d7a49c981c4c34fbc00d9e6; no voters:
I20240502 21:27:28.691861 25013 raft_consensus.cc:2781] T cca45c89b9fe40e7800d95c1598a9e52 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:28.692004 25013 raft_consensus.cc:686] T cca45c89b9fe40e7800d95c1598a9e52 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 1 LEADER]: Becoming Leader. State: Replica: 009d3e7e8a8b4e2497de3a73d01ced49, State: Running, Role: LEADER
I20240502 21:27:28.692235 25013 consensus_queue.cc:237] T cca45c89b9fe40e7800d95c1598a9e52 P 009d3e7e8a8b4e2497de3a73d01ced49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } }
I20240502 21:27:28.694810 23764 catalog_manager.cc:5484] T cca45c89b9fe40e7800d95c1598a9e52 P 009d3e7e8a8b4e2497de3a73d01ced49 reported cstate change: term changed from 0 to 1, leader changed from <none> to 009d3e7e8a8b4e2497de3a73d01ced49 (127.15.255.131). New cstate: current_term: 1 leader_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } health_report { overall_health: HEALTHY } } }
I20240502 21:27:28.710428 24943 raft_consensus.cc:482] T c9b5be01b5e941b1848df79c7a9db348 P 6c194a6db6c145acafc0f3068b8ce1fd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:28.710577 24943 raft_consensus.cc:504] T c9b5be01b5e941b1848df79c7a9db348 P 6c194a6db6c145acafc0f3068b8ce1fd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } }
I20240502 21:27:28.711176 24943 leader_election.cc:288] T c9b5be01b5e941b1848df79c7a9db348 P 6c194a6db6c145acafc0f3068b8ce1fd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 038c5d9d1d7a49c981c4c34fbc00d9e6 (127.15.255.132:46813), 6810a866decb4f5089f7d35c870e4204 (127.15.255.129:40435)
I20240502 21:27:28.711772 23915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b5be01b5e941b1848df79c7a9db348" candidate_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6810a866decb4f5089f7d35c870e4204" is_pre_election: true
I20240502 21:27:28.712033 23915 raft_consensus.cc:2454] T c9b5be01b5e941b1848df79c7a9db348 P 6810a866decb4f5089f7d35c870e4204 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6c194a6db6c145acafc0f3068b8ce1fd in term 0.
I20240502 21:27:28.712498 23984 leader_election.cc:302] T c9b5be01b5e941b1848df79c7a9db348 P 6c194a6db6c145acafc0f3068b8ce1fd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6810a866decb4f5089f7d35c870e4204, 6c194a6db6c145acafc0f3068b8ce1fd; no voters:
I20240502 21:27:28.712833 24943 raft_consensus.cc:2781] T c9b5be01b5e941b1848df79c7a9db348 P 6c194a6db6c145acafc0f3068b8ce1fd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:28.712960 24943 raft_consensus.cc:482] T c9b5be01b5e941b1848df79c7a9db348 P 6c194a6db6c145acafc0f3068b8ce1fd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:28.713089 24943 raft_consensus.cc:3037] T c9b5be01b5e941b1848df79c7a9db348 P 6c194a6db6c145acafc0f3068b8ce1fd [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:28.715113 24943 raft_consensus.cc:504] T c9b5be01b5e941b1848df79c7a9db348 P 6c194a6db6c145acafc0f3068b8ce1fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } }
I20240502 21:27:28.715834 24943 leader_election.cc:288] T c9b5be01b5e941b1848df79c7a9db348 P 6c194a6db6c145acafc0f3068b8ce1fd [CANDIDATE]: Term 1 election: Requested vote from peers 038c5d9d1d7a49c981c4c34fbc00d9e6 (127.15.255.132:46813), 6810a866decb4f5089f7d35c870e4204 (127.15.255.129:40435)
I20240502 21:27:28.716495 23915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b5be01b5e941b1848df79c7a9db348" candidate_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6810a866decb4f5089f7d35c870e4204"
I20240502 21:27:28.716717 23915 raft_consensus.cc:3037] T c9b5be01b5e941b1848df79c7a9db348 P 6810a866decb4f5089f7d35c870e4204 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:28.718513 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b5be01b5e941b1848df79c7a9db348" candidate_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" is_pre_election: true
I20240502 21:27:28.718611 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b5be01b5e941b1848df79c7a9db348" candidate_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6"
I20240502 21:27:28.718761 24316 raft_consensus.cc:2454] T c9b5be01b5e941b1848df79c7a9db348 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6c194a6db6c145acafc0f3068b8ce1fd in term 0.
I20240502 21:27:28.719211 23915 raft_consensus.cc:2454] T c9b5be01b5e941b1848df79c7a9db348 P 6810a866decb4f5089f7d35c870e4204 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c194a6db6c145acafc0f3068b8ce1fd in term 1.
I20240502 21:27:28.719604 23984 leader_election.cc:302] T c9b5be01b5e941b1848df79c7a9db348 P 6c194a6db6c145acafc0f3068b8ce1fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6810a866decb4f5089f7d35c870e4204, 6c194a6db6c145acafc0f3068b8ce1fd; no voters: 038c5d9d1d7a49c981c4c34fbc00d9e6
I20240502 21:27:28.719857 24943 raft_consensus.cc:2781] T c9b5be01b5e941b1848df79c7a9db348 P 6c194a6db6c145acafc0f3068b8ce1fd [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:28.720197 24943 raft_consensus.cc:686] T c9b5be01b5e941b1848df79c7a9db348 P 6c194a6db6c145acafc0f3068b8ce1fd [term 1 LEADER]: Becoming Leader. State: Replica: 6c194a6db6c145acafc0f3068b8ce1fd, State: Running, Role: LEADER
I20240502 21:27:28.720482 24943 consensus_queue.cc:237] T c9b5be01b5e941b1848df79c7a9db348 P 6c194a6db6c145acafc0f3068b8ce1fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } }
I20240502 21:27:28.722951 23764 catalog_manager.cc:5484] T c9b5be01b5e941b1848df79c7a9db348 P 6c194a6db6c145acafc0f3068b8ce1fd reported cstate change: term changed from 0 to 1, leader changed from <none> to 6c194a6db6c145acafc0f3068b8ce1fd (127.15.255.130). New cstate: current_term: 1 leader_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } health_report { overall_health: HEALTHY } } }
I20240502 21:27:29.028232 25088 consensus_queue.cc:1035] T 906cfe7530754271bb1ee57c470109bd P 6810a866decb4f5089f7d35c870e4204 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:29.033033 25088 consensus_queue.cc:1035] T 906cfe7530754271bb1ee57c470109bd P 6810a866decb4f5089f7d35c870e4204 [LEADER]: Connected to new peer: Peer: permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 21:27:29.085935 25126 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:27:29.086246 25126 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:27:29.094190 25126 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 21:27:29.098218 24943 consensus_queue.cc:1035] T c9b5be01b5e941b1848df79c7a9db348 P 6c194a6db6c145acafc0f3068b8ce1fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:29.103168 24317 raft_consensus.cc:3037] T c9b5be01b5e941b1848df79c7a9db348 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:29.108306 24943 consensus_queue.cc:1035] T c9b5be01b5e941b1848df79c7a9db348 P 6c194a6db6c145acafc0f3068b8ce1fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_2'
I20240502 21:27:29.244993 24049 raft_consensus.cc:1261] T cca45c89b9fe40e7800d95c1598a9e52 P 6c194a6db6c145acafc0f3068b8ce1fd [term 1 FOLLOWER]: Refusing update from remote peer 009d3e7e8a8b4e2497de3a73d01ced49: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:27:29.245751 25083 consensus_queue.cc:1035] T cca45c89b9fe40e7800d95c1598a9e52 P 009d3e7e8a8b4e2497de3a73d01ced49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:29.250272 24317 raft_consensus.cc:1261] T cca45c89b9fe40e7800d95c1598a9e52 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 1 FOLLOWER]: Refusing update from remote peer 009d3e7e8a8b4e2497de3a73d01ced49: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:27:29.250872 25083 consensus_queue.cc:1035] T cca45c89b9fe40e7800d95c1598a9e52 P 009d3e7e8a8b4e2497de3a73d01ced49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:29.348608 25175 mvcc.cc:205] Tried to move back new op lower bound from 7023350780887015424 to 7023350778645712896. Current Snapshot: MvccSnapshot[applied={T|T < 7023350780887015424}]
INSERT report
rows total: 2000
time total: 232.366 ms
time per row: 0.116183 ms
-----------------------------------------------------
Suppressions used:
count bytes template
4 1192 libcrypto.so.1.1
-----------------------------------------------------
I20240502 21:27:30.318480 23764 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41618:
name: "rebalance_test_table_3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 21:27:30.327309 24164 tablet_service.cc:1467] Processing CreateTablet for tablet 549666d19e724a96801db183a07809d8 (DEFAULT_TABLE table=rebalance_test_table_3 [id=933ce90acb9f4b91ab6bd41e87a8fe99]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:30.327626 24029 tablet_service.cc:1467] Processing CreateTablet for tablet 549666d19e724a96801db183a07809d8 (DEFAULT_TABLE table=rebalance_test_table_3 [id=933ce90acb9f4b91ab6bd41e87a8fe99]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:30.327816 24164 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 549666d19e724a96801db183a07809d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:30.328039 24029 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 549666d19e724a96801db183a07809d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:30.328087 23896 tablet_service.cc:1467] Processing CreateTablet for tablet 549666d19e724a96801db183a07809d8 (DEFAULT_TABLE table=rebalance_test_table_3 [id=933ce90acb9f4b91ab6bd41e87a8fe99]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:30.328522 23896 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 549666d19e724a96801db183a07809d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:30.333034 24296 tablet_service.cc:1467] Processing CreateTablet for tablet 89f86d58fe414b579845214534d338d8 (DEFAULT_TABLE table=rebalance_test_table_3 [id=933ce90acb9f4b91ab6bd41e87a8fe99]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:30.333433 24296 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89f86d58fe414b579845214534d338d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:30.334245 23895 tablet_service.cc:1467] Processing CreateTablet for tablet 89f86d58fe414b579845214534d338d8 (DEFAULT_TABLE table=rebalance_test_table_3 [id=933ce90acb9f4b91ab6bd41e87a8fe99]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:30.334456 24162 tablet_service.cc:1467] Processing CreateTablet for tablet 89f86d58fe414b579845214534d338d8 (DEFAULT_TABLE table=rebalance_test_table_3 [id=933ce90acb9f4b91ab6bd41e87a8fe99]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:30.334606 23895 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89f86d58fe414b579845214534d338d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:30.334812 24162 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89f86d58fe414b579845214534d338d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:30.338747 24163 tablet_service.cc:1467] Processing CreateTablet for tablet a274546743354e79bffa26ae7000e895 (DEFAULT_TABLE table=rebalance_test_table_3 [id=933ce90acb9f4b91ab6bd41e87a8fe99]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:30.339076 24163 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a274546743354e79bffa26ae7000e895. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:30.340047 24295 tablet_service.cc:1467] Processing CreateTablet for tablet a274546743354e79bffa26ae7000e895 (DEFAULT_TABLE table=rebalance_test_table_3 [id=933ce90acb9f4b91ab6bd41e87a8fe99]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:30.341498 24295 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a274546743354e79bffa26ae7000e895. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:30.344861 24027 tablet_service.cc:1467] Processing CreateTablet for tablet a274546743354e79bffa26ae7000e895 (DEFAULT_TABLE table=rebalance_test_table_3 [id=933ce90acb9f4b91ab6bd41e87a8fe99]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:30.345229 24027 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a274546743354e79bffa26ae7000e895. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:30.349766 25198 tablet_bootstrap.cc:492] T 549666d19e724a96801db183a07809d8 P 009d3e7e8a8b4e2497de3a73d01ced49: Bootstrap starting.
I20240502 21:27:30.350069 25196 tablet_bootstrap.cc:492] T 549666d19e724a96801db183a07809d8 P 6c194a6db6c145acafc0f3068b8ce1fd: Bootstrap starting.
I20240502 21:27:30.351323 25198 tablet_bootstrap.cc:654] T 549666d19e724a96801db183a07809d8 P 009d3e7e8a8b4e2497de3a73d01ced49: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:30.351549 25196 tablet_bootstrap.cc:654] T 549666d19e724a96801db183a07809d8 P 6c194a6db6c145acafc0f3068b8ce1fd: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:30.353304 25198 tablet_bootstrap.cc:492] T 549666d19e724a96801db183a07809d8 P 009d3e7e8a8b4e2497de3a73d01ced49: No bootstrap required, opened a new log
I20240502 21:27:30.353441 25198 ts_tablet_manager.cc:1367] T 549666d19e724a96801db183a07809d8 P 009d3e7e8a8b4e2497de3a73d01ced49: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240502 21:27:30.354167 25198 raft_consensus.cc:348] T 549666d19e724a96801db183a07809d8 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } }
I20240502 21:27:30.354372 25198 raft_consensus.cc:374] T 549666d19e724a96801db183a07809d8 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:30.354457 25198 raft_consensus.cc:729] T 549666d19e724a96801db183a07809d8 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 009d3e7e8a8b4e2497de3a73d01ced49, State: Initialized, Role: FOLLOWER
I20240502 21:27:30.354703 25198 consensus_queue.cc:260] T 549666d19e724a96801db183a07809d8 P 009d3e7e8a8b4e2497de3a73d01ced49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } }
I20240502 21:27:30.355330 25198 ts_tablet_manager.cc:1398] T 549666d19e724a96801db183a07809d8 P 009d3e7e8a8b4e2497de3a73d01ced49: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:27:30.355938 25198 tablet_bootstrap.cc:492] T a274546743354e79bffa26ae7000e895 P 009d3e7e8a8b4e2497de3a73d01ced49: Bootstrap starting.
I20240502 21:27:30.356932 25196 tablet_bootstrap.cc:492] T 549666d19e724a96801db183a07809d8 P 6c194a6db6c145acafc0f3068b8ce1fd: No bootstrap required, opened a new log
I20240502 21:27:30.357071 25196 ts_tablet_manager.cc:1367] T 549666d19e724a96801db183a07809d8 P 6c194a6db6c145acafc0f3068b8ce1fd: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20240502 21:27:30.357605 25198 tablet_bootstrap.cc:654] T a274546743354e79bffa26ae7000e895 P 009d3e7e8a8b4e2497de3a73d01ced49: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:30.359124 25196 raft_consensus.cc:348] T 549666d19e724a96801db183a07809d8 P 6c194a6db6c145acafc0f3068b8ce1fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } }
I20240502 21:27:30.359329 25196 raft_consensus.cc:374] T 549666d19e724a96801db183a07809d8 P 6c194a6db6c145acafc0f3068b8ce1fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:30.359408 25196 raft_consensus.cc:729] T 549666d19e724a96801db183a07809d8 P 6c194a6db6c145acafc0f3068b8ce1fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c194a6db6c145acafc0f3068b8ce1fd, State: Initialized, Role: FOLLOWER
I20240502 21:27:30.359607 25196 consensus_queue.cc:260] T 549666d19e724a96801db183a07809d8 P 6c194a6db6c145acafc0f3068b8ce1fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } }
I20240502 21:27:30.361392 25197 tablet_bootstrap.cc:492] T 89f86d58fe414b579845214534d338d8 P 038c5d9d1d7a49c981c4c34fbc00d9e6: Bootstrap starting.
I20240502 21:27:30.362079 25196 ts_tablet_manager.cc:1398] T 549666d19e724a96801db183a07809d8 P 6c194a6db6c145acafc0f3068b8ce1fd: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20240502 21:27:30.362340 25196 tablet_bootstrap.cc:492] T a274546743354e79bffa26ae7000e895 P 6c194a6db6c145acafc0f3068b8ce1fd: Bootstrap starting.
I20240502 21:27:30.362349 25201 tablet_bootstrap.cc:492] T 549666d19e724a96801db183a07809d8 P 6810a866decb4f5089f7d35c870e4204: Bootstrap starting.
I20240502 21:27:30.363492 25197 tablet_bootstrap.cc:654] T 89f86d58fe414b579845214534d338d8 P 038c5d9d1d7a49c981c4c34fbc00d9e6: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:30.363920 25196 tablet_bootstrap.cc:654] T a274546743354e79bffa26ae7000e895 P 6c194a6db6c145acafc0f3068b8ce1fd: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:30.366580 25201 tablet_bootstrap.cc:654] T 549666d19e724a96801db183a07809d8 P 6810a866decb4f5089f7d35c870e4204: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:30.368369 25198 tablet_bootstrap.cc:492] T a274546743354e79bffa26ae7000e895 P 009d3e7e8a8b4e2497de3a73d01ced49: No bootstrap required, opened a new log
I20240502 21:27:30.368544 25198 ts_tablet_manager.cc:1367] T a274546743354e79bffa26ae7000e895 P 009d3e7e8a8b4e2497de3a73d01ced49: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.006s
I20240502 21:27:30.369186 25198 raft_consensus.cc:348] T a274546743354e79bffa26ae7000e895 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } }
I20240502 21:27:30.369370 25196 tablet_bootstrap.cc:492] T a274546743354e79bffa26ae7000e895 P 6c194a6db6c145acafc0f3068b8ce1fd: No bootstrap required, opened a new log
I20240502 21:27:30.369387 25198 raft_consensus.cc:374] T a274546743354e79bffa26ae7000e895 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:30.369454 25198 raft_consensus.cc:729] T a274546743354e79bffa26ae7000e895 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 009d3e7e8a8b4e2497de3a73d01ced49, State: Initialized, Role: FOLLOWER
I20240502 21:27:30.369509 25196 ts_tablet_manager.cc:1367] T a274546743354e79bffa26ae7000e895 P 6c194a6db6c145acafc0f3068b8ce1fd: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.003s
I20240502 21:27:30.369525 25201 tablet_bootstrap.cc:492] T 549666d19e724a96801db183a07809d8 P 6810a866decb4f5089f7d35c870e4204: No bootstrap required, opened a new log
I20240502 21:27:30.369531 25197 tablet_bootstrap.cc:492] T 89f86d58fe414b579845214534d338d8 P 038c5d9d1d7a49c981c4c34fbc00d9e6: No bootstrap required, opened a new log
I20240502 21:27:30.369642 25201 ts_tablet_manager.cc:1367] T 549666d19e724a96801db183a07809d8 P 6810a866decb4f5089f7d35c870e4204: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.000s
I20240502 21:27:30.369652 25197 ts_tablet_manager.cc:1367] T 89f86d58fe414b579845214534d338d8 P 038c5d9d1d7a49c981c4c34fbc00d9e6: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20240502 21:27:30.369705 25198 consensus_queue.cc:260] T a274546743354e79bffa26ae7000e895 P 009d3e7e8a8b4e2497de3a73d01ced49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } }
I20240502 21:27:30.370261 25198 ts_tablet_manager.cc:1398] T a274546743354e79bffa26ae7000e895 P 009d3e7e8a8b4e2497de3a73d01ced49: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 21:27:30.370321 25197 raft_consensus.cc:348] T 89f86d58fe414b579845214534d338d8 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } }
I20240502 21:27:30.370332 25201 raft_consensus.cc:348] T 549666d19e724a96801db183a07809d8 P 6810a866decb4f5089f7d35c870e4204 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } }
I20240502 21:27:30.370502 25197 raft_consensus.cc:374] T 89f86d58fe414b579845214534d338d8 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:30.370525 25201 raft_consensus.cc:374] T 549666d19e724a96801db183a07809d8 P 6810a866decb4f5089f7d35c870e4204 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:30.370594 25197 raft_consensus.cc:729] T 89f86d58fe414b579845214534d338d8 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 038c5d9d1d7a49c981c4c34fbc00d9e6, State: Initialized, Role: FOLLOWER
I20240502 21:27:30.370612 25201 raft_consensus.cc:729] T 549666d19e724a96801db183a07809d8 P 6810a866decb4f5089f7d35c870e4204 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6810a866decb4f5089f7d35c870e4204, State: Initialized, Role: FOLLOWER
I20240502 21:27:30.370656 25196 raft_consensus.cc:348] T a274546743354e79bffa26ae7000e895 P 6c194a6db6c145acafc0f3068b8ce1fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } }
I20240502 21:27:30.370864 25196 raft_consensus.cc:374] T a274546743354e79bffa26ae7000e895 P 6c194a6db6c145acafc0f3068b8ce1fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:30.370878 25198 tablet_bootstrap.cc:492] T 89f86d58fe414b579845214534d338d8 P 009d3e7e8a8b4e2497de3a73d01ced49: Bootstrap starting.
I20240502 21:27:30.370927 25196 raft_consensus.cc:729] T a274546743354e79bffa26ae7000e895 P 6c194a6db6c145acafc0f3068b8ce1fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c194a6db6c145acafc0f3068b8ce1fd, State: Initialized, Role: FOLLOWER
I20240502 21:27:30.370848 25201 consensus_queue.cc:260] T 549666d19e724a96801db183a07809d8 P 6810a866decb4f5089f7d35c870e4204 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } }
I20240502 21:27:30.370860 25197 consensus_queue.cc:260] T 89f86d58fe414b579845214534d338d8 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } }
I20240502 21:27:30.371126 25196 consensus_queue.cc:260] T a274546743354e79bffa26ae7000e895 P 6c194a6db6c145acafc0f3068b8ce1fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } }
I20240502 21:27:30.371335 25201 ts_tablet_manager.cc:1398] T 549666d19e724a96801db183a07809d8 P 6810a866decb4f5089f7d35c870e4204: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:27:30.371598 25201 tablet_bootstrap.cc:492] T 89f86d58fe414b579845214534d338d8 P 6810a866decb4f5089f7d35c870e4204: Bootstrap starting.
I20240502 21:27:30.372100 25196 ts_tablet_manager.cc:1398] T a274546743354e79bffa26ae7000e895 P 6c194a6db6c145acafc0f3068b8ce1fd: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240502 21:27:30.372218 25198 tablet_bootstrap.cc:654] T 89f86d58fe414b579845214534d338d8 P 009d3e7e8a8b4e2497de3a73d01ced49: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:30.375373 25201 tablet_bootstrap.cc:654] T 89f86d58fe414b579845214534d338d8 P 6810a866decb4f5089f7d35c870e4204: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:30.379086 25197 ts_tablet_manager.cc:1398] T 89f86d58fe414b579845214534d338d8 P 038c5d9d1d7a49c981c4c34fbc00d9e6: Time spent starting tablet: real 0.009s user 0.004s sys 0.004s
I20240502 21:27:30.379396 25197 tablet_bootstrap.cc:492] T a274546743354e79bffa26ae7000e895 P 038c5d9d1d7a49c981c4c34fbc00d9e6: Bootstrap starting.
I20240502 21:27:30.380223 25198 tablet_bootstrap.cc:492] T 89f86d58fe414b579845214534d338d8 P 009d3e7e8a8b4e2497de3a73d01ced49: No bootstrap required, opened a new log
I20240502 21:27:30.380362 25198 ts_tablet_manager.cc:1367] T 89f86d58fe414b579845214534d338d8 P 009d3e7e8a8b4e2497de3a73d01ced49: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.007s
I20240502 21:27:30.381053 25198 raft_consensus.cc:348] T 89f86d58fe414b579845214534d338d8 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } }
I20240502 21:27:30.381304 25198 raft_consensus.cc:374] T 89f86d58fe414b579845214534d338d8 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:30.381438 25198 raft_consensus.cc:729] T 89f86d58fe414b579845214534d338d8 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 009d3e7e8a8b4e2497de3a73d01ced49, State: Initialized, Role: FOLLOWER
I20240502 21:27:30.381522 25201 tablet_bootstrap.cc:492] T 89f86d58fe414b579845214534d338d8 P 6810a866decb4f5089f7d35c870e4204: No bootstrap required, opened a new log
I20240502 21:27:30.381717 25201 ts_tablet_manager.cc:1367] T 89f86d58fe414b579845214534d338d8 P 6810a866decb4f5089f7d35c870e4204: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.000s
I20240502 21:27:30.381734 25198 consensus_queue.cc:260] T 89f86d58fe414b579845214534d338d8 P 009d3e7e8a8b4e2497de3a73d01ced49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } }
I20240502 21:27:30.381953 25197 tablet_bootstrap.cc:654] T a274546743354e79bffa26ae7000e895 P 038c5d9d1d7a49c981c4c34fbc00d9e6: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:30.382426 25198 ts_tablet_manager.cc:1398] T 89f86d58fe414b579845214534d338d8 P 009d3e7e8a8b4e2497de3a73d01ced49: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 21:27:30.382411 25201 raft_consensus.cc:348] T 89f86d58fe414b579845214534d338d8 P 6810a866decb4f5089f7d35c870e4204 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } }
I20240502 21:27:30.382799 25201 raft_consensus.cc:374] T 89f86d58fe414b579845214534d338d8 P 6810a866decb4f5089f7d35c870e4204 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:30.382913 25201 raft_consensus.cc:729] T 89f86d58fe414b579845214534d338d8 P 6810a866decb4f5089f7d35c870e4204 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6810a866decb4f5089f7d35c870e4204, State: Initialized, Role: FOLLOWER
I20240502 21:27:30.383168 25201 consensus_queue.cc:260] T 89f86d58fe414b579845214534d338d8 P 6810a866decb4f5089f7d35c870e4204 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } }
I20240502 21:27:30.383680 25201 ts_tablet_manager.cc:1398] T 89f86d58fe414b579845214534d338d8 P 6810a866decb4f5089f7d35c870e4204: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 21:27:30.384826 25197 tablet_bootstrap.cc:492] T a274546743354e79bffa26ae7000e895 P 038c5d9d1d7a49c981c4c34fbc00d9e6: No bootstrap required, opened a new log
I20240502 21:27:30.384986 25197 ts_tablet_manager.cc:1367] T a274546743354e79bffa26ae7000e895 P 038c5d9d1d7a49c981c4c34fbc00d9e6: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.001s
I20240502 21:27:30.385694 25197 raft_consensus.cc:348] T a274546743354e79bffa26ae7000e895 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } }
I20240502 21:27:30.385886 25197 raft_consensus.cc:374] T a274546743354e79bffa26ae7000e895 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:30.385967 25197 raft_consensus.cc:729] T a274546743354e79bffa26ae7000e895 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 038c5d9d1d7a49c981c4c34fbc00d9e6, State: Initialized, Role: FOLLOWER
I20240502 21:27:30.386178 25197 consensus_queue.cc:260] T a274546743354e79bffa26ae7000e895 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } }
I20240502 21:27:30.386586 25197 ts_tablet_manager.cc:1398] T a274546743354e79bffa26ae7000e895 P 038c5d9d1d7a49c981c4c34fbc00d9e6: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 21:27:30.417872 25168 raft_consensus.cc:482] T a274546743354e79bffa26ae7000e895 P 6c194a6db6c145acafc0f3068b8ce1fd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:30.418022 25168 raft_consensus.cc:504] T a274546743354e79bffa26ae7000e895 P 6c194a6db6c145acafc0f3068b8ce1fd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } }
I20240502 21:27:30.418618 25168 leader_election.cc:288] T a274546743354e79bffa26ae7000e895 P 6c194a6db6c145acafc0f3068b8ce1fd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 038c5d9d1d7a49c981c4c34fbc00d9e6 (127.15.255.132:46813), 009d3e7e8a8b4e2497de3a73d01ced49 (127.15.255.131:37895)
I20240502 21:27:30.419003 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a274546743354e79bffa26ae7000e895" candidate_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" is_pre_election: true
I20240502 21:27:30.419013 24185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a274546743354e79bffa26ae7000e895" candidate_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" is_pre_election: true
I20240502 21:27:30.419189 24185 raft_consensus.cc:2454] T a274546743354e79bffa26ae7000e895 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6c194a6db6c145acafc0f3068b8ce1fd in term 0.
I20240502 21:27:30.419186 24316 raft_consensus.cc:2454] T a274546743354e79bffa26ae7000e895 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6c194a6db6c145acafc0f3068b8ce1fd in term 0.
I20240502 21:27:30.419572 23982 leader_election.cc:302] T a274546743354e79bffa26ae7000e895 P 6c194a6db6c145acafc0f3068b8ce1fd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 009d3e7e8a8b4e2497de3a73d01ced49, 6c194a6db6c145acafc0f3068b8ce1fd; no voters:
I20240502 21:27:30.419813 25168 raft_consensus.cc:2781] T a274546743354e79bffa26ae7000e895 P 6c194a6db6c145acafc0f3068b8ce1fd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:30.419920 25168 raft_consensus.cc:482] T a274546743354e79bffa26ae7000e895 P 6c194a6db6c145acafc0f3068b8ce1fd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:30.420006 25168 raft_consensus.cc:3037] T a274546743354e79bffa26ae7000e895 P 6c194a6db6c145acafc0f3068b8ce1fd [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:30.421744 25168 raft_consensus.cc:504] T a274546743354e79bffa26ae7000e895 P 6c194a6db6c145acafc0f3068b8ce1fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } }
I20240502 21:27:30.422207 25168 leader_election.cc:288] T a274546743354e79bffa26ae7000e895 P 6c194a6db6c145acafc0f3068b8ce1fd [CANDIDATE]: Term 1 election: Requested vote from peers 038c5d9d1d7a49c981c4c34fbc00d9e6 (127.15.255.132:46813), 009d3e7e8a8b4e2497de3a73d01ced49 (127.15.255.131:37895)
I20240502 21:27:30.422523 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a274546743354e79bffa26ae7000e895" candidate_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6"
I20240502 21:27:30.422616 24185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a274546743354e79bffa26ae7000e895" candidate_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "009d3e7e8a8b4e2497de3a73d01ced49"
I20240502 21:27:30.422693 24316 raft_consensus.cc:3037] T a274546743354e79bffa26ae7000e895 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:30.422783 24185 raft_consensus.cc:3037] T a274546743354e79bffa26ae7000e895 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:30.424449 24316 raft_consensus.cc:2454] T a274546743354e79bffa26ae7000e895 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c194a6db6c145acafc0f3068b8ce1fd in term 1.
I20240502 21:27:30.424448 24185 raft_consensus.cc:2454] T a274546743354e79bffa26ae7000e895 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c194a6db6c145acafc0f3068b8ce1fd in term 1.
I20240502 21:27:30.424799 23981 leader_election.cc:302] T a274546743354e79bffa26ae7000e895 P 6c194a6db6c145acafc0f3068b8ce1fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 038c5d9d1d7a49c981c4c34fbc00d9e6, 6c194a6db6c145acafc0f3068b8ce1fd; no voters:
I20240502 21:27:30.425118 25168 raft_consensus.cc:2781] T a274546743354e79bffa26ae7000e895 P 6c194a6db6c145acafc0f3068b8ce1fd [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:30.425418 25168 raft_consensus.cc:686] T a274546743354e79bffa26ae7000e895 P 6c194a6db6c145acafc0f3068b8ce1fd [term 1 LEADER]: Becoming Leader. State: Replica: 6c194a6db6c145acafc0f3068b8ce1fd, State: Running, Role: LEADER
I20240502 21:27:30.425668 25168 consensus_queue.cc:237] T a274546743354e79bffa26ae7000e895 P 6c194a6db6c145acafc0f3068b8ce1fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } }
I20240502 21:27:30.427752 23765 catalog_manager.cc:5484] T a274546743354e79bffa26ae7000e895 P 6c194a6db6c145acafc0f3068b8ce1fd reported cstate change: term changed from 0 to 1, leader changed from <none> to 6c194a6db6c145acafc0f3068b8ce1fd (127.15.255.130). New cstate: current_term: 1 leader_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:30.518774 25096 raft_consensus.cc:482] T 549666d19e724a96801db183a07809d8 P 6810a866decb4f5089f7d35c870e4204 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:30.518944 25096 raft_consensus.cc:504] T 549666d19e724a96801db183a07809d8 P 6810a866decb4f5089f7d35c870e4204 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } }
I20240502 21:27:30.519588 25096 leader_election.cc:288] T 549666d19e724a96801db183a07809d8 P 6810a866decb4f5089f7d35c870e4204 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 009d3e7e8a8b4e2497de3a73d01ced49 (127.15.255.131:37895), 6c194a6db6c145acafc0f3068b8ce1fd (127.15.255.130:41235)
I20240502 21:27:30.520005 24185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "549666d19e724a96801db183a07809d8" candidate_uuid: "6810a866decb4f5089f7d35c870e4204" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" is_pre_election: true
I20240502 21:27:30.520128 24048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "549666d19e724a96801db183a07809d8" candidate_uuid: "6810a866decb4f5089f7d35c870e4204" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" is_pre_election: true
I20240502 21:27:30.520210 24185 raft_consensus.cc:2454] T 549666d19e724a96801db183a07809d8 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6810a866decb4f5089f7d35c870e4204 in term 0.
I20240502 21:27:30.520371 24048 raft_consensus.cc:2454] T 549666d19e724a96801db183a07809d8 P 6c194a6db6c145acafc0f3068b8ce1fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6810a866decb4f5089f7d35c870e4204 in term 0.
I20240502 21:27:30.520709 23850 leader_election.cc:302] T 549666d19e724a96801db183a07809d8 P 6810a866decb4f5089f7d35c870e4204 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 009d3e7e8a8b4e2497de3a73d01ced49, 6810a866decb4f5089f7d35c870e4204; no voters:
I20240502 21:27:30.521104 25096 raft_consensus.cc:2781] T 549666d19e724a96801db183a07809d8 P 6810a866decb4f5089f7d35c870e4204 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:30.521229 25096 raft_consensus.cc:482] T 549666d19e724a96801db183a07809d8 P 6810a866decb4f5089f7d35c870e4204 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:30.521306 25096 raft_consensus.cc:3037] T 549666d19e724a96801db183a07809d8 P 6810a866decb4f5089f7d35c870e4204 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:30.523442 25212 raft_consensus.cc:482] T 89f86d58fe414b579845214534d338d8 P 6810a866decb4f5089f7d35c870e4204 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:30.523599 25212 raft_consensus.cc:504] T 89f86d58fe414b579845214534d338d8 P 6810a866decb4f5089f7d35c870e4204 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } }
I20240502 21:27:30.524106 25096 raft_consensus.cc:504] T 549666d19e724a96801db183a07809d8 P 6810a866decb4f5089f7d35c870e4204 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } }
I20240502 21:27:30.524502 25212 leader_election.cc:288] T 89f86d58fe414b579845214534d338d8 P 6810a866decb4f5089f7d35c870e4204 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 038c5d9d1d7a49c981c4c34fbc00d9e6 (127.15.255.132:46813), 009d3e7e8a8b4e2497de3a73d01ced49 (127.15.255.131:37895)
I20240502 21:27:30.524581 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89f86d58fe414b579845214534d338d8" candidate_uuid: "6810a866decb4f5089f7d35c870e4204" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" is_pre_election: true
I20240502 21:27:30.524794 24316 raft_consensus.cc:2454] T 89f86d58fe414b579845214534d338d8 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6810a866decb4f5089f7d35c870e4204 in term 0.
I20240502 21:27:30.525163 24185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89f86d58fe414b579845214534d338d8" candidate_uuid: "6810a866decb4f5089f7d35c870e4204" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" is_pre_election: true
I20240502 21:27:30.525219 24184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "549666d19e724a96801db183a07809d8" candidate_uuid: "6810a866decb4f5089f7d35c870e4204" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "009d3e7e8a8b4e2497de3a73d01ced49"
I20240502 21:27:30.525226 24048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "549666d19e724a96801db183a07809d8" candidate_uuid: "6810a866decb4f5089f7d35c870e4204" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c194a6db6c145acafc0f3068b8ce1fd"
I20240502 21:27:30.525378 24184 raft_consensus.cc:3037] T 549666d19e724a96801db183a07809d8 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:30.525424 24048 raft_consensus.cc:3037] T 549666d19e724a96801db183a07809d8 P 6c194a6db6c145acafc0f3068b8ce1fd [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:30.525514 23849 leader_election.cc:302] T 89f86d58fe414b579845214534d338d8 P 6810a866decb4f5089f7d35c870e4204 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 038c5d9d1d7a49c981c4c34fbc00d9e6, 6810a866decb4f5089f7d35c870e4204; no voters:
I20240502 21:27:30.525825 25212 raft_consensus.cc:2781] T 89f86d58fe414b579845214534d338d8 P 6810a866decb4f5089f7d35c870e4204 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:30.525940 25212 raft_consensus.cc:482] T 89f86d58fe414b579845214534d338d8 P 6810a866decb4f5089f7d35c870e4204 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:30.526029 25212 raft_consensus.cc:3037] T 89f86d58fe414b579845214534d338d8 P 6810a866decb4f5089f7d35c870e4204 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:30.528026 24184 raft_consensus.cc:2454] T 549666d19e724a96801db183a07809d8 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6810a866decb4f5089f7d35c870e4204 in term 1.
I20240502 21:27:30.528185 24048 raft_consensus.cc:2454] T 549666d19e724a96801db183a07809d8 P 6c194a6db6c145acafc0f3068b8ce1fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6810a866decb4f5089f7d35c870e4204 in term 1.
I20240502 21:27:30.528401 23850 leader_election.cc:302] T 549666d19e724a96801db183a07809d8 P 6810a866decb4f5089f7d35c870e4204 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 009d3e7e8a8b4e2497de3a73d01ced49, 6810a866decb4f5089f7d35c870e4204; no voters:
I20240502 21:27:30.528561 25212 raft_consensus.cc:504] T 89f86d58fe414b579845214534d338d8 P 6810a866decb4f5089f7d35c870e4204 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } }
I20240502 21:27:30.528785 25096 leader_election.cc:288] T 549666d19e724a96801db183a07809d8 P 6810a866decb4f5089f7d35c870e4204 [CANDIDATE]: Term 1 election: Requested vote from peers 009d3e7e8a8b4e2497de3a73d01ced49 (127.15.255.131:37895), 6c194a6db6c145acafc0f3068b8ce1fd (127.15.255.130:41235)
I20240502 21:27:30.528975 25096 raft_consensus.cc:2781] T 549666d19e724a96801db183a07809d8 P 6810a866decb4f5089f7d35c870e4204 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:30.529081 25096 raft_consensus.cc:686] T 549666d19e724a96801db183a07809d8 P 6810a866decb4f5089f7d35c870e4204 [term 1 LEADER]: Becoming Leader. State: Replica: 6810a866decb4f5089f7d35c870e4204, State: Running, Role: LEADER
I20240502 21:27:30.529310 25212 leader_election.cc:288] T 89f86d58fe414b579845214534d338d8 P 6810a866decb4f5089f7d35c870e4204 [CANDIDATE]: Term 1 election: Requested vote from peers 038c5d9d1d7a49c981c4c34fbc00d9e6 (127.15.255.132:46813), 009d3e7e8a8b4e2497de3a73d01ced49 (127.15.255.131:37895)
I20240502 21:27:30.529472 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89f86d58fe414b579845214534d338d8" candidate_uuid: "6810a866decb4f5089f7d35c870e4204" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6"
I20240502 21:27:30.525375 24185 raft_consensus.cc:2454] T 89f86d58fe414b579845214534d338d8 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6810a866decb4f5089f7d35c870e4204 in term 0.
I20240502 21:27:30.529640 24316 raft_consensus.cc:3037] T 89f86d58fe414b579845214534d338d8 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:30.529345 25096 consensus_queue.cc:237] T 549666d19e724a96801db183a07809d8 P 6810a866decb4f5089f7d35c870e4204 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } }
I20240502 21:27:30.532994 24316 raft_consensus.cc:2454] T 89f86d58fe414b579845214534d338d8 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6810a866decb4f5089f7d35c870e4204 in term 1.
I20240502 21:27:30.533350 24185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89f86d58fe414b579845214534d338d8" candidate_uuid: "6810a866decb4f5089f7d35c870e4204" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "009d3e7e8a8b4e2497de3a73d01ced49"
I20240502 21:27:30.533414 23849 leader_election.cc:302] T 89f86d58fe414b579845214534d338d8 P 6810a866decb4f5089f7d35c870e4204 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 038c5d9d1d7a49c981c4c34fbc00d9e6, 6810a866decb4f5089f7d35c870e4204; no voters:
I20240502 21:27:30.533524 24185 raft_consensus.cc:3037] T 89f86d58fe414b579845214534d338d8 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:30.533746 25213 raft_consensus.cc:2781] T 89f86d58fe414b579845214534d338d8 P 6810a866decb4f5089f7d35c870e4204 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:30.533895 25213 raft_consensus.cc:686] T 89f86d58fe414b579845214534d338d8 P 6810a866decb4f5089f7d35c870e4204 [term 1 LEADER]: Becoming Leader. State: Replica: 6810a866decb4f5089f7d35c870e4204, State: Running, Role: LEADER
I20240502 21:27:30.534147 25213 consensus_queue.cc:237] T 89f86d58fe414b579845214534d338d8 P 6810a866decb4f5089f7d35c870e4204 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } }
I20240502 21:27:30.537853 24185 raft_consensus.cc:2454] T 89f86d58fe414b579845214534d338d8 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6810a866decb4f5089f7d35c870e4204 in term 1.
I20240502 21:27:30.539032 23765 catalog_manager.cc:5484] T 549666d19e724a96801db183a07809d8 P 6810a866decb4f5089f7d35c870e4204 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6810a866decb4f5089f7d35c870e4204 (127.15.255.129). New cstate: current_term: 1 leader_uuid: "6810a866decb4f5089f7d35c870e4204" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } health_report { overall_health: HEALTHY } } }
I20240502 21:27:30.543243 23765 catalog_manager.cc:5484] T 89f86d58fe414b579845214534d338d8 P 6810a866decb4f5089f7d35c870e4204 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6810a866decb4f5089f7d35c870e4204 (127.15.255.129). New cstate: current_term: 1 leader_uuid: "6810a866decb4f5089f7d35c870e4204" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } health_report { overall_health: UNKNOWN } } }
W20240502 21:27:30.908463 25216 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:27:30.908771 25216 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:27:30.916056 25216 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 21:27:30.933655 25212 consensus_queue.cc:1035] T 549666d19e724a96801db183a07809d8 P 6810a866decb4f5089f7d35c870e4204 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:30.943172 25212 consensus_queue.cc:1035] T 549666d19e724a96801db183a07809d8 P 6810a866decb4f5089f7d35c870e4204 [LEADER]: Connected to new peer: Peer: permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_3'
I20240502 21:27:30.995386 25168 consensus_queue.cc:1035] T a274546743354e79bffa26ae7000e895 P 6c194a6db6c145acafc0f3068b8ce1fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:31.001358 25168 consensus_queue.cc:1035] T a274546743354e79bffa26ae7000e895 P 6c194a6db6c145acafc0f3068b8ce1fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:31.048379 25212 consensus_queue.cc:1035] T 89f86d58fe414b579845214534d338d8 P 6810a866decb4f5089f7d35c870e4204 [LEADER]: Connected to new peer: Peer: permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:31.066123 25212 consensus_queue.cc:1035] T 89f86d58fe414b579845214534d338d8 P 6810a866decb4f5089f7d35c870e4204 [LEADER]: Connected to new peer: Peer: permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 248.703 ms
time per row: 0.124352 ms
I20240502 21:27:31.300055 23767 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41618:
name: "rebalance_test_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 21:27:31.308598 23896 tablet_service.cc:1467] Processing CreateTablet for tablet 30236241b670443489240f191d9b9d50 (DEFAULT_TABLE table=rebalance_test_table_4 [id=9d2b330e886e4e83a437506bf99d77a3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:31.309047 23896 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30236241b670443489240f191d9b9d50. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:31.308969 24296 tablet_service.cc:1467] Processing CreateTablet for tablet 30236241b670443489240f191d9b9d50 (DEFAULT_TABLE table=rebalance_test_table_4 [id=9d2b330e886e4e83a437506bf99d77a3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:31.309214 24027 tablet_service.cc:1467] Processing CreateTablet for tablet 30236241b670443489240f191d9b9d50 (DEFAULT_TABLE table=rebalance_test_table_4 [id=9d2b330e886e4e83a437506bf99d77a3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:31.309386 24296 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30236241b670443489240f191d9b9d50. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:31.309676 24027 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30236241b670443489240f191d9b9d50. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:31.313242 24295 tablet_service.cc:1467] Processing CreateTablet for tablet 8f14db150fda449e962be2466f1d2a0e (DEFAULT_TABLE table=rebalance_test_table_4 [id=9d2b330e886e4e83a437506bf99d77a3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:31.313604 24295 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f14db150fda449e962be2466f1d2a0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:31.314411 25284 tablet_bootstrap.cc:492] T 30236241b670443489240f191d9b9d50 P 6c194a6db6c145acafc0f3068b8ce1fd: Bootstrap starting.
I20240502 21:27:31.314924 24164 tablet_service.cc:1467] Processing CreateTablet for tablet 3cb3e62ca0e94e659efb3b7b0c2c9c91 (DEFAULT_TABLE table=rebalance_test_table_4 [id=9d2b330e886e4e83a437506bf99d77a3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:31.315308 24164 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3cb3e62ca0e94e659efb3b7b0c2c9c91. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:31.316010 25284 tablet_bootstrap.cc:654] T 30236241b670443489240f191d9b9d50 P 6c194a6db6c145acafc0f3068b8ce1fd: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:31.320798 24163 tablet_service.cc:1467] Processing CreateTablet for tablet 8f14db150fda449e962be2466f1d2a0e (DEFAULT_TABLE table=rebalance_test_table_4 [id=9d2b330e886e4e83a437506bf99d77a3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:31.321158 24163 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f14db150fda449e962be2466f1d2a0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:31.326089 25287 tablet_bootstrap.cc:492] T 30236241b670443489240f191d9b9d50 P 6810a866decb4f5089f7d35c870e4204: Bootstrap starting.
I20240502 21:27:31.326709 23895 tablet_service.cc:1467] Processing CreateTablet for tablet 3cb3e62ca0e94e659efb3b7b0c2c9c91 (DEFAULT_TABLE table=rebalance_test_table_4 [id=9d2b330e886e4e83a437506bf99d77a3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:31.326930 24029 tablet_service.cc:1467] Processing CreateTablet for tablet 3cb3e62ca0e94e659efb3b7b0c2c9c91 (DEFAULT_TABLE table=rebalance_test_table_4 [id=9d2b330e886e4e83a437506bf99d77a3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:31.327087 23895 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3cb3e62ca0e94e659efb3b7b0c2c9c91. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:31.327342 24029 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3cb3e62ca0e94e659efb3b7b0c2c9c91. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:31.330142 25287 tablet_bootstrap.cc:654] T 30236241b670443489240f191d9b9d50 P 6810a866decb4f5089f7d35c870e4204: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:31.331494 23894 tablet_service.cc:1467] Processing CreateTablet for tablet 8f14db150fda449e962be2466f1d2a0e (DEFAULT_TABLE table=rebalance_test_table_4 [id=9d2b330e886e4e83a437506bf99d77a3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:31.331861 23894 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f14db150fda449e962be2466f1d2a0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:31.340296 25285 tablet_bootstrap.cc:492] T 30236241b670443489240f191d9b9d50 P 038c5d9d1d7a49c981c4c34fbc00d9e6: Bootstrap starting.
I20240502 21:27:31.345763 25285 tablet_bootstrap.cc:654] T 30236241b670443489240f191d9b9d50 P 038c5d9d1d7a49c981c4c34fbc00d9e6: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:31.353132 25290 tablet_bootstrap.cc:492] T 3cb3e62ca0e94e659efb3b7b0c2c9c91 P 009d3e7e8a8b4e2497de3a73d01ced49: Bootstrap starting.
I20240502 21:27:31.353817 25285 tablet_bootstrap.cc:492] T 30236241b670443489240f191d9b9d50 P 038c5d9d1d7a49c981c4c34fbc00d9e6: No bootstrap required, opened a new log
I20240502 21:27:31.354023 25285 ts_tablet_manager.cc:1367] T 30236241b670443489240f191d9b9d50 P 038c5d9d1d7a49c981c4c34fbc00d9e6: Time spent bootstrapping tablet: real 0.014s user 0.010s sys 0.000s
I20240502 21:27:31.354933 25285 raft_consensus.cc:348] T 30236241b670443489240f191d9b9d50 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } }
I20240502 21:27:31.355266 25285 raft_consensus.cc:374] T 30236241b670443489240f191d9b9d50 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:31.355443 25290 tablet_bootstrap.cc:654] T 3cb3e62ca0e94e659efb3b7b0c2c9c91 P 009d3e7e8a8b4e2497de3a73d01ced49: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:31.355520 25285 raft_consensus.cc:729] T 30236241b670443489240f191d9b9d50 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 038c5d9d1d7a49c981c4c34fbc00d9e6, State: Initialized, Role: FOLLOWER
I20240502 21:27:31.355865 25285 consensus_queue.cc:260] T 30236241b670443489240f191d9b9d50 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } }
I20240502 21:27:31.356690 25285 ts_tablet_manager.cc:1398] T 30236241b670443489240f191d9b9d50 P 038c5d9d1d7a49c981c4c34fbc00d9e6: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 21:27:31.357002 25285 tablet_bootstrap.cc:492] T 8f14db150fda449e962be2466f1d2a0e P 038c5d9d1d7a49c981c4c34fbc00d9e6: Bootstrap starting.
I20240502 21:27:31.358513 25285 tablet_bootstrap.cc:654] T 8f14db150fda449e962be2466f1d2a0e P 038c5d9d1d7a49c981c4c34fbc00d9e6: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:31.362112 25287 tablet_bootstrap.cc:492] T 30236241b670443489240f191d9b9d50 P 6810a866decb4f5089f7d35c870e4204: No bootstrap required, opened a new log
I20240502 21:27:31.362278 25287 ts_tablet_manager.cc:1367] T 30236241b670443489240f191d9b9d50 P 6810a866decb4f5089f7d35c870e4204: Time spent bootstrapping tablet: real 0.036s user 0.018s sys 0.007s
I20240502 21:27:31.362358 25290 tablet_bootstrap.cc:492] T 3cb3e62ca0e94e659efb3b7b0c2c9c91 P 009d3e7e8a8b4e2497de3a73d01ced49: No bootstrap required, opened a new log
I20240502 21:27:31.362519 25290 ts_tablet_manager.cc:1367] T 3cb3e62ca0e94e659efb3b7b0c2c9c91 P 009d3e7e8a8b4e2497de3a73d01ced49: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s
I20240502 21:27:31.362811 25285 tablet_bootstrap.cc:492] T 8f14db150fda449e962be2466f1d2a0e P 038c5d9d1d7a49c981c4c34fbc00d9e6: No bootstrap required, opened a new log
I20240502 21:27:31.362957 25285 ts_tablet_manager.cc:1367] T 8f14db150fda449e962be2466f1d2a0e P 038c5d9d1d7a49c981c4c34fbc00d9e6: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.003s
I20240502 21:27:31.363049 25287 raft_consensus.cc:348] T 30236241b670443489240f191d9b9d50 P 6810a866decb4f5089f7d35c870e4204 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } }
I20240502 21:27:31.363236 25287 raft_consensus.cc:374] T 30236241b670443489240f191d9b9d50 P 6810a866decb4f5089f7d35c870e4204 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:31.363224 25290 raft_consensus.cc:348] T 3cb3e62ca0e94e659efb3b7b0c2c9c91 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } }
I20240502 21:27:31.363373 25287 raft_consensus.cc:729] T 30236241b670443489240f191d9b9d50 P 6810a866decb4f5089f7d35c870e4204 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6810a866decb4f5089f7d35c870e4204, State: Initialized, Role: FOLLOWER
I20240502 21:27:31.363418 25290 raft_consensus.cc:374] T 3cb3e62ca0e94e659efb3b7b0c2c9c91 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:31.363507 25290 raft_consensus.cc:729] T 3cb3e62ca0e94e659efb3b7b0c2c9c91 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 009d3e7e8a8b4e2497de3a73d01ced49, State: Initialized, Role: FOLLOWER
I20240502 21:27:31.363617 25287 consensus_queue.cc:260] T 30236241b670443489240f191d9b9d50 P 6810a866decb4f5089f7d35c870e4204 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } }
I20240502 21:27:31.363718 25290 consensus_queue.cc:260] T 3cb3e62ca0e94e659efb3b7b0c2c9c91 P 009d3e7e8a8b4e2497de3a73d01ced49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } }
I20240502 21:27:31.363917 25285 raft_consensus.cc:348] T 8f14db150fda449e962be2466f1d2a0e P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } }
I20240502 21:27:31.364106 25285 raft_consensus.cc:374] T 8f14db150fda449e962be2466f1d2a0e P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:31.364172 25285 raft_consensus.cc:729] T 8f14db150fda449e962be2466f1d2a0e P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 038c5d9d1d7a49c981c4c34fbc00d9e6, State: Initialized, Role: FOLLOWER
I20240502 21:27:31.364181 25290 ts_tablet_manager.cc:1398] T 3cb3e62ca0e94e659efb3b7b0c2c9c91 P 009d3e7e8a8b4e2497de3a73d01ced49: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 21:27:31.364408 25285 consensus_queue.cc:260] T 8f14db150fda449e962be2466f1d2a0e P 038c5d9d1d7a49c981c4c34fbc00d9e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } }
I20240502 21:27:31.364980 25287 ts_tablet_manager.cc:1398] T 30236241b670443489240f191d9b9d50 P 6810a866decb4f5089f7d35c870e4204: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 21:27:31.364987 25285 ts_tablet_manager.cc:1398] T 8f14db150fda449e962be2466f1d2a0e P 038c5d9d1d7a49c981c4c34fbc00d9e6: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:27:31.365238 25287 tablet_bootstrap.cc:492] T 3cb3e62ca0e94e659efb3b7b0c2c9c91 P 6810a866decb4f5089f7d35c870e4204: Bootstrap starting.
I20240502 21:27:31.365258 25284 tablet_bootstrap.cc:492] T 30236241b670443489240f191d9b9d50 P 6c194a6db6c145acafc0f3068b8ce1fd: No bootstrap required, opened a new log
I20240502 21:27:31.365406 25284 ts_tablet_manager.cc:1367] T 30236241b670443489240f191d9b9d50 P 6c194a6db6c145acafc0f3068b8ce1fd: Time spent bootstrapping tablet: real 0.051s user 0.012s sys 0.033s
I20240502 21:27:31.366151 25284 raft_consensus.cc:348] T 30236241b670443489240f191d9b9d50 P 6c194a6db6c145acafc0f3068b8ce1fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } }
I20240502 21:27:31.366358 25284 raft_consensus.cc:374] T 30236241b670443489240f191d9b9d50 P 6c194a6db6c145acafc0f3068b8ce1fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:31.366451 25284 raft_consensus.cc:729] T 30236241b670443489240f191d9b9d50 P 6c194a6db6c145acafc0f3068b8ce1fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c194a6db6c145acafc0f3068b8ce1fd, State: Initialized, Role: FOLLOWER
I20240502 21:27:31.366652 25287 tablet_bootstrap.cc:654] T 3cb3e62ca0e94e659efb3b7b0c2c9c91 P 6810a866decb4f5089f7d35c870e4204: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:31.367159 25290 tablet_bootstrap.cc:492] T 8f14db150fda449e962be2466f1d2a0e P 009d3e7e8a8b4e2497de3a73d01ced49: Bootstrap starting.
I20240502 21:27:31.368036 25284 consensus_queue.cc:260] T 30236241b670443489240f191d9b9d50 P 6c194a6db6c145acafc0f3068b8ce1fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } }
I20240502 21:27:31.369416 25284 ts_tablet_manager.cc:1398] T 30236241b670443489240f191d9b9d50 P 6c194a6db6c145acafc0f3068b8ce1fd: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240502 21:27:31.369728 25284 tablet_bootstrap.cc:492] T 3cb3e62ca0e94e659efb3b7b0c2c9c91 P 6c194a6db6c145acafc0f3068b8ce1fd: Bootstrap starting.
I20240502 21:27:31.371368 25284 tablet_bootstrap.cc:654] T 3cb3e62ca0e94e659efb3b7b0c2c9c91 P 6c194a6db6c145acafc0f3068b8ce1fd: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:31.373353 25290 tablet_bootstrap.cc:654] T 8f14db150fda449e962be2466f1d2a0e P 009d3e7e8a8b4e2497de3a73d01ced49: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:31.373363 25284 tablet_bootstrap.cc:492] T 3cb3e62ca0e94e659efb3b7b0c2c9c91 P 6c194a6db6c145acafc0f3068b8ce1fd: No bootstrap required, opened a new log
I20240502 21:27:31.373509 25284 ts_tablet_manager.cc:1367] T 3cb3e62ca0e94e659efb3b7b0c2c9c91 P 6c194a6db6c145acafc0f3068b8ce1fd: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.001s
I20240502 21:27:31.374183 25284 raft_consensus.cc:348] T 3cb3e62ca0e94e659efb3b7b0c2c9c91 P 6c194a6db6c145acafc0f3068b8ce1fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } }
I20240502 21:27:31.374369 25284 raft_consensus.cc:374] T 3cb3e62ca0e94e659efb3b7b0c2c9c91 P 6c194a6db6c145acafc0f3068b8ce1fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:31.374442 25284 raft_consensus.cc:729] T 3cb3e62ca0e94e659efb3b7b0c2c9c91 P 6c194a6db6c145acafc0f3068b8ce1fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c194a6db6c145acafc0f3068b8ce1fd, State: Initialized, Role: FOLLOWER
I20240502 21:27:31.374668 25284 consensus_queue.cc:260] T 3cb3e62ca0e94e659efb3b7b0c2c9c91 P 6c194a6db6c145acafc0f3068b8ce1fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } }
I20240502 21:27:31.375038 25290 tablet_bootstrap.cc:492] T 8f14db150fda449e962be2466f1d2a0e P 009d3e7e8a8b4e2497de3a73d01ced49: No bootstrap required, opened a new log
I20240502 21:27:31.375120 25284 ts_tablet_manager.cc:1398] T 3cb3e62ca0e94e659efb3b7b0c2c9c91 P 6c194a6db6c145acafc0f3068b8ce1fd: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 21:27:31.375188 25290 ts_tablet_manager.cc:1367] T 8f14db150fda449e962be2466f1d2a0e P 009d3e7e8a8b4e2497de3a73d01ced49: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240502 21:27:31.376096 25287 tablet_bootstrap.cc:492] T 3cb3e62ca0e94e659efb3b7b0c2c9c91 P 6810a866decb4f5089f7d35c870e4204: No bootstrap required, opened a new log
I20240502 21:27:31.376260 25287 ts_tablet_manager.cc:1367] T 3cb3e62ca0e94e659efb3b7b0c2c9c91 P 6810a866decb4f5089f7d35c870e4204: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.004s
I20240502 21:27:31.376173 25290 raft_consensus.cc:348] T 8f14db150fda449e962be2466f1d2a0e P 009d3e7e8a8b4e2497de3a73d01ced49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } }
I20240502 21:27:31.376366 25290 raft_consensus.cc:374] T 8f14db150fda449e962be2466f1d2a0e P 009d3e7e8a8b4e2497de3a73d01ced49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:31.376466 25290 raft_consensus.cc:729] T 8f14db150fda449e962be2466f1d2a0e P 009d3e7e8a8b4e2497de3a73d01ced49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 009d3e7e8a8b4e2497de3a73d01ced49, State: Initialized, Role: FOLLOWER
I20240502 21:27:31.376698 25290 consensus_queue.cc:260] T 8f14db150fda449e962be2466f1d2a0e P 009d3e7e8a8b4e2497de3a73d01ced49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } }
I20240502 21:27:31.376988 25287 raft_consensus.cc:348] T 3cb3e62ca0e94e659efb3b7b0c2c9c91 P 6810a866decb4f5089f7d35c870e4204 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } }
I20240502 21:27:31.377203 25290 ts_tablet_manager.cc:1398] T 8f14db150fda449e962be2466f1d2a0e P 009d3e7e8a8b4e2497de3a73d01ced49: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 21:27:31.377197 25287 raft_consensus.cc:374] T 3cb3e62ca0e94e659efb3b7b0c2c9c91 P 6810a866decb4f5089f7d35c870e4204 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:31.377283 25287 raft_consensus.cc:729] T 3cb3e62ca0e94e659efb3b7b0c2c9c91 P 6810a866decb4f5089f7d35c870e4204 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6810a866decb4f5089f7d35c870e4204, State: Initialized, Role: FOLLOWER
I20240502 21:27:31.377516 25287 consensus_queue.cc:260] T 3cb3e62ca0e94e659efb3b7b0c2c9c91 P 6810a866decb4f5089f7d35c870e4204 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } }
I20240502 21:27:31.377935 25287 ts_tablet_manager.cc:1398] T 3cb3e62ca0e94e659efb3b7b0c2c9c91 P 6810a866decb4f5089f7d35c870e4204: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 21:27:31.378167 25287 tablet_bootstrap.cc:492] T 8f14db150fda449e962be2466f1d2a0e P 6810a866decb4f5089f7d35c870e4204: Bootstrap starting.
I20240502 21:27:31.379477 25287 tablet_bootstrap.cc:654] T 8f14db150fda449e962be2466f1d2a0e P 6810a866decb4f5089f7d35c870e4204: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:31.381368 25287 tablet_bootstrap.cc:492] T 8f14db150fda449e962be2466f1d2a0e P 6810a866decb4f5089f7d35c870e4204: No bootstrap required, opened a new log
I20240502 21:27:31.381491 25287 ts_tablet_manager.cc:1367] T 8f14db150fda449e962be2466f1d2a0e P 6810a866decb4f5089f7d35c870e4204: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240502 21:27:31.382074 25287 raft_consensus.cc:348] T 8f14db150fda449e962be2466f1d2a0e P 6810a866decb4f5089f7d35c870e4204 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } }
I20240502 21:27:31.382241 25287 raft_consensus.cc:374] T 8f14db150fda449e962be2466f1d2a0e P 6810a866decb4f5089f7d35c870e4204 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:31.382335 25287 raft_consensus.cc:729] T 8f14db150fda449e962be2466f1d2a0e P 6810a866decb4f5089f7d35c870e4204 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6810a866decb4f5089f7d35c870e4204, State: Initialized, Role: FOLLOWER
I20240502 21:27:31.382522 25287 consensus_queue.cc:260] T 8f14db150fda449e962be2466f1d2a0e P 6810a866decb4f5089f7d35c870e4204 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } }
I20240502 21:27:31.382956 25287 ts_tablet_manager.cc:1398] T 8f14db150fda449e962be2466f1d2a0e P 6810a866decb4f5089f7d35c870e4204: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 21:27:31.401674 25265 raft_consensus.cc:482] T 30236241b670443489240f191d9b9d50 P 6810a866decb4f5089f7d35c870e4204 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:31.401804 25265 raft_consensus.cc:504] T 30236241b670443489240f191d9b9d50 P 6810a866decb4f5089f7d35c870e4204 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } }
I20240502 21:27:31.402462 25265 leader_election.cc:288] T 30236241b670443489240f191d9b9d50 P 6810a866decb4f5089f7d35c870e4204 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 038c5d9d1d7a49c981c4c34fbc00d9e6 (127.15.255.132:46813), 6c194a6db6c145acafc0f3068b8ce1fd (127.15.255.130:41235)
I20240502 21:27:31.402890 24315 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30236241b670443489240f191d9b9d50" candidate_uuid: "6810a866decb4f5089f7d35c870e4204" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" is_pre_election: true
I20240502 21:27:31.402964 24049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30236241b670443489240f191d9b9d50" candidate_uuid: "6810a866decb4f5089f7d35c870e4204" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" is_pre_election: true
I20240502 21:27:31.403110 24315 raft_consensus.cc:2454] T 30236241b670443489240f191d9b9d50 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6810a866decb4f5089f7d35c870e4204 in term 0.
I20240502 21:27:31.403173 24049 raft_consensus.cc:2454] T 30236241b670443489240f191d9b9d50 P 6c194a6db6c145acafc0f3068b8ce1fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6810a866decb4f5089f7d35c870e4204 in term 0.
I20240502 21:27:31.403493 23849 leader_election.cc:302] T 30236241b670443489240f191d9b9d50 P 6810a866decb4f5089f7d35c870e4204 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 038c5d9d1d7a49c981c4c34fbc00d9e6, 6810a866decb4f5089f7d35c870e4204; no voters:
I20240502 21:27:31.403784 25265 raft_consensus.cc:2781] T 30236241b670443489240f191d9b9d50 P 6810a866decb4f5089f7d35c870e4204 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:31.403914 25265 raft_consensus.cc:482] T 30236241b670443489240f191d9b9d50 P 6810a866decb4f5089f7d35c870e4204 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:31.403993 25265 raft_consensus.cc:3037] T 30236241b670443489240f191d9b9d50 P 6810a866decb4f5089f7d35c870e4204 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:31.405841 25265 raft_consensus.cc:504] T 30236241b670443489240f191d9b9d50 P 6810a866decb4f5089f7d35c870e4204 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } }
I20240502 21:27:31.405925 25246 raft_consensus.cc:482] T 3cb3e62ca0e94e659efb3b7b0c2c9c91 P 6c194a6db6c145acafc0f3068b8ce1fd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:31.406057 25246 raft_consensus.cc:504] T 3cb3e62ca0e94e659efb3b7b0c2c9c91 P 6c194a6db6c145acafc0f3068b8ce1fd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } }
I20240502 21:27:31.406456 25265 leader_election.cc:288] T 30236241b670443489240f191d9b9d50 P 6810a866decb4f5089f7d35c870e4204 [CANDIDATE]: Term 1 election: Requested vote from peers 038c5d9d1d7a49c981c4c34fbc00d9e6 (127.15.255.132:46813), 6c194a6db6c145acafc0f3068b8ce1fd (127.15.255.130:41235)
I20240502 21:27:31.406752 25246 leader_election.cc:288] T 3cb3e62ca0e94e659efb3b7b0c2c9c91 P 6c194a6db6c145acafc0f3068b8ce1fd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 009d3e7e8a8b4e2497de3a73d01ced49 (127.15.255.131:37895), 6810a866decb4f5089f7d35c870e4204 (127.15.255.129:40435)
I20240502 21:27:31.406791 24315 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30236241b670443489240f191d9b9d50" candidate_uuid: "6810a866decb4f5089f7d35c870e4204" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6"
I20240502 21:27:31.406970 24315 raft_consensus.cc:3037] T 30236241b670443489240f191d9b9d50 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:31.407374 24185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cb3e62ca0e94e659efb3b7b0c2c9c91" candidate_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" is_pre_election: true
I20240502 21:27:31.407616 24185 raft_consensus.cc:2454] T 3cb3e62ca0e94e659efb3b7b0c2c9c91 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6c194a6db6c145acafc0f3068b8ce1fd in term 0.
I20240502 21:27:31.407994 23982 leader_election.cc:302] T 3cb3e62ca0e94e659efb3b7b0c2c9c91 P 6c194a6db6c145acafc0f3068b8ce1fd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 009d3e7e8a8b4e2497de3a73d01ced49, 6c194a6db6c145acafc0f3068b8ce1fd; no voters:
I20240502 21:27:31.408221 25246 raft_consensus.cc:2781] T 3cb3e62ca0e94e659efb3b7b0c2c9c91 P 6c194a6db6c145acafc0f3068b8ce1fd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:31.408330 25246 raft_consensus.cc:482] T 3cb3e62ca0e94e659efb3b7b0c2c9c91 P 6c194a6db6c145acafc0f3068b8ce1fd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:31.408448 25246 raft_consensus.cc:3037] T 3cb3e62ca0e94e659efb3b7b0c2c9c91 P 6c194a6db6c145acafc0f3068b8ce1fd [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:31.408928 23915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cb3e62ca0e94e659efb3b7b0c2c9c91" candidate_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6810a866decb4f5089f7d35c870e4204" is_pre_election: true
I20240502 21:27:31.409106 23915 raft_consensus.cc:2454] T 3cb3e62ca0e94e659efb3b7b0c2c9c91 P 6810a866decb4f5089f7d35c870e4204 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6c194a6db6c145acafc0f3068b8ce1fd in term 0.
I20240502 21:27:31.409659 24049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30236241b670443489240f191d9b9d50" candidate_uuid: "6810a866decb4f5089f7d35c870e4204" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c194a6db6c145acafc0f3068b8ce1fd"
I20240502 21:27:31.409839 24049 raft_consensus.cc:3037] T 30236241b670443489240f191d9b9d50 P 6c194a6db6c145acafc0f3068b8ce1fd [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:31.411130 25246 raft_consensus.cc:504] T 3cb3e62ca0e94e659efb3b7b0c2c9c91 P 6c194a6db6c145acafc0f3068b8ce1fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } }
I20240502 21:27:31.411957 25246 leader_election.cc:288] T 3cb3e62ca0e94e659efb3b7b0c2c9c91 P 6c194a6db6c145acafc0f3068b8ce1fd [CANDIDATE]: Term 1 election: Requested vote from peers 009d3e7e8a8b4e2497de3a73d01ced49 (127.15.255.131:37895), 6810a866decb4f5089f7d35c870e4204 (127.15.255.129:40435)
I20240502 21:27:31.412096 24185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cb3e62ca0e94e659efb3b7b0c2c9c91" candidate_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "009d3e7e8a8b4e2497de3a73d01ced49"
I20240502 21:27:31.412284 24185 raft_consensus.cc:3037] T 3cb3e62ca0e94e659efb3b7b0c2c9c91 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:31.412357 24049 raft_consensus.cc:2454] T 30236241b670443489240f191d9b9d50 P 6c194a6db6c145acafc0f3068b8ce1fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6810a866decb4f5089f7d35c870e4204 in term 1.
I20240502 21:27:31.412940 23851 leader_election.cc:302] T 30236241b670443489240f191d9b9d50 P 6810a866decb4f5089f7d35c870e4204 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6810a866decb4f5089f7d35c870e4204, 6c194a6db6c145acafc0f3068b8ce1fd; no voters:
I20240502 21:27:31.413197 25265 raft_consensus.cc:2781] T 30236241b670443489240f191d9b9d50 P 6810a866decb4f5089f7d35c870e4204 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:31.413282 23915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3cb3e62ca0e94e659efb3b7b0c2c9c91" candidate_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6810a866decb4f5089f7d35c870e4204"
I20240502 21:27:31.413452 23915 raft_consensus.cc:3037] T 3cb3e62ca0e94e659efb3b7b0c2c9c91 P 6810a866decb4f5089f7d35c870e4204 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:31.413455 25265 raft_consensus.cc:686] T 30236241b670443489240f191d9b9d50 P 6810a866decb4f5089f7d35c870e4204 [term 1 LEADER]: Becoming Leader. State: Replica: 6810a866decb4f5089f7d35c870e4204, State: Running, Role: LEADER
I20240502 21:27:31.413805 25265 consensus_queue.cc:237] T 30236241b670443489240f191d9b9d50 P 6810a866decb4f5089f7d35c870e4204 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } }
I20240502 21:27:31.415000 24185 raft_consensus.cc:2454] T 3cb3e62ca0e94e659efb3b7b0c2c9c91 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c194a6db6c145acafc0f3068b8ce1fd in term 1.
I20240502 21:27:31.415709 23982 leader_election.cc:302] T 3cb3e62ca0e94e659efb3b7b0c2c9c91 P 6c194a6db6c145acafc0f3068b8ce1fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 009d3e7e8a8b4e2497de3a73d01ced49, 6c194a6db6c145acafc0f3068b8ce1fd; no voters:
I20240502 21:27:31.415961 25246 raft_consensus.cc:2781] T 3cb3e62ca0e94e659efb3b7b0c2c9c91 P 6c194a6db6c145acafc0f3068b8ce1fd [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:31.416146 23915 raft_consensus.cc:2454] T 3cb3e62ca0e94e659efb3b7b0c2c9c91 P 6810a866decb4f5089f7d35c870e4204 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c194a6db6c145acafc0f3068b8ce1fd in term 1.
I20240502 21:27:31.416199 25246 raft_consensus.cc:686] T 3cb3e62ca0e94e659efb3b7b0c2c9c91 P 6c194a6db6c145acafc0f3068b8ce1fd [term 1 LEADER]: Becoming Leader. State: Replica: 6c194a6db6c145acafc0f3068b8ce1fd, State: Running, Role: LEADER
I20240502 21:27:31.416903 25246 consensus_queue.cc:237] T 3cb3e62ca0e94e659efb3b7b0c2c9c91 P 6c194a6db6c145acafc0f3068b8ce1fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } }
I20240502 21:27:31.417953 24315 raft_consensus.cc:2454] T 30236241b670443489240f191d9b9d50 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6810a866decb4f5089f7d35c870e4204 in term 1.
I20240502 21:27:31.420672 23772 catalog_manager.cc:5484] T 3cb3e62ca0e94e659efb3b7b0c2c9c91 P 6c194a6db6c145acafc0f3068b8ce1fd reported cstate change: term changed from 0 to 1, leader changed from <none> to 6c194a6db6c145acafc0f3068b8ce1fd (127.15.255.130). New cstate: current_term: 1 leader_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } health_report { overall_health: HEALTHY } } }
I20240502 21:27:31.420627 23770 catalog_manager.cc:5484] T 30236241b670443489240f191d9b9d50 P 6810a866decb4f5089f7d35c870e4204 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6810a866decb4f5089f7d35c870e4204 (127.15.255.129). New cstate: current_term: 1 leader_uuid: "6810a866decb4f5089f7d35c870e4204" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:31.620983 25265 raft_consensus.cc:482] T 8f14db150fda449e962be2466f1d2a0e P 6810a866decb4f5089f7d35c870e4204 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:31.621162 25265 raft_consensus.cc:504] T 8f14db150fda449e962be2466f1d2a0e P 6810a866decb4f5089f7d35c870e4204 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } }
I20240502 21:27:31.621881 25265 leader_election.cc:288] T 8f14db150fda449e962be2466f1d2a0e P 6810a866decb4f5089f7d35c870e4204 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 038c5d9d1d7a49c981c4c34fbc00d9e6 (127.15.255.132:46813), 009d3e7e8a8b4e2497de3a73d01ced49 (127.15.255.131:37895)
I20240502 21:27:31.622501 24185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f14db150fda449e962be2466f1d2a0e" candidate_uuid: "6810a866decb4f5089f7d35c870e4204" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" is_pre_election: true
I20240502 21:27:31.622483 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f14db150fda449e962be2466f1d2a0e" candidate_uuid: "6810a866decb4f5089f7d35c870e4204" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" is_pre_election: true
I20240502 21:27:31.622730 24316 raft_consensus.cc:2454] T 8f14db150fda449e962be2466f1d2a0e P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6810a866decb4f5089f7d35c870e4204 in term 0.
I20240502 21:27:31.622720 24185 raft_consensus.cc:2454] T 8f14db150fda449e962be2466f1d2a0e P 009d3e7e8a8b4e2497de3a73d01ced49 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6810a866decb4f5089f7d35c870e4204 in term 0.
I20240502 21:27:31.623152 23850 leader_election.cc:302] T 8f14db150fda449e962be2466f1d2a0e P 6810a866decb4f5089f7d35c870e4204 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 009d3e7e8a8b4e2497de3a73d01ced49, 6810a866decb4f5089f7d35c870e4204; no voters:
I20240502 21:27:31.623437 25265 raft_consensus.cc:2781] T 8f14db150fda449e962be2466f1d2a0e P 6810a866decb4f5089f7d35c870e4204 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:31.623521 25265 raft_consensus.cc:482] T 8f14db150fda449e962be2466f1d2a0e P 6810a866decb4f5089f7d35c870e4204 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:31.623618 25265 raft_consensus.cc:3037] T 8f14db150fda449e962be2466f1d2a0e P 6810a866decb4f5089f7d35c870e4204 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:31.625398 25265 raft_consensus.cc:504] T 8f14db150fda449e962be2466f1d2a0e P 6810a866decb4f5089f7d35c870e4204 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } }
I20240502 21:27:31.625895 25265 leader_election.cc:288] T 8f14db150fda449e962be2466f1d2a0e P 6810a866decb4f5089f7d35c870e4204 [CANDIDATE]: Term 1 election: Requested vote from peers 038c5d9d1d7a49c981c4c34fbc00d9e6 (127.15.255.132:46813), 009d3e7e8a8b4e2497de3a73d01ced49 (127.15.255.131:37895)
I20240502 21:27:31.626276 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f14db150fda449e962be2466f1d2a0e" candidate_uuid: "6810a866decb4f5089f7d35c870e4204" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6"
I20240502 21:27:31.626298 24185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f14db150fda449e962be2466f1d2a0e" candidate_uuid: "6810a866decb4f5089f7d35c870e4204" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "009d3e7e8a8b4e2497de3a73d01ced49"
I20240502 21:27:31.626446 24185 raft_consensus.cc:3037] T 8f14db150fda449e962be2466f1d2a0e P 009d3e7e8a8b4e2497de3a73d01ced49 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:31.626451 24316 raft_consensus.cc:3037] T 8f14db150fda449e962be2466f1d2a0e P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:31.628377 24316 raft_consensus.cc:2454] T 8f14db150fda449e962be2466f1d2a0e P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6810a866decb4f5089f7d35c870e4204 in term 1.
I20240502 21:27:31.628584 24185 raft_consensus.cc:2454] T 8f14db150fda449e962be2466f1d2a0e P 009d3e7e8a8b4e2497de3a73d01ced49 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6810a866decb4f5089f7d35c870e4204 in term 1.
I20240502 21:27:31.628818 23849 leader_election.cc:302] T 8f14db150fda449e962be2466f1d2a0e P 6810a866decb4f5089f7d35c870e4204 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 038c5d9d1d7a49c981c4c34fbc00d9e6, 6810a866decb4f5089f7d35c870e4204; no voters:
I20240502 21:27:31.629088 25265 raft_consensus.cc:2781] T 8f14db150fda449e962be2466f1d2a0e P 6810a866decb4f5089f7d35c870e4204 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:31.629256 25265 raft_consensus.cc:686] T 8f14db150fda449e962be2466f1d2a0e P 6810a866decb4f5089f7d35c870e4204 [term 1 LEADER]: Becoming Leader. State: Replica: 6810a866decb4f5089f7d35c870e4204, State: Running, Role: LEADER
I20240502 21:27:31.629523 25265 consensus_queue.cc:237] T 8f14db150fda449e962be2466f1d2a0e P 6810a866decb4f5089f7d35c870e4204 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } }
I20240502 21:27:31.631968 23770 catalog_manager.cc:5484] T 8f14db150fda449e962be2466f1d2a0e P 6810a866decb4f5089f7d35c870e4204 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6810a866decb4f5089f7d35c870e4204 (127.15.255.129). New cstate: current_term: 1 leader_uuid: "6810a866decb4f5089f7d35c870e4204" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:31.803160 25265 consensus_queue.cc:1035] T 30236241b670443489240f191d9b9d50 P 6810a866decb4f5089f7d35c870e4204 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:31.810940 25265 consensus_queue.cc:1035] T 30236241b670443489240f191d9b9d50 P 6810a866decb4f5089f7d35c870e4204 [LEADER]: Connected to new peer: Peer: permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:31.846848 25246 consensus_queue.cc:1035] T 3cb3e62ca0e94e659efb3b7b0c2c9c91 P 6c194a6db6c145acafc0f3068b8ce1fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:31.851421 25168 consensus_queue.cc:1035] T 3cb3e62ca0e94e659efb3b7b0c2c9c91 P 6c194a6db6c145acafc0f3068b8ce1fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 21:27:32.039070 25303 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:27:32.039350 25303 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:27:32.046365 25303 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_4'
I20240502 21:27:32.188580 24185 raft_consensus.cc:1261] T 8f14db150fda449e962be2466f1d2a0e P 009d3e7e8a8b4e2497de3a73d01ced49 [term 1 FOLLOWER]: Refusing update from remote peer 6810a866decb4f5089f7d35c870e4204: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:27:32.188792 24316 raft_consensus.cc:1261] T 8f14db150fda449e962be2466f1d2a0e P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 1 FOLLOWER]: Refusing update from remote peer 6810a866decb4f5089f7d35c870e4204: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:27:32.189188 25265 consensus_queue.cc:1035] T 8f14db150fda449e962be2466f1d2a0e P 6810a866decb4f5089f7d35c870e4204 [LEADER]: Connected to new peer: Peer: permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:32.189627 25306 consensus_queue.cc:1035] T 8f14db150fda449e962be2466f1d2a0e P 6810a866decb4f5089f7d35c870e4204 [LEADER]: Connected to new peer: Peer: permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:32.223640 25339 mvcc.cc:205] Tried to move back new op lower bound from 7023350792958160896 to 7023350790676480000. Current Snapshot: MvccSnapshot[applied={T|T < 7023350792958160896}]
INSERT report
rows total: 2000
time total: 238.604 ms
time per row: 0.119302 ms
-----------------------------------------------------
Suppressions used:
count bytes template
12 3028 libcrypto.so.1.1
-----------------------------------------------------
I20240502 21:27:33.286594 23765 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41618:
name: "rebalance_test_table_5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 21:27:33.295404 24029 tablet_service.cc:1467] Processing CreateTablet for tablet 11cb9a2fbda946fb9f303f2c465bccb6 (DEFAULT_TABLE table=rebalance_test_table_5 [id=e10e5576c0ea45a09c0eab75c2023b78]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:33.295706 24164 tablet_service.cc:1467] Processing CreateTablet for tablet 11cb9a2fbda946fb9f303f2c465bccb6 (DEFAULT_TABLE table=rebalance_test_table_5 [id=e10e5576c0ea45a09c0eab75c2023b78]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:33.295861 24029 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11cb9a2fbda946fb9f303f2c465bccb6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:33.296109 24164 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11cb9a2fbda946fb9f303f2c465bccb6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:33.296154 24163 tablet_service.cc:1467] Processing CreateTablet for tablet 8af15fc41246471db31f7824985b5903 (DEFAULT_TABLE table=rebalance_test_table_5 [id=e10e5576c0ea45a09c0eab75c2023b78]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:33.296578 24163 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8af15fc41246471db31f7824985b5903. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:33.300211 24027 tablet_service.cc:1467] Processing CreateTablet for tablet 8af15fc41246471db31f7824985b5903 (DEFAULT_TABLE table=rebalance_test_table_5 [id=e10e5576c0ea45a09c0eab75c2023b78]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:33.300616 24027 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8af15fc41246471db31f7824985b5903. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:33.300393 23896 tablet_service.cc:1467] Processing CreateTablet for tablet 52f621ea3f174490948eae854c706a92 (DEFAULT_TABLE table=rebalance_test_table_5 [id=e10e5576c0ea45a09c0eab75c2023b78]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:33.300937 23896 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 52f621ea3f174490948eae854c706a92. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:33.301079 24296 tablet_service.cc:1467] Processing CreateTablet for tablet 11cb9a2fbda946fb9f303f2c465bccb6 (DEFAULT_TABLE table=rebalance_test_table_5 [id=e10e5576c0ea45a09c0eab75c2023b78]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:33.301468 24296 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11cb9a2fbda946fb9f303f2c465bccb6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:33.301635 24295 tablet_service.cc:1467] Processing CreateTablet for tablet 8af15fc41246471db31f7824985b5903 (DEFAULT_TABLE table=rebalance_test_table_5 [id=e10e5576c0ea45a09c0eab75c2023b78]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:33.301996 24295 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8af15fc41246471db31f7824985b5903. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:33.304754 24297 tablet_service.cc:1467] Processing CreateTablet for tablet 52f621ea3f174490948eae854c706a92 (DEFAULT_TABLE table=rebalance_test_table_5 [id=e10e5576c0ea45a09c0eab75c2023b78]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:33.304761 24028 tablet_service.cc:1467] Processing CreateTablet for tablet 52f621ea3f174490948eae854c706a92 (DEFAULT_TABLE table=rebalance_test_table_5 [id=e10e5576c0ea45a09c0eab75c2023b78]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:33.305083 24297 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 52f621ea3f174490948eae854c706a92. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:33.305085 24028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 52f621ea3f174490948eae854c706a92. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:33.314535 25372 tablet_bootstrap.cc:492] T 8af15fc41246471db31f7824985b5903 P 009d3e7e8a8b4e2497de3a73d01ced49: Bootstrap starting.
I20240502 21:27:33.316222 25372 tablet_bootstrap.cc:654] T 8af15fc41246471db31f7824985b5903 P 009d3e7e8a8b4e2497de3a73d01ced49: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:33.317589 25370 tablet_bootstrap.cc:492] T 8af15fc41246471db31f7824985b5903 P 038c5d9d1d7a49c981c4c34fbc00d9e6: Bootstrap starting.
I20240502 21:27:33.319193 25370 tablet_bootstrap.cc:654] T 8af15fc41246471db31f7824985b5903 P 038c5d9d1d7a49c981c4c34fbc00d9e6: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:33.319499 25369 tablet_bootstrap.cc:492] T 52f621ea3f174490948eae854c706a92 P 6810a866decb4f5089f7d35c870e4204: Bootstrap starting.
I20240502 21:27:33.319597 25371 tablet_bootstrap.cc:492] T 11cb9a2fbda946fb9f303f2c465bccb6 P 6c194a6db6c145acafc0f3068b8ce1fd: Bootstrap starting.
I20240502 21:27:33.320997 25372 tablet_bootstrap.cc:492] T 8af15fc41246471db31f7824985b5903 P 009d3e7e8a8b4e2497de3a73d01ced49: No bootstrap required, opened a new log
I20240502 21:27:33.321190 25372 ts_tablet_manager.cc:1367] T 8af15fc41246471db31f7824985b5903 P 009d3e7e8a8b4e2497de3a73d01ced49: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20240502 21:27:33.321303 25369 tablet_bootstrap.cc:654] T 52f621ea3f174490948eae854c706a92 P 6810a866decb4f5089f7d35c870e4204: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:33.321302 25371 tablet_bootstrap.cc:654] T 11cb9a2fbda946fb9f303f2c465bccb6 P 6c194a6db6c145acafc0f3068b8ce1fd: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:33.321988 25372 raft_consensus.cc:348] T 8af15fc41246471db31f7824985b5903 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } }
I20240502 21:27:33.322256 25372 raft_consensus.cc:374] T 8af15fc41246471db31f7824985b5903 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:33.322399 25372 raft_consensus.cc:729] T 8af15fc41246471db31f7824985b5903 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 009d3e7e8a8b4e2497de3a73d01ced49, State: Initialized, Role: FOLLOWER
I20240502 21:27:33.322695 25372 consensus_queue.cc:260] T 8af15fc41246471db31f7824985b5903 P 009d3e7e8a8b4e2497de3a73d01ced49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } }
I20240502 21:27:33.323171 25369 tablet_bootstrap.cc:492] T 52f621ea3f174490948eae854c706a92 P 6810a866decb4f5089f7d35c870e4204: No bootstrap required, opened a new log
I20240502 21:27:33.323171 25371 tablet_bootstrap.cc:492] T 11cb9a2fbda946fb9f303f2c465bccb6 P 6c194a6db6c145acafc0f3068b8ce1fd: No bootstrap required, opened a new log
I20240502 21:27:33.323309 25369 ts_tablet_manager.cc:1367] T 52f621ea3f174490948eae854c706a92 P 6810a866decb4f5089f7d35c870e4204: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240502 21:27:33.323320 25371 ts_tablet_manager.cc:1367] T 11cb9a2fbda946fb9f303f2c465bccb6 P 6c194a6db6c145acafc0f3068b8ce1fd: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20240502 21:27:33.323796 25372 ts_tablet_manager.cc:1398] T 8af15fc41246471db31f7824985b5903 P 009d3e7e8a8b4e2497de3a73d01ced49: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 21:27:33.323993 25371 raft_consensus.cc:348] T 11cb9a2fbda946fb9f303f2c465bccb6 P 6c194a6db6c145acafc0f3068b8ce1fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } }
I20240502 21:27:33.324038 25369 raft_consensus.cc:348] T 52f621ea3f174490948eae854c706a92 P 6810a866decb4f5089f7d35c870e4204 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } }
I20240502 21:27:33.324187 25371 raft_consensus.cc:374] T 11cb9a2fbda946fb9f303f2c465bccb6 P 6c194a6db6c145acafc0f3068b8ce1fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:33.324205 25369 raft_consensus.cc:374] T 52f621ea3f174490948eae854c706a92 P 6810a866decb4f5089f7d35c870e4204 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:33.324283 25369 raft_consensus.cc:729] T 52f621ea3f174490948eae854c706a92 P 6810a866decb4f5089f7d35c870e4204 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6810a866decb4f5089f7d35c870e4204, State: Initialized, Role: FOLLOWER
I20240502 21:27:33.324280 25371 raft_consensus.cc:729] T 11cb9a2fbda946fb9f303f2c465bccb6 P 6c194a6db6c145acafc0f3068b8ce1fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c194a6db6c145acafc0f3068b8ce1fd, State: Initialized, Role: FOLLOWER
I20240502 21:27:33.324510 25371 consensus_queue.cc:260] T 11cb9a2fbda946fb9f303f2c465bccb6 P 6c194a6db6c145acafc0f3068b8ce1fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } }
I20240502 21:27:33.324534 25369 consensus_queue.cc:260] T 52f621ea3f174490948eae854c706a92 P 6810a866decb4f5089f7d35c870e4204 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } }
I20240502 21:27:33.325035 25370 tablet_bootstrap.cc:492] T 8af15fc41246471db31f7824985b5903 P 038c5d9d1d7a49c981c4c34fbc00d9e6: No bootstrap required, opened a new log
I20240502 21:27:33.325034 25371 ts_tablet_manager.cc:1398] T 11cb9a2fbda946fb9f303f2c465bccb6 P 6c194a6db6c145acafc0f3068b8ce1fd: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:27:33.324649 25372 tablet_bootstrap.cc:492] T 11cb9a2fbda946fb9f303f2c465bccb6 P 009d3e7e8a8b4e2497de3a73d01ced49: Bootstrap starting.
I20240502 21:27:33.325305 25371 tablet_bootstrap.cc:492] T 52f621ea3f174490948eae854c706a92 P 6c194a6db6c145acafc0f3068b8ce1fd: Bootstrap starting.
I20240502 21:27:33.325315 25369 ts_tablet_manager.cc:1398] T 52f621ea3f174490948eae854c706a92 P 6810a866decb4f5089f7d35c870e4204: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:27:33.325507 25370 ts_tablet_manager.cc:1367] T 8af15fc41246471db31f7824985b5903 P 038c5d9d1d7a49c981c4c34fbc00d9e6: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.000s
I20240502 21:27:33.326205 25370 raft_consensus.cc:348] T 8af15fc41246471db31f7824985b5903 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } }
I20240502 21:27:33.326406 25370 raft_consensus.cc:374] T 8af15fc41246471db31f7824985b5903 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:33.326495 25370 raft_consensus.cc:729] T 8af15fc41246471db31f7824985b5903 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 038c5d9d1d7a49c981c4c34fbc00d9e6, State: Initialized, Role: FOLLOWER
I20240502 21:27:33.326535 25371 tablet_bootstrap.cc:654] T 52f621ea3f174490948eae854c706a92 P 6c194a6db6c145acafc0f3068b8ce1fd: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:33.326709 25370 consensus_queue.cc:260] T 8af15fc41246471db31f7824985b5903 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } }
I20240502 21:27:33.328941 25372 tablet_bootstrap.cc:654] T 11cb9a2fbda946fb9f303f2c465bccb6 P 009d3e7e8a8b4e2497de3a73d01ced49: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:33.331199 25370 ts_tablet_manager.cc:1398] T 8af15fc41246471db31f7824985b5903 P 038c5d9d1d7a49c981c4c34fbc00d9e6: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20240502 21:27:33.331542 25370 tablet_bootstrap.cc:492] T 52f621ea3f174490948eae854c706a92 P 038c5d9d1d7a49c981c4c34fbc00d9e6: Bootstrap starting.
I20240502 21:27:33.332062 25371 tablet_bootstrap.cc:492] T 52f621ea3f174490948eae854c706a92 P 6c194a6db6c145acafc0f3068b8ce1fd: No bootstrap required, opened a new log
I20240502 21:27:33.332218 25371 ts_tablet_manager.cc:1367] T 52f621ea3f174490948eae854c706a92 P 6c194a6db6c145acafc0f3068b8ce1fd: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.003s
I20240502 21:27:33.333206 25370 tablet_bootstrap.cc:654] T 52f621ea3f174490948eae854c706a92 P 038c5d9d1d7a49c981c4c34fbc00d9e6: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:33.333474 25372 tablet_bootstrap.cc:492] T 11cb9a2fbda946fb9f303f2c465bccb6 P 009d3e7e8a8b4e2497de3a73d01ced49: No bootstrap required, opened a new log
I20240502 21:27:33.333448 25371 raft_consensus.cc:348] T 52f621ea3f174490948eae854c706a92 P 6c194a6db6c145acafc0f3068b8ce1fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } }
I20240502 21:27:33.333619 25371 raft_consensus.cc:374] T 52f621ea3f174490948eae854c706a92 P 6c194a6db6c145acafc0f3068b8ce1fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:33.333690 25371 raft_consensus.cc:729] T 52f621ea3f174490948eae854c706a92 P 6c194a6db6c145acafc0f3068b8ce1fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c194a6db6c145acafc0f3068b8ce1fd, State: Initialized, Role: FOLLOWER
I20240502 21:27:33.333639 25372 ts_tablet_manager.cc:1367] T 11cb9a2fbda946fb9f303f2c465bccb6 P 009d3e7e8a8b4e2497de3a73d01ced49: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.000s
I20240502 21:27:33.334885 25371 consensus_queue.cc:260] T 52f621ea3f174490948eae854c706a92 P 6c194a6db6c145acafc0f3068b8ce1fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } }
I20240502 21:27:33.335139 25372 raft_consensus.cc:348] T 11cb9a2fbda946fb9f303f2c465bccb6 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } }
I20240502 21:27:33.335382 25371 ts_tablet_manager.cc:1398] T 52f621ea3f174490948eae854c706a92 P 6c194a6db6c145acafc0f3068b8ce1fd: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240502 21:27:33.335402 25372 raft_consensus.cc:374] T 11cb9a2fbda946fb9f303f2c465bccb6 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:33.335566 25372 raft_consensus.cc:729] T 11cb9a2fbda946fb9f303f2c465bccb6 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 009d3e7e8a8b4e2497de3a73d01ced49, State: Initialized, Role: FOLLOWER
I20240502 21:27:33.335676 25371 tablet_bootstrap.cc:492] T 8af15fc41246471db31f7824985b5903 P 6c194a6db6c145acafc0f3068b8ce1fd: Bootstrap starting.
I20240502 21:27:33.335870 25372 consensus_queue.cc:260] T 11cb9a2fbda946fb9f303f2c465bccb6 P 009d3e7e8a8b4e2497de3a73d01ced49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } }
I20240502 21:27:33.337213 25372 ts_tablet_manager.cc:1398] T 11cb9a2fbda946fb9f303f2c465bccb6 P 009d3e7e8a8b4e2497de3a73d01ced49: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 21:27:33.337291 25371 tablet_bootstrap.cc:654] T 8af15fc41246471db31f7824985b5903 P 6c194a6db6c145acafc0f3068b8ce1fd: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:33.339068 25371 tablet_bootstrap.cc:492] T 8af15fc41246471db31f7824985b5903 P 6c194a6db6c145acafc0f3068b8ce1fd: No bootstrap required, opened a new log
I20240502 21:27:33.339119 25370 tablet_bootstrap.cc:492] T 52f621ea3f174490948eae854c706a92 P 038c5d9d1d7a49c981c4c34fbc00d9e6: No bootstrap required, opened a new log
I20240502 21:27:33.339198 25371 ts_tablet_manager.cc:1367] T 8af15fc41246471db31f7824985b5903 P 6c194a6db6c145acafc0f3068b8ce1fd: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20240502 21:27:33.339246 25370 ts_tablet_manager.cc:1367] T 52f621ea3f174490948eae854c706a92 P 038c5d9d1d7a49c981c4c34fbc00d9e6: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.004s
I20240502 21:27:33.339843 25371 raft_consensus.cc:348] T 8af15fc41246471db31f7824985b5903 P 6c194a6db6c145acafc0f3068b8ce1fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } }
I20240502 21:27:33.339859 25370 raft_consensus.cc:348] T 52f621ea3f174490948eae854c706a92 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } }
I20240502 21:27:33.340019 25371 raft_consensus.cc:374] T 8af15fc41246471db31f7824985b5903 P 6c194a6db6c145acafc0f3068b8ce1fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:33.340047 25370 raft_consensus.cc:374] T 52f621ea3f174490948eae854c706a92 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:33.340099 25371 raft_consensus.cc:729] T 8af15fc41246471db31f7824985b5903 P 6c194a6db6c145acafc0f3068b8ce1fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c194a6db6c145acafc0f3068b8ce1fd, State: Initialized, Role: FOLLOWER
I20240502 21:27:33.340113 25370 raft_consensus.cc:729] T 52f621ea3f174490948eae854c706a92 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 038c5d9d1d7a49c981c4c34fbc00d9e6, State: Initialized, Role: FOLLOWER
I20240502 21:27:33.340306 25371 consensus_queue.cc:260] T 8af15fc41246471db31f7824985b5903 P 6c194a6db6c145acafc0f3068b8ce1fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } }
I20240502 21:27:33.340325 25370 consensus_queue.cc:260] T 52f621ea3f174490948eae854c706a92 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } }
I20240502 21:27:33.340818 25370 ts_tablet_manager.cc:1398] T 52f621ea3f174490948eae854c706a92 P 038c5d9d1d7a49c981c4c34fbc00d9e6: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20240502 21:27:33.340881 25371 ts_tablet_manager.cc:1398] T 8af15fc41246471db31f7824985b5903 P 6c194a6db6c145acafc0f3068b8ce1fd: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:27:33.341140 25370 tablet_bootstrap.cc:492] T 11cb9a2fbda946fb9f303f2c465bccb6 P 038c5d9d1d7a49c981c4c34fbc00d9e6: Bootstrap starting.
I20240502 21:27:33.342312 25370 tablet_bootstrap.cc:654] T 11cb9a2fbda946fb9f303f2c465bccb6 P 038c5d9d1d7a49c981c4c34fbc00d9e6: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:33.343959 25370 tablet_bootstrap.cc:492] T 11cb9a2fbda946fb9f303f2c465bccb6 P 038c5d9d1d7a49c981c4c34fbc00d9e6: No bootstrap required, opened a new log
I20240502 21:27:33.344077 25370 ts_tablet_manager.cc:1367] T 11cb9a2fbda946fb9f303f2c465bccb6 P 038c5d9d1d7a49c981c4c34fbc00d9e6: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240502 21:27:33.344695 25370 raft_consensus.cc:348] T 11cb9a2fbda946fb9f303f2c465bccb6 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } }
I20240502 21:27:33.344830 25370 raft_consensus.cc:374] T 11cb9a2fbda946fb9f303f2c465bccb6 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:33.344913 25370 raft_consensus.cc:729] T 11cb9a2fbda946fb9f303f2c465bccb6 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 038c5d9d1d7a49c981c4c34fbc00d9e6, State: Initialized, Role: FOLLOWER
I20240502 21:27:33.345116 25370 consensus_queue.cc:260] T 11cb9a2fbda946fb9f303f2c465bccb6 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } }
I20240502 21:27:33.345538 25370 ts_tablet_manager.cc:1398] T 11cb9a2fbda946fb9f303f2c465bccb6 P 038c5d9d1d7a49c981c4c34fbc00d9e6: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240502 21:27:33.408916 25355 raft_consensus.cc:482] T 8af15fc41246471db31f7824985b5903 P 6c194a6db6c145acafc0f3068b8ce1fd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:33.409071 25355 raft_consensus.cc:504] T 8af15fc41246471db31f7824985b5903 P 6c194a6db6c145acafc0f3068b8ce1fd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } }
I20240502 21:27:33.409731 25355 leader_election.cc:288] T 8af15fc41246471db31f7824985b5903 P 6c194a6db6c145acafc0f3068b8ce1fd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 009d3e7e8a8b4e2497de3a73d01ced49 (127.15.255.131:37895), 038c5d9d1d7a49c981c4c34fbc00d9e6 (127.15.255.132:46813)
I20240502 21:27:33.409713 25377 raft_consensus.cc:482] T 8af15fc41246471db31f7824985b5903 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:33.409866 25377 raft_consensus.cc:504] T 8af15fc41246471db31f7824985b5903 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } }
I20240502 21:27:33.410038 24185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8af15fc41246471db31f7824985b5903" candidate_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" is_pre_election: true
I20240502 21:27:33.410235 24185 raft_consensus.cc:2454] T 8af15fc41246471db31f7824985b5903 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6c194a6db6c145acafc0f3068b8ce1fd in term 0.
I20240502 21:27:33.410181 24315 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8af15fc41246471db31f7824985b5903" candidate_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" is_pre_election: true
I20240502 21:27:33.410527 24315 raft_consensus.cc:2454] T 8af15fc41246471db31f7824985b5903 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6c194a6db6c145acafc0f3068b8ce1fd in term 0.
I20240502 21:27:33.410631 23982 leader_election.cc:302] T 8af15fc41246471db31f7824985b5903 P 6c194a6db6c145acafc0f3068b8ce1fd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 009d3e7e8a8b4e2497de3a73d01ced49, 6c194a6db6c145acafc0f3068b8ce1fd; no voters:
I20240502 21:27:33.410892 25355 raft_consensus.cc:2781] T 8af15fc41246471db31f7824985b5903 P 6c194a6db6c145acafc0f3068b8ce1fd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:33.411022 25355 raft_consensus.cc:482] T 8af15fc41246471db31f7824985b5903 P 6c194a6db6c145acafc0f3068b8ce1fd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:33.411115 25355 raft_consensus.cc:3037] T 8af15fc41246471db31f7824985b5903 P 6c194a6db6c145acafc0f3068b8ce1fd [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:33.411636 25377 leader_election.cc:288] T 8af15fc41246471db31f7824985b5903 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 009d3e7e8a8b4e2497de3a73d01ced49 (127.15.255.131:37895), 6c194a6db6c145acafc0f3068b8ce1fd (127.15.255.130:41235)
I20240502 21:27:33.413766 25355 raft_consensus.cc:504] T 8af15fc41246471db31f7824985b5903 P 6c194a6db6c145acafc0f3068b8ce1fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } }
I20240502 21:27:33.414525 25355 leader_election.cc:288] T 8af15fc41246471db31f7824985b5903 P 6c194a6db6c145acafc0f3068b8ce1fd [CANDIDATE]: Term 1 election: Requested vote from peers 009d3e7e8a8b4e2497de3a73d01ced49 (127.15.255.131:37895), 038c5d9d1d7a49c981c4c34fbc00d9e6 (127.15.255.132:46813)
I20240502 21:27:33.415968 24315 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8af15fc41246471db31f7824985b5903" candidate_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6"
I20240502 21:27:33.416185 24315 raft_consensus.cc:3037] T 8af15fc41246471db31f7824985b5903 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:33.416697 24185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8af15fc41246471db31f7824985b5903" candidate_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "009d3e7e8a8b4e2497de3a73d01ced49"
I20240502 21:27:33.416884 24185 raft_consensus.cc:3037] T 8af15fc41246471db31f7824985b5903 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:33.419005 24315 raft_consensus.cc:2454] T 8af15fc41246471db31f7824985b5903 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c194a6db6c145acafc0f3068b8ce1fd in term 1.
I20240502 21:27:33.419462 24185 raft_consensus.cc:2454] T 8af15fc41246471db31f7824985b5903 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c194a6db6c145acafc0f3068b8ce1fd in term 1.
I20240502 21:27:33.419869 23981 leader_election.cc:302] T 8af15fc41246471db31f7824985b5903 P 6c194a6db6c145acafc0f3068b8ce1fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 038c5d9d1d7a49c981c4c34fbc00d9e6, 6c194a6db6c145acafc0f3068b8ce1fd; no voters:
I20240502 21:27:33.420161 25355 raft_consensus.cc:2781] T 8af15fc41246471db31f7824985b5903 P 6c194a6db6c145acafc0f3068b8ce1fd [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:33.420688 25355 raft_consensus.cc:686] T 8af15fc41246471db31f7824985b5903 P 6c194a6db6c145acafc0f3068b8ce1fd [term 1 LEADER]: Becoming Leader. State: Replica: 6c194a6db6c145acafc0f3068b8ce1fd, State: Running, Role: LEADER
I20240502 21:27:33.420987 25355 consensus_queue.cc:237] T 8af15fc41246471db31f7824985b5903 P 6c194a6db6c145acafc0f3068b8ce1fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } }
I20240502 21:27:33.423040 24049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8af15fc41246471db31f7824985b5903" candidate_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" is_pre_election: true
I20240502 21:27:33.424021 24185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8af15fc41246471db31f7824985b5903" candidate_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" is_pre_election: true
I20240502 21:27:33.424237 24185 raft_consensus.cc:2379] T 8af15fc41246471db31f7824985b5903 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 038c5d9d1d7a49c981c4c34fbc00d9e6 in current term 1: Already voted for candidate 6c194a6db6c145acafc0f3068b8ce1fd in this term.
I20240502 21:27:33.424643 24252 leader_election.cc:302] T 8af15fc41246471db31f7824985b5903 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 038c5d9d1d7a49c981c4c34fbc00d9e6; no voters: 009d3e7e8a8b4e2497de3a73d01ced49, 6c194a6db6c145acafc0f3068b8ce1fd
I20240502 21:27:33.425006 25377 raft_consensus.cc:2726] T 8af15fc41246471db31f7824985b5903 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 21:27:33.432072 23772 catalog_manager.cc:5484] T 8af15fc41246471db31f7824985b5903 P 6c194a6db6c145acafc0f3068b8ce1fd reported cstate change: term changed from 0 to 1, leader changed from <none> to 6c194a6db6c145acafc0f3068b8ce1fd (127.15.255.130). New cstate: current_term: 1 leader_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:33.470496 25377 raft_consensus.cc:482] T 52f621ea3f174490948eae854c706a92 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:33.470659 25377 raft_consensus.cc:504] T 52f621ea3f174490948eae854c706a92 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } }
I20240502 21:27:33.471259 25377 leader_election.cc:288] T 52f621ea3f174490948eae854c706a92 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6810a866decb4f5089f7d35c870e4204 (127.15.255.129:40435), 6c194a6db6c145acafc0f3068b8ce1fd (127.15.255.130:41235)
I20240502 21:27:33.472069 24049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52f621ea3f174490948eae854c706a92" candidate_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" is_pre_election: true
I20240502 21:27:33.472268 24049 raft_consensus.cc:2454] T 52f621ea3f174490948eae854c706a92 P 6c194a6db6c145acafc0f3068b8ce1fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 038c5d9d1d7a49c981c4c34fbc00d9e6 in term 0.
I20240502 21:27:33.472704 24253 leader_election.cc:302] T 52f621ea3f174490948eae854c706a92 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 038c5d9d1d7a49c981c4c34fbc00d9e6, 6c194a6db6c145acafc0f3068b8ce1fd; no voters:
I20240502 21:27:33.472937 25377 raft_consensus.cc:2781] T 52f621ea3f174490948eae854c706a92 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:33.473040 25377 raft_consensus.cc:482] T 52f621ea3f174490948eae854c706a92 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:33.473124 25377 raft_consensus.cc:3037] T 52f621ea3f174490948eae854c706a92 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:33.475621 25377 raft_consensus.cc:504] T 52f621ea3f174490948eae854c706a92 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } }
I20240502 21:27:33.476279 25377 leader_election.cc:288] T 52f621ea3f174490948eae854c706a92 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [CANDIDATE]: Term 1 election: Requested vote from peers 6810a866decb4f5089f7d35c870e4204 (127.15.255.129:40435), 6c194a6db6c145acafc0f3068b8ce1fd (127.15.255.130:41235)
I20240502 21:27:33.476788 24049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52f621ea3f174490948eae854c706a92" candidate_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c194a6db6c145acafc0f3068b8ce1fd"
I20240502 21:27:33.476946 24049 raft_consensus.cc:3037] T 52f621ea3f174490948eae854c706a92 P 6c194a6db6c145acafc0f3068b8ce1fd [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:33.477396 23915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52f621ea3f174490948eae854c706a92" candidate_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6810a866decb4f5089f7d35c870e4204" is_pre_election: true
I20240502 21:27:33.477521 23916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52f621ea3f174490948eae854c706a92" candidate_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6810a866decb4f5089f7d35c870e4204"
I20240502 21:27:33.477640 23915 raft_consensus.cc:2454] T 52f621ea3f174490948eae854c706a92 P 6810a866decb4f5089f7d35c870e4204 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 038c5d9d1d7a49c981c4c34fbc00d9e6 in term 0.
I20240502 21:27:33.479023 24049 raft_consensus.cc:2454] T 52f621ea3f174490948eae854c706a92 P 6c194a6db6c145acafc0f3068b8ce1fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 038c5d9d1d7a49c981c4c34fbc00d9e6 in term 1.
I20240502 21:27:33.479328 24253 leader_election.cc:302] T 52f621ea3f174490948eae854c706a92 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 038c5d9d1d7a49c981c4c34fbc00d9e6, 6c194a6db6c145acafc0f3068b8ce1fd; no voters: 6810a866decb4f5089f7d35c870e4204
I20240502 21:27:33.479611 25377 raft_consensus.cc:2781] T 52f621ea3f174490948eae854c706a92 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:33.479970 25377 raft_consensus.cc:686] T 52f621ea3f174490948eae854c706a92 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 1 LEADER]: Becoming Leader. State: Replica: 038c5d9d1d7a49c981c4c34fbc00d9e6, State: Running, Role: LEADER
I20240502 21:27:33.480377 25377 consensus_queue.cc:237] T 52f621ea3f174490948eae854c706a92 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } }
I20240502 21:27:33.501894 25377 raft_consensus.cc:482] T 11cb9a2fbda946fb9f303f2c465bccb6 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:33.502059 25377 raft_consensus.cc:504] T 11cb9a2fbda946fb9f303f2c465bccb6 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } }
I20240502 21:27:33.502758 25377 leader_election.cc:288] T 11cb9a2fbda946fb9f303f2c465bccb6 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6c194a6db6c145acafc0f3068b8ce1fd (127.15.255.130:41235), 009d3e7e8a8b4e2497de3a73d01ced49 (127.15.255.131:37895)
I20240502 21:27:33.502988 24049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11cb9a2fbda946fb9f303f2c465bccb6" candidate_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" is_pre_election: true
I20240502 21:27:33.503129 24185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11cb9a2fbda946fb9f303f2c465bccb6" candidate_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" is_pre_election: true
I20240502 21:27:33.503223 24049 raft_consensus.cc:2454] T 11cb9a2fbda946fb9f303f2c465bccb6 P 6c194a6db6c145acafc0f3068b8ce1fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 038c5d9d1d7a49c981c4c34fbc00d9e6 in term 0.
I20240502 21:27:33.503399 24185 raft_consensus.cc:2454] T 11cb9a2fbda946fb9f303f2c465bccb6 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 038c5d9d1d7a49c981c4c34fbc00d9e6 in term 0.
I20240502 21:27:33.503588 24253 leader_election.cc:302] T 11cb9a2fbda946fb9f303f2c465bccb6 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 038c5d9d1d7a49c981c4c34fbc00d9e6, 6c194a6db6c145acafc0f3068b8ce1fd; no voters:
I20240502 21:27:33.503875 25377 raft_consensus.cc:2781] T 11cb9a2fbda946fb9f303f2c465bccb6 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:33.503971 25377 raft_consensus.cc:482] T 11cb9a2fbda946fb9f303f2c465bccb6 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:33.504081 25377 raft_consensus.cc:3037] T 11cb9a2fbda946fb9f303f2c465bccb6 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:33.504256 23772 catalog_manager.cc:5484] T 52f621ea3f174490948eae854c706a92 P 038c5d9d1d7a49c981c4c34fbc00d9e6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 038c5d9d1d7a49c981c4c34fbc00d9e6 (127.15.255.132). New cstate: current_term: 1 leader_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } health_report { overall_health: HEALTHY } } }
I20240502 21:27:33.506274 25377 raft_consensus.cc:504] T 11cb9a2fbda946fb9f303f2c465bccb6 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } }
I20240502 21:27:33.506982 25377 leader_election.cc:288] T 11cb9a2fbda946fb9f303f2c465bccb6 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [CANDIDATE]: Term 1 election: Requested vote from peers 6c194a6db6c145acafc0f3068b8ce1fd (127.15.255.130:41235), 009d3e7e8a8b4e2497de3a73d01ced49 (127.15.255.131:37895)
I20240502 21:27:33.507185 24049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11cb9a2fbda946fb9f303f2c465bccb6" candidate_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c194a6db6c145acafc0f3068b8ce1fd"
I20240502 21:27:33.507220 24185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11cb9a2fbda946fb9f303f2c465bccb6" candidate_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "009d3e7e8a8b4e2497de3a73d01ced49"
I20240502 21:27:33.507354 24049 raft_consensus.cc:3037] T 11cb9a2fbda946fb9f303f2c465bccb6 P 6c194a6db6c145acafc0f3068b8ce1fd [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:33.507380 24185 raft_consensus.cc:3037] T 11cb9a2fbda946fb9f303f2c465bccb6 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:33.509195 24049 raft_consensus.cc:2454] T 11cb9a2fbda946fb9f303f2c465bccb6 P 6c194a6db6c145acafc0f3068b8ce1fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 038c5d9d1d7a49c981c4c34fbc00d9e6 in term 1.
I20240502 21:27:33.509207 24185 raft_consensus.cc:2454] T 11cb9a2fbda946fb9f303f2c465bccb6 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 038c5d9d1d7a49c981c4c34fbc00d9e6 in term 1.
I20240502 21:27:33.509534 24253 leader_election.cc:302] T 11cb9a2fbda946fb9f303f2c465bccb6 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 038c5d9d1d7a49c981c4c34fbc00d9e6, 6c194a6db6c145acafc0f3068b8ce1fd; no voters:
I20240502 21:27:33.509776 25377 raft_consensus.cc:2781] T 11cb9a2fbda946fb9f303f2c465bccb6 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:33.509958 25377 raft_consensus.cc:686] T 11cb9a2fbda946fb9f303f2c465bccb6 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 1 LEADER]: Becoming Leader. State: Replica: 038c5d9d1d7a49c981c4c34fbc00d9e6, State: Running, Role: LEADER
I20240502 21:27:33.510169 25377 consensus_queue.cc:237] T 11cb9a2fbda946fb9f303f2c465bccb6 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } }
I20240502 21:27:33.512626 23772 catalog_manager.cc:5484] T 11cb9a2fbda946fb9f303f2c465bccb6 P 038c5d9d1d7a49c981c4c34fbc00d9e6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 038c5d9d1d7a49c981c4c34fbc00d9e6 (127.15.255.132). New cstate: current_term: 1 leader_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } health_report { overall_health: HEALTHY } } }
W20240502 21:27:33.870338 25393 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:27:33.870627 25393 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:27:33.877987 25393 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_5'
I20240502 21:27:33.917511 25377 consensus_queue.cc:1035] T 11cb9a2fbda946fb9f303f2c465bccb6 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:33.924487 25377 consensus_queue.cc:1035] T 11cb9a2fbda946fb9f303f2c465bccb6 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:33.934893 23915 raft_consensus.cc:3037] T 52f621ea3f174490948eae854c706a92 P 6810a866decb4f5089f7d35c870e4204 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:33.938205 25377 consensus_queue.cc:1035] T 52f621ea3f174490948eae854c706a92 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:33.943535 25390 consensus_queue.cc:1035] T 52f621ea3f174490948eae854c706a92 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:33.973693 25424 consensus_queue.cc:1035] T 8af15fc41246471db31f7824985b5903 P 6c194a6db6c145acafc0f3068b8ce1fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:33.983963 25355 consensus_queue.cc:1035] T 8af15fc41246471db31f7824985b5903 P 6c194a6db6c145acafc0f3068b8ce1fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 21:27:34.116782 24116 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.132:46813, user_credentials={real_user=slave}} blocked reactor thread for 65763.5us
INSERT report
rows total: 2000
time total: 286.261 ms
time per row: 0.14313 ms
I20240502 21:27:34.311849 23767 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41618:
name: "rebalance_test_table_6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 21:27:34.320650 24163 tablet_service.cc:1467] Processing CreateTablet for tablet 256ae04a48b849c4a9086b6b1ff5df19 (DEFAULT_TABLE table=rebalance_test_table_6 [id=9aaf56e5a31f42a79b45624cc2c43779]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:34.320644 23896 tablet_service.cc:1467] Processing CreateTablet for tablet 256ae04a48b849c4a9086b6b1ff5df19 (DEFAULT_TABLE table=rebalance_test_table_6 [id=9aaf56e5a31f42a79b45624cc2c43779]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:34.321130 24163 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 256ae04a48b849c4a9086b6b1ff5df19. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:34.321125 23896 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 256ae04a48b849c4a9086b6b1ff5df19. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:34.321086 24164 tablet_service.cc:1467] Processing CreateTablet for tablet fe044374ad7b4999afc72ab812466749 (DEFAULT_TABLE table=rebalance_test_table_6 [id=9aaf56e5a31f42a79b45624cc2c43779]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:34.321513 24164 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe044374ad7b4999afc72ab812466749. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:34.322300 24029 tablet_service.cc:1467] Processing CreateTablet for tablet 256ae04a48b849c4a9086b6b1ff5df19 (DEFAULT_TABLE table=rebalance_test_table_6 [id=9aaf56e5a31f42a79b45624cc2c43779]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:34.322715 24029 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 256ae04a48b849c4a9086b6b1ff5df19. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:34.325088 24297 tablet_service.cc:1467] Processing CreateTablet for tablet fe044374ad7b4999afc72ab812466749 (DEFAULT_TABLE table=rebalance_test_table_6 [id=9aaf56e5a31f42a79b45624cc2c43779]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:34.325321 24162 tablet_service.cc:1467] Processing CreateTablet for tablet a2a0537a60f849daae2061c2b0c4c89f (DEFAULT_TABLE table=rebalance_test_table_6 [id=9aaf56e5a31f42a79b45624cc2c43779]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:34.325474 24297 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe044374ad7b4999afc72ab812466749. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:34.325659 24162 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a2a0537a60f849daae2061c2b0c4c89f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:34.329557 23894 tablet_service.cc:1467] Processing CreateTablet for tablet fe044374ad7b4999afc72ab812466749 (DEFAULT_TABLE table=rebalance_test_table_6 [id=9aaf56e5a31f42a79b45624cc2c43779]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:34.329941 23894 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe044374ad7b4999afc72ab812466749. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:34.332808 23895 tablet_service.cc:1467] Processing CreateTablet for tablet a2a0537a60f849daae2061c2b0c4c89f (DEFAULT_TABLE table=rebalance_test_table_6 [id=9aaf56e5a31f42a79b45624cc2c43779]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:34.333185 23895 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a2a0537a60f849daae2061c2b0c4c89f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:34.336185 25460 tablet_bootstrap.cc:492] T 256ae04a48b849c4a9086b6b1ff5df19 P 009d3e7e8a8b4e2497de3a73d01ced49: Bootstrap starting.
I20240502 21:27:34.337252 24027 tablet_service.cc:1467] Processing CreateTablet for tablet a2a0537a60f849daae2061c2b0c4c89f (DEFAULT_TABLE table=rebalance_test_table_6 [id=9aaf56e5a31f42a79b45624cc2c43779]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:34.337682 24027 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a2a0537a60f849daae2061c2b0c4c89f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:34.339577 25459 tablet_bootstrap.cc:492] T fe044374ad7b4999afc72ab812466749 P 038c5d9d1d7a49c981c4c34fbc00d9e6: Bootstrap starting.
I20240502 21:27:34.339593 25460 tablet_bootstrap.cc:654] T 256ae04a48b849c4a9086b6b1ff5df19 P 009d3e7e8a8b4e2497de3a73d01ced49: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:34.339617 25458 tablet_bootstrap.cc:492] T 256ae04a48b849c4a9086b6b1ff5df19 P 6c194a6db6c145acafc0f3068b8ce1fd: Bootstrap starting.
I20240502 21:27:34.341341 25458 tablet_bootstrap.cc:654] T 256ae04a48b849c4a9086b6b1ff5df19 P 6c194a6db6c145acafc0f3068b8ce1fd: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:34.341353 25459 tablet_bootstrap.cc:654] T fe044374ad7b4999afc72ab812466749 P 038c5d9d1d7a49c981c4c34fbc00d9e6: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:34.342901 25457 tablet_bootstrap.cc:492] T 256ae04a48b849c4a9086b6b1ff5df19 P 6810a866decb4f5089f7d35c870e4204: Bootstrap starting.
I20240502 21:27:34.344763 25457 tablet_bootstrap.cc:654] T 256ae04a48b849c4a9086b6b1ff5df19 P 6810a866decb4f5089f7d35c870e4204: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:34.346896 25459 tablet_bootstrap.cc:492] T fe044374ad7b4999afc72ab812466749 P 038c5d9d1d7a49c981c4c34fbc00d9e6: No bootstrap required, opened a new log
I20240502 21:27:34.347100 25459 ts_tablet_manager.cc:1367] T fe044374ad7b4999afc72ab812466749 P 038c5d9d1d7a49c981c4c34fbc00d9e6: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20240502 21:27:34.347973 25459 raft_consensus.cc:348] T fe044374ad7b4999afc72ab812466749 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } }
I20240502 21:27:34.348317 25459 raft_consensus.cc:374] T fe044374ad7b4999afc72ab812466749 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:34.348474 25459 raft_consensus.cc:729] T fe044374ad7b4999afc72ab812466749 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 038c5d9d1d7a49c981c4c34fbc00d9e6, State: Initialized, Role: FOLLOWER
I20240502 21:27:34.348812 25459 consensus_queue.cc:260] T fe044374ad7b4999afc72ab812466749 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } }
I20240502 21:27:34.349490 25459 ts_tablet_manager.cc:1398] T fe044374ad7b4999afc72ab812466749 P 038c5d9d1d7a49c981c4c34fbc00d9e6: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:27:34.350865 25457 tablet_bootstrap.cc:492] T 256ae04a48b849c4a9086b6b1ff5df19 P 6810a866decb4f5089f7d35c870e4204: No bootstrap required, opened a new log
I20240502 21:27:34.351073 25457 ts_tablet_manager.cc:1367] T 256ae04a48b849c4a9086b6b1ff5df19 P 6810a866decb4f5089f7d35c870e4204: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20240502 21:27:34.352806 25457 raft_consensus.cc:348] T 256ae04a48b849c4a9086b6b1ff5df19 P 6810a866decb4f5089f7d35c870e4204 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } }
I20240502 21:27:34.353080 25457 raft_consensus.cc:374] T 256ae04a48b849c4a9086b6b1ff5df19 P 6810a866decb4f5089f7d35c870e4204 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:34.353211 25457 raft_consensus.cc:729] T 256ae04a48b849c4a9086b6b1ff5df19 P 6810a866decb4f5089f7d35c870e4204 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6810a866decb4f5089f7d35c870e4204, State: Initialized, Role: FOLLOWER
I20240502 21:27:34.353524 25457 consensus_queue.cc:260] T 256ae04a48b849c4a9086b6b1ff5df19 P 6810a866decb4f5089f7d35c870e4204 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } }
I20240502 21:27:34.354250 25457 ts_tablet_manager.cc:1398] T 256ae04a48b849c4a9086b6b1ff5df19 P 6810a866decb4f5089f7d35c870e4204: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 21:27:34.355681 25458 tablet_bootstrap.cc:492] T 256ae04a48b849c4a9086b6b1ff5df19 P 6c194a6db6c145acafc0f3068b8ce1fd: No bootstrap required, opened a new log
I20240502 21:27:34.355901 25458 ts_tablet_manager.cc:1367] T 256ae04a48b849c4a9086b6b1ff5df19 P 6c194a6db6c145acafc0f3068b8ce1fd: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.012s
I20240502 21:27:34.356812 25458 raft_consensus.cc:348] T 256ae04a48b849c4a9086b6b1ff5df19 P 6c194a6db6c145acafc0f3068b8ce1fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } }
I20240502 21:27:34.357082 25458 raft_consensus.cc:374] T 256ae04a48b849c4a9086b6b1ff5df19 P 6c194a6db6c145acafc0f3068b8ce1fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:34.357214 25458 raft_consensus.cc:729] T 256ae04a48b849c4a9086b6b1ff5df19 P 6c194a6db6c145acafc0f3068b8ce1fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c194a6db6c145acafc0f3068b8ce1fd, State: Initialized, Role: FOLLOWER
I20240502 21:27:34.357611 25458 consensus_queue.cc:260] T 256ae04a48b849c4a9086b6b1ff5df19 P 6c194a6db6c145acafc0f3068b8ce1fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } }
I20240502 21:27:34.358234 25458 ts_tablet_manager.cc:1398] T 256ae04a48b849c4a9086b6b1ff5df19 P 6c194a6db6c145acafc0f3068b8ce1fd: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:27:34.364450 25457 tablet_bootstrap.cc:492] T a2a0537a60f849daae2061c2b0c4c89f P 6810a866decb4f5089f7d35c870e4204: Bootstrap starting.
I20240502 21:27:34.366125 25457 tablet_bootstrap.cc:654] T a2a0537a60f849daae2061c2b0c4c89f P 6810a866decb4f5089f7d35c870e4204: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:34.366353 25458 tablet_bootstrap.cc:492] T a2a0537a60f849daae2061c2b0c4c89f P 6c194a6db6c145acafc0f3068b8ce1fd: Bootstrap starting.
I20240502 21:27:34.368095 25458 tablet_bootstrap.cc:654] T a2a0537a60f849daae2061c2b0c4c89f P 6c194a6db6c145acafc0f3068b8ce1fd: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:34.368216 25460 tablet_bootstrap.cc:492] T 256ae04a48b849c4a9086b6b1ff5df19 P 009d3e7e8a8b4e2497de3a73d01ced49: No bootstrap required, opened a new log
I20240502 21:27:34.368388 25460 ts_tablet_manager.cc:1367] T 256ae04a48b849c4a9086b6b1ff5df19 P 009d3e7e8a8b4e2497de3a73d01ced49: Time spent bootstrapping tablet: real 0.032s user 0.014s sys 0.011s
I20240502 21:27:34.368690 25457 tablet_bootstrap.cc:492] T a2a0537a60f849daae2061c2b0c4c89f P 6810a866decb4f5089f7d35c870e4204: No bootstrap required, opened a new log
I20240502 21:27:34.368841 25457 ts_tablet_manager.cc:1367] T a2a0537a60f849daae2061c2b0c4c89f P 6810a866decb4f5089f7d35c870e4204: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20240502 21:27:34.369207 25460 raft_consensus.cc:348] T 256ae04a48b849c4a9086b6b1ff5df19 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } }
I20240502 21:27:34.369421 25460 raft_consensus.cc:374] T 256ae04a48b849c4a9086b6b1ff5df19 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:34.369520 25460 raft_consensus.cc:729] T 256ae04a48b849c4a9086b6b1ff5df19 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 009d3e7e8a8b4e2497de3a73d01ced49, State: Initialized, Role: FOLLOWER
I20240502 21:27:34.369586 25457 raft_consensus.cc:348] T a2a0537a60f849daae2061c2b0c4c89f P 6810a866decb4f5089f7d35c870e4204 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } }
I20240502 21:27:34.369783 25457 raft_consensus.cc:374] T a2a0537a60f849daae2061c2b0c4c89f P 6810a866decb4f5089f7d35c870e4204 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:34.369859 25457 raft_consensus.cc:729] T a2a0537a60f849daae2061c2b0c4c89f P 6810a866decb4f5089f7d35c870e4204 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6810a866decb4f5089f7d35c870e4204, State: Initialized, Role: FOLLOWER
I20240502 21:27:34.369788 25460 consensus_queue.cc:260] T 256ae04a48b849c4a9086b6b1ff5df19 P 009d3e7e8a8b4e2497de3a73d01ced49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } }
I20240502 21:27:34.370432 25460 ts_tablet_manager.cc:1398] T 256ae04a48b849c4a9086b6b1ff5df19 P 009d3e7e8a8b4e2497de3a73d01ced49: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 21:27:34.370779 25460 tablet_bootstrap.cc:492] T fe044374ad7b4999afc72ab812466749 P 009d3e7e8a8b4e2497de3a73d01ced49: Bootstrap starting.
I20240502 21:27:34.370095 25457 consensus_queue.cc:260] T a2a0537a60f849daae2061c2b0c4c89f P 6810a866decb4f5089f7d35c870e4204 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } }
I20240502 21:27:34.372014 25458 tablet_bootstrap.cc:492] T a2a0537a60f849daae2061c2b0c4c89f P 6c194a6db6c145acafc0f3068b8ce1fd: No bootstrap required, opened a new log
I20240502 21:27:34.372201 25457 ts_tablet_manager.cc:1398] T a2a0537a60f849daae2061c2b0c4c89f P 6810a866decb4f5089f7d35c870e4204: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 21:27:34.372221 25458 ts_tablet_manager.cc:1367] T a2a0537a60f849daae2061c2b0c4c89f P 6c194a6db6c145acafc0f3068b8ce1fd: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.003s
I20240502 21:27:34.372467 25460 tablet_bootstrap.cc:654] T fe044374ad7b4999afc72ab812466749 P 009d3e7e8a8b4e2497de3a73d01ced49: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:34.372797 25457 tablet_bootstrap.cc:492] T fe044374ad7b4999afc72ab812466749 P 6810a866decb4f5089f7d35c870e4204: Bootstrap starting.
I20240502 21:27:34.373003 25458 raft_consensus.cc:348] T a2a0537a60f849daae2061c2b0c4c89f P 6c194a6db6c145acafc0f3068b8ce1fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } }
I20240502 21:27:34.373198 25458 raft_consensus.cc:374] T a2a0537a60f849daae2061c2b0c4c89f P 6c194a6db6c145acafc0f3068b8ce1fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:34.373267 25458 raft_consensus.cc:729] T a2a0537a60f849daae2061c2b0c4c89f P 6c194a6db6c145acafc0f3068b8ce1fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c194a6db6c145acafc0f3068b8ce1fd, State: Initialized, Role: FOLLOWER
I20240502 21:27:34.373550 25458 consensus_queue.cc:260] T a2a0537a60f849daae2061c2b0c4c89f P 6c194a6db6c145acafc0f3068b8ce1fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } }
I20240502 21:27:34.374194 25458 ts_tablet_manager.cc:1398] T a2a0537a60f849daae2061c2b0c4c89f P 6c194a6db6c145acafc0f3068b8ce1fd: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 21:27:34.375309 25460 tablet_bootstrap.cc:492] T fe044374ad7b4999afc72ab812466749 P 009d3e7e8a8b4e2497de3a73d01ced49: No bootstrap required, opened a new log
I20240502 21:27:34.375448 25460 ts_tablet_manager.cc:1367] T fe044374ad7b4999afc72ab812466749 P 009d3e7e8a8b4e2497de3a73d01ced49: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20240502 21:27:34.376077 25460 raft_consensus.cc:348] T fe044374ad7b4999afc72ab812466749 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } }
I20240502 21:27:34.376286 25460 raft_consensus.cc:374] T fe044374ad7b4999afc72ab812466749 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:34.376359 25460 raft_consensus.cc:729] T fe044374ad7b4999afc72ab812466749 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 009d3e7e8a8b4e2497de3a73d01ced49, State: Initialized, Role: FOLLOWER
I20240502 21:27:34.376616 25460 consensus_queue.cc:260] T fe044374ad7b4999afc72ab812466749 P 009d3e7e8a8b4e2497de3a73d01ced49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } }
I20240502 21:27:34.377238 25460 ts_tablet_manager.cc:1398] T fe044374ad7b4999afc72ab812466749 P 009d3e7e8a8b4e2497de3a73d01ced49: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:27:34.377584 25460 tablet_bootstrap.cc:492] T a2a0537a60f849daae2061c2b0c4c89f P 009d3e7e8a8b4e2497de3a73d01ced49: Bootstrap starting.
I20240502 21:27:34.378784 25457 tablet_bootstrap.cc:654] T fe044374ad7b4999afc72ab812466749 P 6810a866decb4f5089f7d35c870e4204: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:34.378861 25460 tablet_bootstrap.cc:654] T a2a0537a60f849daae2061c2b0c4c89f P 009d3e7e8a8b4e2497de3a73d01ced49: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:34.381407 25457 tablet_bootstrap.cc:492] T fe044374ad7b4999afc72ab812466749 P 6810a866decb4f5089f7d35c870e4204: No bootstrap required, opened a new log
I20240502 21:27:34.381537 25457 ts_tablet_manager.cc:1367] T fe044374ad7b4999afc72ab812466749 P 6810a866decb4f5089f7d35c870e4204: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240502 21:27:34.381866 25460 tablet_bootstrap.cc:492] T a2a0537a60f849daae2061c2b0c4c89f P 009d3e7e8a8b4e2497de3a73d01ced49: No bootstrap required, opened a new log
I20240502 21:27:34.381986 25460 ts_tablet_manager.cc:1367] T a2a0537a60f849daae2061c2b0c4c89f P 009d3e7e8a8b4e2497de3a73d01ced49: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.004s
I20240502 21:27:34.382180 25457 raft_consensus.cc:348] T fe044374ad7b4999afc72ab812466749 P 6810a866decb4f5089f7d35c870e4204 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } }
I20240502 21:27:34.382320 25457 raft_consensus.cc:374] T fe044374ad7b4999afc72ab812466749 P 6810a866decb4f5089f7d35c870e4204 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:34.382400 25457 raft_consensus.cc:729] T fe044374ad7b4999afc72ab812466749 P 6810a866decb4f5089f7d35c870e4204 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6810a866decb4f5089f7d35c870e4204, State: Initialized, Role: FOLLOWER
I20240502 21:27:34.382583 25457 consensus_queue.cc:260] T fe044374ad7b4999afc72ab812466749 P 6810a866decb4f5089f7d35c870e4204 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } }
I20240502 21:27:34.382579 25460 raft_consensus.cc:348] T a2a0537a60f849daae2061c2b0c4c89f P 009d3e7e8a8b4e2497de3a73d01ced49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } }
I20240502 21:27:34.382717 25460 raft_consensus.cc:374] T a2a0537a60f849daae2061c2b0c4c89f P 009d3e7e8a8b4e2497de3a73d01ced49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:34.382807 25460 raft_consensus.cc:729] T a2a0537a60f849daae2061c2b0c4c89f P 009d3e7e8a8b4e2497de3a73d01ced49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 009d3e7e8a8b4e2497de3a73d01ced49, State: Initialized, Role: FOLLOWER
I20240502 21:27:34.383040 25457 ts_tablet_manager.cc:1398] T fe044374ad7b4999afc72ab812466749 P 6810a866decb4f5089f7d35c870e4204: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240502 21:27:34.383002 25460 consensus_queue.cc:260] T a2a0537a60f849daae2061c2b0c4c89f P 009d3e7e8a8b4e2497de3a73d01ced49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } }
I20240502 21:27:34.383430 25460 ts_tablet_manager.cc:1398] T a2a0537a60f849daae2061c2b0c4c89f P 009d3e7e8a8b4e2497de3a73d01ced49: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 21:27:34.415485 25289 raft_consensus.cc:482] T a2a0537a60f849daae2061c2b0c4c89f P 009d3e7e8a8b4e2497de3a73d01ced49 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:34.415656 25289 raft_consensus.cc:504] T a2a0537a60f849daae2061c2b0c4c89f P 009d3e7e8a8b4e2497de3a73d01ced49 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } }
I20240502 21:27:34.416378 25289 leader_election.cc:288] T a2a0537a60f849daae2061c2b0c4c89f P 009d3e7e8a8b4e2497de3a73d01ced49 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6c194a6db6c145acafc0f3068b8ce1fd (127.15.255.130:41235), 6810a866decb4f5089f7d35c870e4204 (127.15.255.129:40435)
I20240502 21:27:34.416800 24049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2a0537a60f849daae2061c2b0c4c89f" candidate_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" is_pre_election: true
I20240502 21:27:34.416973 23915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2a0537a60f849daae2061c2b0c4c89f" candidate_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6810a866decb4f5089f7d35c870e4204" is_pre_election: true
I20240502 21:27:34.417047 24049 raft_consensus.cc:2454] T a2a0537a60f849daae2061c2b0c4c89f P 6c194a6db6c145acafc0f3068b8ce1fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 009d3e7e8a8b4e2497de3a73d01ced49 in term 0.
I20240502 21:27:34.417189 23915 raft_consensus.cc:2454] T a2a0537a60f849daae2061c2b0c4c89f P 6810a866decb4f5089f7d35c870e4204 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 009d3e7e8a8b4e2497de3a73d01ced49 in term 0.
I20240502 21:27:34.417490 24118 leader_election.cc:302] T a2a0537a60f849daae2061c2b0c4c89f P 009d3e7e8a8b4e2497de3a73d01ced49 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 009d3e7e8a8b4e2497de3a73d01ced49, 6c194a6db6c145acafc0f3068b8ce1fd; no voters:
I20240502 21:27:34.417740 25289 raft_consensus.cc:2781] T a2a0537a60f849daae2061c2b0c4c89f P 009d3e7e8a8b4e2497de3a73d01ced49 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:34.417858 25289 raft_consensus.cc:482] T a2a0537a60f849daae2061c2b0c4c89f P 009d3e7e8a8b4e2497de3a73d01ced49 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:34.417946 25289 raft_consensus.cc:3037] T a2a0537a60f849daae2061c2b0c4c89f P 009d3e7e8a8b4e2497de3a73d01ced49 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:34.420053 25289 raft_consensus.cc:504] T a2a0537a60f849daae2061c2b0c4c89f P 009d3e7e8a8b4e2497de3a73d01ced49 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } }
I20240502 21:27:34.420703 25289 leader_election.cc:288] T a2a0537a60f849daae2061c2b0c4c89f P 009d3e7e8a8b4e2497de3a73d01ced49 [CANDIDATE]: Term 1 election: Requested vote from peers 6c194a6db6c145acafc0f3068b8ce1fd (127.15.255.130:41235), 6810a866decb4f5089f7d35c870e4204 (127.15.255.129:40435)
I20240502 21:27:34.421202 23915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2a0537a60f849daae2061c2b0c4c89f" candidate_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6810a866decb4f5089f7d35c870e4204"
I20240502 21:27:34.421382 23915 raft_consensus.cc:3037] T a2a0537a60f849daae2061c2b0c4c89f P 6810a866decb4f5089f7d35c870e4204 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:34.421399 24049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2a0537a60f849daae2061c2b0c4c89f" candidate_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c194a6db6c145acafc0f3068b8ce1fd"
I20240502 21:27:34.421591 24049 raft_consensus.cc:3037] T a2a0537a60f849daae2061c2b0c4c89f P 6c194a6db6c145acafc0f3068b8ce1fd [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:34.424017 23915 raft_consensus.cc:2454] T a2a0537a60f849daae2061c2b0c4c89f P 6810a866decb4f5089f7d35c870e4204 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 009d3e7e8a8b4e2497de3a73d01ced49 in term 1.
I20240502 21:27:34.424279 24049 raft_consensus.cc:2454] T a2a0537a60f849daae2061c2b0c4c89f P 6c194a6db6c145acafc0f3068b8ce1fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 009d3e7e8a8b4e2497de3a73d01ced49 in term 1.
I20240502 21:27:34.424412 24118 leader_election.cc:302] T a2a0537a60f849daae2061c2b0c4c89f P 009d3e7e8a8b4e2497de3a73d01ced49 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 009d3e7e8a8b4e2497de3a73d01ced49, 6810a866decb4f5089f7d35c870e4204; no voters:
I20240502 21:27:34.424703 25289 raft_consensus.cc:2781] T a2a0537a60f849daae2061c2b0c4c89f P 009d3e7e8a8b4e2497de3a73d01ced49 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:34.424837 25289 raft_consensus.cc:686] T a2a0537a60f849daae2061c2b0c4c89f P 009d3e7e8a8b4e2497de3a73d01ced49 [term 1 LEADER]: Becoming Leader. State: Replica: 009d3e7e8a8b4e2497de3a73d01ced49, State: Running, Role: LEADER
I20240502 21:27:34.425073 25289 consensus_queue.cc:237] T a2a0537a60f849daae2061c2b0c4c89f P 009d3e7e8a8b4e2497de3a73d01ced49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } }
I20240502 21:27:34.427702 23765 catalog_manager.cc:5484] T a2a0537a60f849daae2061c2b0c4c89f P 009d3e7e8a8b4e2497de3a73d01ced49 reported cstate change: term changed from 0 to 1, leader changed from <none> to 009d3e7e8a8b4e2497de3a73d01ced49 (127.15.255.131). New cstate: current_term: 1 leader_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:34.521555 25390 raft_consensus.cc:482] T fe044374ad7b4999afc72ab812466749 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:34.521737 25390 raft_consensus.cc:504] T fe044374ad7b4999afc72ab812466749 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } }
I20240502 21:27:34.522423 25390 leader_election.cc:288] T fe044374ad7b4999afc72ab812466749 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 009d3e7e8a8b4e2497de3a73d01ced49 (127.15.255.131:37895), 6810a866decb4f5089f7d35c870e4204 (127.15.255.129:40435)
I20240502 21:27:34.523026 24185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe044374ad7b4999afc72ab812466749" candidate_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" is_pre_election: true
I20240502 21:27:34.523005 23915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe044374ad7b4999afc72ab812466749" candidate_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6810a866decb4f5089f7d35c870e4204" is_pre_election: true
I20240502 21:27:34.523241 24185 raft_consensus.cc:2454] T fe044374ad7b4999afc72ab812466749 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 038c5d9d1d7a49c981c4c34fbc00d9e6 in term 0.
I20240502 21:27:34.523263 23915 raft_consensus.cc:2454] T fe044374ad7b4999afc72ab812466749 P 6810a866decb4f5089f7d35c870e4204 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 038c5d9d1d7a49c981c4c34fbc00d9e6 in term 0.
I20240502 21:27:34.523665 24252 leader_election.cc:302] T fe044374ad7b4999afc72ab812466749 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 009d3e7e8a8b4e2497de3a73d01ced49, 038c5d9d1d7a49c981c4c34fbc00d9e6; no voters:
I20240502 21:27:34.523984 25390 raft_consensus.cc:2781] T fe044374ad7b4999afc72ab812466749 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:34.524072 25390 raft_consensus.cc:482] T fe044374ad7b4999afc72ab812466749 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:34.524170 25390 raft_consensus.cc:3037] T fe044374ad7b4999afc72ab812466749 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:34.526013 25390 raft_consensus.cc:504] T fe044374ad7b4999afc72ab812466749 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } }
I20240502 21:27:34.526538 25390 leader_election.cc:288] T fe044374ad7b4999afc72ab812466749 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [CANDIDATE]: Term 1 election: Requested vote from peers 009d3e7e8a8b4e2497de3a73d01ced49 (127.15.255.131:37895), 6810a866decb4f5089f7d35c870e4204 (127.15.255.129:40435)
I20240502 21:27:34.526921 24185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe044374ad7b4999afc72ab812466749" candidate_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "009d3e7e8a8b4e2497de3a73d01ced49"
I20240502 21:27:34.527108 24185 raft_consensus.cc:3037] T fe044374ad7b4999afc72ab812466749 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:34.526991 23915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe044374ad7b4999afc72ab812466749" candidate_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6810a866decb4f5089f7d35c870e4204"
I20240502 21:27:34.527323 23915 raft_consensus.cc:3037] T fe044374ad7b4999afc72ab812466749 P 6810a866decb4f5089f7d35c870e4204 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:34.529069 24185 raft_consensus.cc:2454] T fe044374ad7b4999afc72ab812466749 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 038c5d9d1d7a49c981c4c34fbc00d9e6 in term 1.
I20240502 21:27:34.529415 24252 leader_election.cc:302] T fe044374ad7b4999afc72ab812466749 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 009d3e7e8a8b4e2497de3a73d01ced49, 038c5d9d1d7a49c981c4c34fbc00d9e6; no voters:
I20240502 21:27:34.529670 25390 raft_consensus.cc:2781] T fe044374ad7b4999afc72ab812466749 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:34.529978 25390 raft_consensus.cc:686] T fe044374ad7b4999afc72ab812466749 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 1 LEADER]: Becoming Leader. State: Replica: 038c5d9d1d7a49c981c4c34fbc00d9e6, State: Running, Role: LEADER
I20240502 21:27:34.530218 25390 consensus_queue.cc:237] T fe044374ad7b4999afc72ab812466749 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } }
I20240502 21:27:34.531522 23915 raft_consensus.cc:2454] T fe044374ad7b4999afc72ab812466749 P 6810a866decb4f5089f7d35c870e4204 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 038c5d9d1d7a49c981c4c34fbc00d9e6 in term 1.
I20240502 21:27:34.533447 23765 catalog_manager.cc:5484] T fe044374ad7b4999afc72ab812466749 P 038c5d9d1d7a49c981c4c34fbc00d9e6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 038c5d9d1d7a49c981c4c34fbc00d9e6 (127.15.255.132). New cstate: current_term: 1 leader_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } health_report { overall_health: HEALTHY } } }
I20240502 21:27:34.645897 25472 raft_consensus.cc:482] T 256ae04a48b849c4a9086b6b1ff5df19 P 6810a866decb4f5089f7d35c870e4204 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:34.646056 25472 raft_consensus.cc:504] T 256ae04a48b849c4a9086b6b1ff5df19 P 6810a866decb4f5089f7d35c870e4204 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } }
I20240502 21:27:34.646672 25472 leader_election.cc:288] T 256ae04a48b849c4a9086b6b1ff5df19 P 6810a866decb4f5089f7d35c870e4204 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 009d3e7e8a8b4e2497de3a73d01ced49 (127.15.255.131:37895), 6c194a6db6c145acafc0f3068b8ce1fd (127.15.255.130:41235)
I20240502 21:27:34.646998 24185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256ae04a48b849c4a9086b6b1ff5df19" candidate_uuid: "6810a866decb4f5089f7d35c870e4204" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" is_pre_election: true
I20240502 21:27:34.647094 24049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256ae04a48b849c4a9086b6b1ff5df19" candidate_uuid: "6810a866decb4f5089f7d35c870e4204" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" is_pre_election: true
I20240502 21:27:34.647255 24185 raft_consensus.cc:2454] T 256ae04a48b849c4a9086b6b1ff5df19 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6810a866decb4f5089f7d35c870e4204 in term 0.
I20240502 21:27:34.647329 24049 raft_consensus.cc:2454] T 256ae04a48b849c4a9086b6b1ff5df19 P 6c194a6db6c145acafc0f3068b8ce1fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6810a866decb4f5089f7d35c870e4204 in term 0.
I20240502 21:27:34.647683 23850 leader_election.cc:302] T 256ae04a48b849c4a9086b6b1ff5df19 P 6810a866decb4f5089f7d35c870e4204 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 009d3e7e8a8b4e2497de3a73d01ced49, 6810a866decb4f5089f7d35c870e4204; no voters:
I20240502 21:27:34.647984 25472 raft_consensus.cc:2781] T 256ae04a48b849c4a9086b6b1ff5df19 P 6810a866decb4f5089f7d35c870e4204 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:34.648108 25472 raft_consensus.cc:482] T 256ae04a48b849c4a9086b6b1ff5df19 P 6810a866decb4f5089f7d35c870e4204 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:34.648207 25472 raft_consensus.cc:3037] T 256ae04a48b849c4a9086b6b1ff5df19 P 6810a866decb4f5089f7d35c870e4204 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:34.650496 25472 raft_consensus.cc:504] T 256ae04a48b849c4a9086b6b1ff5df19 P 6810a866decb4f5089f7d35c870e4204 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } }
I20240502 21:27:34.651000 25472 leader_election.cc:288] T 256ae04a48b849c4a9086b6b1ff5df19 P 6810a866decb4f5089f7d35c870e4204 [CANDIDATE]: Term 1 election: Requested vote from peers 009d3e7e8a8b4e2497de3a73d01ced49 (127.15.255.131:37895), 6c194a6db6c145acafc0f3068b8ce1fd (127.15.255.130:41235)
I20240502 21:27:34.651331 24185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256ae04a48b849c4a9086b6b1ff5df19" candidate_uuid: "6810a866decb4f5089f7d35c870e4204" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "009d3e7e8a8b4e2497de3a73d01ced49"
I20240502 21:27:34.651427 24049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256ae04a48b849c4a9086b6b1ff5df19" candidate_uuid: "6810a866decb4f5089f7d35c870e4204" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c194a6db6c145acafc0f3068b8ce1fd"
I20240502 21:27:34.651530 24185 raft_consensus.cc:3037] T 256ae04a48b849c4a9086b6b1ff5df19 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:34.651630 24049 raft_consensus.cc:3037] T 256ae04a48b849c4a9086b6b1ff5df19 P 6c194a6db6c145acafc0f3068b8ce1fd [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:34.653419 24185 raft_consensus.cc:2454] T 256ae04a48b849c4a9086b6b1ff5df19 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6810a866decb4f5089f7d35c870e4204 in term 1.
I20240502 21:27:34.653486 24049 raft_consensus.cc:2454] T 256ae04a48b849c4a9086b6b1ff5df19 P 6c194a6db6c145acafc0f3068b8ce1fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6810a866decb4f5089f7d35c870e4204 in term 1.
I20240502 21:27:34.653815 23850 leader_election.cc:302] T 256ae04a48b849c4a9086b6b1ff5df19 P 6810a866decb4f5089f7d35c870e4204 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 009d3e7e8a8b4e2497de3a73d01ced49, 6810a866decb4f5089f7d35c870e4204; no voters:
I20240502 21:27:34.654134 25472 raft_consensus.cc:2781] T 256ae04a48b849c4a9086b6b1ff5df19 P 6810a866decb4f5089f7d35c870e4204 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:34.654278 25472 raft_consensus.cc:686] T 256ae04a48b849c4a9086b6b1ff5df19 P 6810a866decb4f5089f7d35c870e4204 [term 1 LEADER]: Becoming Leader. State: Replica: 6810a866decb4f5089f7d35c870e4204, State: Running, Role: LEADER
I20240502 21:27:34.654536 25472 consensus_queue.cc:237] T 256ae04a48b849c4a9086b6b1ff5df19 P 6810a866decb4f5089f7d35c870e4204 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } }
I20240502 21:27:34.657346 23765 catalog_manager.cc:5484] T 256ae04a48b849c4a9086b6b1ff5df19 P 6810a866decb4f5089f7d35c870e4204 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6810a866decb4f5089f7d35c870e4204 (127.15.255.129). New cstate: current_term: 1 leader_uuid: "6810a866decb4f5089f7d35c870e4204" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:34.894484 25435 consensus_queue.cc:1035] T a2a0537a60f849daae2061c2b0c4c89f P 009d3e7e8a8b4e2497de3a73d01ced49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:34.901950 25435 consensus_queue.cc:1035] T a2a0537a60f849daae2061c2b0c4c89f P 009d3e7e8a8b4e2497de3a73d01ced49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 21:27:35.036996 25474 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:27:35.037307 25474 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240502 21:27:35.038139 25390 consensus_queue.cc:1035] T fe044374ad7b4999afc72ab812466749 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:35.039566 25472 consensus_queue.cc:1035] T 256ae04a48b849c4a9086b6b1ff5df19 P 6810a866decb4f5089f7d35c870e4204 [LEADER]: Connected to new peer: Peer: permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 21:27:35.049090 25474 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 21:27:35.057276 25265 consensus_queue.cc:1035] T 256ae04a48b849c4a9086b6b1ff5df19 P 6810a866decb4f5089f7d35c870e4204 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:35.060456 25390 consensus_queue.cc:1035] T fe044374ad7b4999afc72ab812466749 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_6'
INSERT report
rows total: 2000
time total: 229.775 ms
time per row: 0.114888 ms
I20240502 21:27:35.437031 23771 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41618:
name: "rebalance_test_table_7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 21:27:35.445394 24297 tablet_service.cc:1467] Processing CreateTablet for tablet 77fc7226878f44d3884ccb1fc41c44a4 (DEFAULT_TABLE table=rebalance_test_table_7 [id=754dbb3bce6a4875a0b642faf5315709]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:35.445565 24027 tablet_service.cc:1467] Processing CreateTablet for tablet 77fc7226878f44d3884ccb1fc41c44a4 (DEFAULT_TABLE table=rebalance_test_table_7 [id=754dbb3bce6a4875a0b642faf5315709]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:35.445859 24297 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77fc7226878f44d3884ccb1fc41c44a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:35.445977 24027 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77fc7226878f44d3884ccb1fc41c44a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:35.446480 24295 tablet_service.cc:1467] Processing CreateTablet for tablet f84bbe304597450c978ae28e0c4e7a3e (DEFAULT_TABLE table=rebalance_test_table_7 [id=754dbb3bce6a4875a0b642faf5315709]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:35.446786 24295 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f84bbe304597450c978ae28e0c4e7a3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:35.448577 24162 tablet_service.cc:1467] Processing CreateTablet for tablet 6bd759d700a24e9f8ad11f243cf7ebfa (DEFAULT_TABLE table=rebalance_test_table_7 [id=754dbb3bce6a4875a0b642faf5315709]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:35.448673 24164 tablet_service.cc:1467] Processing CreateTablet for tablet f84bbe304597450c978ae28e0c4e7a3e (DEFAULT_TABLE table=rebalance_test_table_7 [id=754dbb3bce6a4875a0b642faf5315709]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:35.448982 24162 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6bd759d700a24e9f8ad11f243cf7ebfa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:35.449509 23895 tablet_service.cc:1467] Processing CreateTablet for tablet 6bd759d700a24e9f8ad11f243cf7ebfa (DEFAULT_TABLE table=rebalance_test_table_7 [id=754dbb3bce6a4875a0b642faf5315709]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:35.449896 23895 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6bd759d700a24e9f8ad11f243cf7ebfa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:35.450217 24296 tablet_service.cc:1467] Processing CreateTablet for tablet 6bd759d700a24e9f8ad11f243cf7ebfa (DEFAULT_TABLE table=rebalance_test_table_7 [id=754dbb3bce6a4875a0b642faf5315709]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:35.450563 24296 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6bd759d700a24e9f8ad11f243cf7ebfa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:35.452771 23894 tablet_service.cc:1467] Processing CreateTablet for tablet 77fc7226878f44d3884ccb1fc41c44a4 (DEFAULT_TABLE table=rebalance_test_table_7 [id=754dbb3bce6a4875a0b642faf5315709]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:35.453095 23894 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77fc7226878f44d3884ccb1fc41c44a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:35.456413 24164 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f84bbe304597450c978ae28e0c4e7a3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:35.457194 24029 tablet_service.cc:1467] Processing CreateTablet for tablet f84bbe304597450c978ae28e0c4e7a3e (DEFAULT_TABLE table=rebalance_test_table_7 [id=754dbb3bce6a4875a0b642faf5315709]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:35.457573 24029 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f84bbe304597450c978ae28e0c4e7a3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:35.460312 25543 tablet_bootstrap.cc:492] T f84bbe304597450c978ae28e0c4e7a3e P 038c5d9d1d7a49c981c4c34fbc00d9e6: Bootstrap starting.
I20240502 21:27:35.462266 25543 tablet_bootstrap.cc:654] T f84bbe304597450c978ae28e0c4e7a3e P 038c5d9d1d7a49c981c4c34fbc00d9e6: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:35.466883 25542 tablet_bootstrap.cc:492] T 6bd759d700a24e9f8ad11f243cf7ebfa P 6810a866decb4f5089f7d35c870e4204: Bootstrap starting.
I20240502 21:27:35.468384 25541 tablet_bootstrap.cc:492] T 77fc7226878f44d3884ccb1fc41c44a4 P 6c194a6db6c145acafc0f3068b8ce1fd: Bootstrap starting.
I20240502 21:27:35.469444 25542 tablet_bootstrap.cc:654] T 6bd759d700a24e9f8ad11f243cf7ebfa P 6810a866decb4f5089f7d35c870e4204: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:35.469987 25541 tablet_bootstrap.cc:654] T 77fc7226878f44d3884ccb1fc41c44a4 P 6c194a6db6c145acafc0f3068b8ce1fd: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:35.470122 25543 tablet_bootstrap.cc:492] T f84bbe304597450c978ae28e0c4e7a3e P 038c5d9d1d7a49c981c4c34fbc00d9e6: No bootstrap required, opened a new log
I20240502 21:27:35.470300 25543 ts_tablet_manager.cc:1367] T f84bbe304597450c978ae28e0c4e7a3e P 038c5d9d1d7a49c981c4c34fbc00d9e6: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.000s
I20240502 21:27:35.471148 25544 tablet_bootstrap.cc:492] T 6bd759d700a24e9f8ad11f243cf7ebfa P 009d3e7e8a8b4e2497de3a73d01ced49: Bootstrap starting.
I20240502 21:27:35.471143 25543 raft_consensus.cc:348] T f84bbe304597450c978ae28e0c4e7a3e P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } }
I20240502 21:27:35.471349 25543 raft_consensus.cc:374] T f84bbe304597450c978ae28e0c4e7a3e P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:35.471448 25543 raft_consensus.cc:729] T f84bbe304597450c978ae28e0c4e7a3e P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 038c5d9d1d7a49c981c4c34fbc00d9e6, State: Initialized, Role: FOLLOWER
I20240502 21:27:35.471669 25543 consensus_queue.cc:260] T f84bbe304597450c978ae28e0c4e7a3e P 038c5d9d1d7a49c981c4c34fbc00d9e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } }
I20240502 21:27:35.471998 25542 tablet_bootstrap.cc:492] T 6bd759d700a24e9f8ad11f243cf7ebfa P 6810a866decb4f5089f7d35c870e4204: No bootstrap required, opened a new log
I20240502 21:27:35.472174 25542 ts_tablet_manager.cc:1367] T 6bd759d700a24e9f8ad11f243cf7ebfa P 6810a866decb4f5089f7d35c870e4204: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240502 21:27:35.472271 25541 tablet_bootstrap.cc:492] T 77fc7226878f44d3884ccb1fc41c44a4 P 6c194a6db6c145acafc0f3068b8ce1fd: No bootstrap required, opened a new log
I20240502 21:27:35.472262 25543 ts_tablet_manager.cc:1398] T f84bbe304597450c978ae28e0c4e7a3e P 038c5d9d1d7a49c981c4c34fbc00d9e6: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 21:27:35.472501 25541 ts_tablet_manager.cc:1367] T 77fc7226878f44d3884ccb1fc41c44a4 P 6c194a6db6c145acafc0f3068b8ce1fd: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20240502 21:27:35.472609 25543 tablet_bootstrap.cc:492] T 6bd759d700a24e9f8ad11f243cf7ebfa P 038c5d9d1d7a49c981c4c34fbc00d9e6: Bootstrap starting.
I20240502 21:27:35.472975 25542 raft_consensus.cc:348] T 6bd759d700a24e9f8ad11f243cf7ebfa P 6810a866decb4f5089f7d35c870e4204 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } }
I20240502 21:27:35.473237 25542 raft_consensus.cc:374] T 6bd759d700a24e9f8ad11f243cf7ebfa P 6810a866decb4f5089f7d35c870e4204 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:35.473278 25541 raft_consensus.cc:348] T 77fc7226878f44d3884ccb1fc41c44a4 P 6c194a6db6c145acafc0f3068b8ce1fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } }
I20240502 21:27:35.473384 25542 raft_consensus.cc:729] T 6bd759d700a24e9f8ad11f243cf7ebfa P 6810a866decb4f5089f7d35c870e4204 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6810a866decb4f5089f7d35c870e4204, State: Initialized, Role: FOLLOWER
I20240502 21:27:35.473542 25541 raft_consensus.cc:374] T 77fc7226878f44d3884ccb1fc41c44a4 P 6c194a6db6c145acafc0f3068b8ce1fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:35.473693 25541 raft_consensus.cc:729] T 77fc7226878f44d3884ccb1fc41c44a4 P 6c194a6db6c145acafc0f3068b8ce1fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c194a6db6c145acafc0f3068b8ce1fd, State: Initialized, Role: FOLLOWER
I20240502 21:27:35.473628 25542 consensus_queue.cc:260] T 6bd759d700a24e9f8ad11f243cf7ebfa P 6810a866decb4f5089f7d35c870e4204 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } }
I20240502 21:27:35.473997 25541 consensus_queue.cc:260] T 77fc7226878f44d3884ccb1fc41c44a4 P 6c194a6db6c145acafc0f3068b8ce1fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } }
I20240502 21:27:35.474241 25542 ts_tablet_manager.cc:1398] T 6bd759d700a24e9f8ad11f243cf7ebfa P 6810a866decb4f5089f7d35c870e4204: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 21:27:35.474902 25542 tablet_bootstrap.cc:492] T 77fc7226878f44d3884ccb1fc41c44a4 P 6810a866decb4f5089f7d35c870e4204: Bootstrap starting.
I20240502 21:27:35.476070 25544 tablet_bootstrap.cc:654] T 6bd759d700a24e9f8ad11f243cf7ebfa P 009d3e7e8a8b4e2497de3a73d01ced49: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:35.476320 25541 ts_tablet_manager.cc:1398] T 77fc7226878f44d3884ccb1fc41c44a4 P 6c194a6db6c145acafc0f3068b8ce1fd: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240502 21:27:35.476645 25541 tablet_bootstrap.cc:492] T f84bbe304597450c978ae28e0c4e7a3e P 6c194a6db6c145acafc0f3068b8ce1fd: Bootstrap starting.
I20240502 21:27:35.479178 25543 tablet_bootstrap.cc:654] T 6bd759d700a24e9f8ad11f243cf7ebfa P 038c5d9d1d7a49c981c4c34fbc00d9e6: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:35.481453 25541 tablet_bootstrap.cc:654] T f84bbe304597450c978ae28e0c4e7a3e P 6c194a6db6c145acafc0f3068b8ce1fd: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:35.482003 25542 tablet_bootstrap.cc:654] T 77fc7226878f44d3884ccb1fc41c44a4 P 6810a866decb4f5089f7d35c870e4204: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:35.484227 25543 tablet_bootstrap.cc:492] T 6bd759d700a24e9f8ad11f243cf7ebfa P 038c5d9d1d7a49c981c4c34fbc00d9e6: No bootstrap required, opened a new log
I20240502 21:27:35.484277 25544 tablet_bootstrap.cc:492] T 6bd759d700a24e9f8ad11f243cf7ebfa P 009d3e7e8a8b4e2497de3a73d01ced49: No bootstrap required, opened a new log
I20240502 21:27:35.484454 25544 ts_tablet_manager.cc:1367] T 6bd759d700a24e9f8ad11f243cf7ebfa P 009d3e7e8a8b4e2497de3a73d01ced49: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.009s
I20240502 21:27:35.484570 25543 ts_tablet_manager.cc:1367] T 6bd759d700a24e9f8ad11f243cf7ebfa P 038c5d9d1d7a49c981c4c34fbc00d9e6: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.000s
I20240502 21:27:35.485270 25544 raft_consensus.cc:348] T 6bd759d700a24e9f8ad11f243cf7ebfa P 009d3e7e8a8b4e2497de3a73d01ced49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } }
I20240502 21:27:35.485260 25543 raft_consensus.cc:348] T 6bd759d700a24e9f8ad11f243cf7ebfa P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } }
I20240502 21:27:35.485472 25543 raft_consensus.cc:374] T 6bd759d700a24e9f8ad11f243cf7ebfa P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:35.485520 25544 raft_consensus.cc:374] T 6bd759d700a24e9f8ad11f243cf7ebfa P 009d3e7e8a8b4e2497de3a73d01ced49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:35.485585 25543 raft_consensus.cc:729] T 6bd759d700a24e9f8ad11f243cf7ebfa P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 038c5d9d1d7a49c981c4c34fbc00d9e6, State: Initialized, Role: FOLLOWER
I20240502 21:27:35.485682 25544 raft_consensus.cc:729] T 6bd759d700a24e9f8ad11f243cf7ebfa P 009d3e7e8a8b4e2497de3a73d01ced49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 009d3e7e8a8b4e2497de3a73d01ced49, State: Initialized, Role: FOLLOWER
I20240502 21:27:35.485842 25543 consensus_queue.cc:260] T 6bd759d700a24e9f8ad11f243cf7ebfa P 038c5d9d1d7a49c981c4c34fbc00d9e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } }
I20240502 21:27:35.486053 25544 consensus_queue.cc:260] T 6bd759d700a24e9f8ad11f243cf7ebfa P 009d3e7e8a8b4e2497de3a73d01ced49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } }
I20240502 21:27:35.487409 25543 ts_tablet_manager.cc:1398] T 6bd759d700a24e9f8ad11f243cf7ebfa P 038c5d9d1d7a49c981c4c34fbc00d9e6: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 21:27:35.487759 25543 tablet_bootstrap.cc:492] T 77fc7226878f44d3884ccb1fc41c44a4 P 038c5d9d1d7a49c981c4c34fbc00d9e6: Bootstrap starting.
I20240502 21:27:35.487849 25544 ts_tablet_manager.cc:1398] T 6bd759d700a24e9f8ad11f243cf7ebfa P 009d3e7e8a8b4e2497de3a73d01ced49: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240502 21:27:35.488221 25544 tablet_bootstrap.cc:492] T f84bbe304597450c978ae28e0c4e7a3e P 009d3e7e8a8b4e2497de3a73d01ced49: Bootstrap starting.
I20240502 21:27:35.489197 25543 tablet_bootstrap.cc:654] T 77fc7226878f44d3884ccb1fc41c44a4 P 038c5d9d1d7a49c981c4c34fbc00d9e6: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:35.489784 25544 tablet_bootstrap.cc:654] T f84bbe304597450c978ae28e0c4e7a3e P 009d3e7e8a8b4e2497de3a73d01ced49: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:35.491717 25542 tablet_bootstrap.cc:492] T 77fc7226878f44d3884ccb1fc41c44a4 P 6810a866decb4f5089f7d35c870e4204: No bootstrap required, opened a new log
I20240502 21:27:35.492229 25542 ts_tablet_manager.cc:1367] T 77fc7226878f44d3884ccb1fc41c44a4 P 6810a866decb4f5089f7d35c870e4204: Time spent bootstrapping tablet: real 0.017s user 0.001s sys 0.009s
I20240502 21:27:35.493103 25542 raft_consensus.cc:348] T 77fc7226878f44d3884ccb1fc41c44a4 P 6810a866decb4f5089f7d35c870e4204 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } }
I20240502 21:27:35.493382 25542 raft_consensus.cc:374] T 77fc7226878f44d3884ccb1fc41c44a4 P 6810a866decb4f5089f7d35c870e4204 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:35.493512 25542 raft_consensus.cc:729] T 77fc7226878f44d3884ccb1fc41c44a4 P 6810a866decb4f5089f7d35c870e4204 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6810a866decb4f5089f7d35c870e4204, State: Initialized, Role: FOLLOWER
I20240502 21:27:35.493685 25544 tablet_bootstrap.cc:492] T f84bbe304597450c978ae28e0c4e7a3e P 009d3e7e8a8b4e2497de3a73d01ced49: No bootstrap required, opened a new log
I20240502 21:27:35.493902 25544 ts_tablet_manager.cc:1367] T f84bbe304597450c978ae28e0c4e7a3e P 009d3e7e8a8b4e2497de3a73d01ced49: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20240502 21:27:35.494671 25544 raft_consensus.cc:348] T f84bbe304597450c978ae28e0c4e7a3e P 009d3e7e8a8b4e2497de3a73d01ced49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } }
I20240502 21:27:35.495066 25544 raft_consensus.cc:374] T f84bbe304597450c978ae28e0c4e7a3e P 009d3e7e8a8b4e2497de3a73d01ced49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:35.495141 25544 raft_consensus.cc:729] T f84bbe304597450c978ae28e0c4e7a3e P 009d3e7e8a8b4e2497de3a73d01ced49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 009d3e7e8a8b4e2497de3a73d01ced49, State: Initialized, Role: FOLLOWER
I20240502 21:27:35.493865 25542 consensus_queue.cc:260] T 77fc7226878f44d3884ccb1fc41c44a4 P 6810a866decb4f5089f7d35c870e4204 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } }
I20240502 21:27:35.495397 25544 consensus_queue.cc:260] T f84bbe304597450c978ae28e0c4e7a3e P 009d3e7e8a8b4e2497de3a73d01ced49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } }
I20240502 21:27:35.495829 25542 ts_tablet_manager.cc:1398] T 77fc7226878f44d3884ccb1fc41c44a4 P 6810a866decb4f5089f7d35c870e4204: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 21:27:35.496618 25544 ts_tablet_manager.cc:1398] T f84bbe304597450c978ae28e0c4e7a3e P 009d3e7e8a8b4e2497de3a73d01ced49: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 21:27:35.497632 25541 tablet_bootstrap.cc:492] T f84bbe304597450c978ae28e0c4e7a3e P 6c194a6db6c145acafc0f3068b8ce1fd: No bootstrap required, opened a new log
I20240502 21:27:35.497968 25543 tablet_bootstrap.cc:492] T 77fc7226878f44d3884ccb1fc41c44a4 P 038c5d9d1d7a49c981c4c34fbc00d9e6: No bootstrap required, opened a new log
I20240502 21:27:35.498096 25543 ts_tablet_manager.cc:1367] T 77fc7226878f44d3884ccb1fc41c44a4 P 038c5d9d1d7a49c981c4c34fbc00d9e6: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20240502 21:27:35.498627 25541 ts_tablet_manager.cc:1367] T f84bbe304597450c978ae28e0c4e7a3e P 6c194a6db6c145acafc0f3068b8ce1fd: Time spent bootstrapping tablet: real 0.022s user 0.006s sys 0.000s
I20240502 21:27:35.498772 25543 raft_consensus.cc:348] T 77fc7226878f44d3884ccb1fc41c44a4 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } }
I20240502 21:27:35.498962 25543 raft_consensus.cc:374] T 77fc7226878f44d3884ccb1fc41c44a4 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:35.499063 25543 raft_consensus.cc:729] T 77fc7226878f44d3884ccb1fc41c44a4 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 038c5d9d1d7a49c981c4c34fbc00d9e6, State: Initialized, Role: FOLLOWER
I20240502 21:27:35.499284 25543 consensus_queue.cc:260] T 77fc7226878f44d3884ccb1fc41c44a4 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } }
I20240502 21:27:35.499320 25541 raft_consensus.cc:348] T f84bbe304597450c978ae28e0c4e7a3e P 6c194a6db6c145acafc0f3068b8ce1fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } }
I20240502 21:27:35.499827 25543 ts_tablet_manager.cc:1398] T 77fc7226878f44d3884ccb1fc41c44a4 P 038c5d9d1d7a49c981c4c34fbc00d9e6: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 21:27:35.499504 25541 raft_consensus.cc:374] T f84bbe304597450c978ae28e0c4e7a3e P 6c194a6db6c145acafc0f3068b8ce1fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:35.500399 25541 raft_consensus.cc:729] T f84bbe304597450c978ae28e0c4e7a3e P 6c194a6db6c145acafc0f3068b8ce1fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c194a6db6c145acafc0f3068b8ce1fd, State: Initialized, Role: FOLLOWER
I20240502 21:27:35.500664 25541 consensus_queue.cc:260] T f84bbe304597450c978ae28e0c4e7a3e P 6c194a6db6c145acafc0f3068b8ce1fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } }
I20240502 21:27:35.502643 25541 ts_tablet_manager.cc:1398] T f84bbe304597450c978ae28e0c4e7a3e P 6c194a6db6c145acafc0f3068b8ce1fd: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240502 21:27:35.536506 25435 raft_consensus.cc:482] T 6bd759d700a24e9f8ad11f243cf7ebfa P 009d3e7e8a8b4e2497de3a73d01ced49 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:35.536636 25435 raft_consensus.cc:504] T 6bd759d700a24e9f8ad11f243cf7ebfa P 009d3e7e8a8b4e2497de3a73d01ced49 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } }
I20240502 21:27:35.537243 25435 leader_election.cc:288] T 6bd759d700a24e9f8ad11f243cf7ebfa P 009d3e7e8a8b4e2497de3a73d01ced49 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 038c5d9d1d7a49c981c4c34fbc00d9e6 (127.15.255.132:46813), 6810a866decb4f5089f7d35c870e4204 (127.15.255.129:40435)
I20240502 21:27:35.537691 24315 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bd759d700a24e9f8ad11f243cf7ebfa" candidate_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" is_pre_election: true
I20240502 21:27:35.537742 23915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bd759d700a24e9f8ad11f243cf7ebfa" candidate_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6810a866decb4f5089f7d35c870e4204" is_pre_election: true
I20240502 21:27:35.537873 24315 raft_consensus.cc:2454] T 6bd759d700a24e9f8ad11f243cf7ebfa P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 009d3e7e8a8b4e2497de3a73d01ced49 in term 0.
I20240502 21:27:35.537953 23915 raft_consensus.cc:2454] T 6bd759d700a24e9f8ad11f243cf7ebfa P 6810a866decb4f5089f7d35c870e4204 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 009d3e7e8a8b4e2497de3a73d01ced49 in term 0.
I20240502 21:27:35.538312 24116 leader_election.cc:302] T 6bd759d700a24e9f8ad11f243cf7ebfa P 009d3e7e8a8b4e2497de3a73d01ced49 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 009d3e7e8a8b4e2497de3a73d01ced49, 038c5d9d1d7a49c981c4c34fbc00d9e6; no voters:
I20240502 21:27:35.538589 25435 raft_consensus.cc:2781] T 6bd759d700a24e9f8ad11f243cf7ebfa P 009d3e7e8a8b4e2497de3a73d01ced49 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:35.538690 25435 raft_consensus.cc:482] T 6bd759d700a24e9f8ad11f243cf7ebfa P 009d3e7e8a8b4e2497de3a73d01ced49 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:35.538772 25435 raft_consensus.cc:3037] T 6bd759d700a24e9f8ad11f243cf7ebfa P 009d3e7e8a8b4e2497de3a73d01ced49 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:35.540680 25435 raft_consensus.cc:504] T 6bd759d700a24e9f8ad11f243cf7ebfa P 009d3e7e8a8b4e2497de3a73d01ced49 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } }
I20240502 21:27:35.541154 25435 leader_election.cc:288] T 6bd759d700a24e9f8ad11f243cf7ebfa P 009d3e7e8a8b4e2497de3a73d01ced49 [CANDIDATE]: Term 1 election: Requested vote from peers 038c5d9d1d7a49c981c4c34fbc00d9e6 (127.15.255.132:46813), 6810a866decb4f5089f7d35c870e4204 (127.15.255.129:40435)
I20240502 21:27:35.541551 24315 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bd759d700a24e9f8ad11f243cf7ebfa" candidate_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6"
I20240502 21:27:35.541611 23915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bd759d700a24e9f8ad11f243cf7ebfa" candidate_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6810a866decb4f5089f7d35c870e4204"
I20240502 21:27:35.541718 24315 raft_consensus.cc:3037] T 6bd759d700a24e9f8ad11f243cf7ebfa P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:35.541782 23915 raft_consensus.cc:3037] T 6bd759d700a24e9f8ad11f243cf7ebfa P 6810a866decb4f5089f7d35c870e4204 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:35.543896 23915 raft_consensus.cc:2454] T 6bd759d700a24e9f8ad11f243cf7ebfa P 6810a866decb4f5089f7d35c870e4204 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 009d3e7e8a8b4e2497de3a73d01ced49 in term 1.
I20240502 21:27:35.543946 24315 raft_consensus.cc:2454] T 6bd759d700a24e9f8ad11f243cf7ebfa P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 009d3e7e8a8b4e2497de3a73d01ced49 in term 1.
I20240502 21:27:35.544234 24118 leader_election.cc:302] T 6bd759d700a24e9f8ad11f243cf7ebfa P 009d3e7e8a8b4e2497de3a73d01ced49 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 009d3e7e8a8b4e2497de3a73d01ced49, 6810a866decb4f5089f7d35c870e4204; no voters:
I20240502 21:27:35.544605 25435 raft_consensus.cc:2781] T 6bd759d700a24e9f8ad11f243cf7ebfa P 009d3e7e8a8b4e2497de3a73d01ced49 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:35.544723 25435 raft_consensus.cc:686] T 6bd759d700a24e9f8ad11f243cf7ebfa P 009d3e7e8a8b4e2497de3a73d01ced49 [term 1 LEADER]: Becoming Leader. State: Replica: 009d3e7e8a8b4e2497de3a73d01ced49, State: Running, Role: LEADER
I20240502 21:27:35.544953 25435 consensus_queue.cc:237] T 6bd759d700a24e9f8ad11f243cf7ebfa P 009d3e7e8a8b4e2497de3a73d01ced49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } }
I20240502 21:27:35.547619 23765 catalog_manager.cc:5484] T 6bd759d700a24e9f8ad11f243cf7ebfa P 009d3e7e8a8b4e2497de3a73d01ced49 reported cstate change: term changed from 0 to 1, leader changed from <none> to 009d3e7e8a8b4e2497de3a73d01ced49 (127.15.255.131). New cstate: current_term: 1 leader_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:35.665182 25435 raft_consensus.cc:482] T f84bbe304597450c978ae28e0c4e7a3e P 009d3e7e8a8b4e2497de3a73d01ced49 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:35.665364 25435 raft_consensus.cc:504] T f84bbe304597450c978ae28e0c4e7a3e P 009d3e7e8a8b4e2497de3a73d01ced49 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } }
I20240502 21:27:35.665968 25435 leader_election.cc:288] T f84bbe304597450c978ae28e0c4e7a3e P 009d3e7e8a8b4e2497de3a73d01ced49 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 038c5d9d1d7a49c981c4c34fbc00d9e6 (127.15.255.132:46813), 6c194a6db6c145acafc0f3068b8ce1fd (127.15.255.130:41235)
I20240502 21:27:35.666406 24315 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f84bbe304597450c978ae28e0c4e7a3e" candidate_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" is_pre_election: true
I20240502 21:27:35.666457 24049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f84bbe304597450c978ae28e0c4e7a3e" candidate_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" is_pre_election: true
I20240502 21:27:35.666683 24315 raft_consensus.cc:2454] T f84bbe304597450c978ae28e0c4e7a3e P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 009d3e7e8a8b4e2497de3a73d01ced49 in term 0.
I20240502 21:27:35.666687 24049 raft_consensus.cc:2454] T f84bbe304597450c978ae28e0c4e7a3e P 6c194a6db6c145acafc0f3068b8ce1fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 009d3e7e8a8b4e2497de3a73d01ced49 in term 0.
I20240502 21:27:35.667150 24118 leader_election.cc:302] T f84bbe304597450c978ae28e0c4e7a3e P 009d3e7e8a8b4e2497de3a73d01ced49 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 009d3e7e8a8b4e2497de3a73d01ced49, 6c194a6db6c145acafc0f3068b8ce1fd; no voters:
I20240502 21:27:35.667469 25524 raft_consensus.cc:2781] T f84bbe304597450c978ae28e0c4e7a3e P 009d3e7e8a8b4e2497de3a73d01ced49 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:35.667587 25524 raft_consensus.cc:482] T f84bbe304597450c978ae28e0c4e7a3e P 009d3e7e8a8b4e2497de3a73d01ced49 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:35.667695 25524 raft_consensus.cc:3037] T f84bbe304597450c978ae28e0c4e7a3e P 009d3e7e8a8b4e2497de3a73d01ced49 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:35.669607 25524 raft_consensus.cc:504] T f84bbe304597450c978ae28e0c4e7a3e P 009d3e7e8a8b4e2497de3a73d01ced49 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } }
I20240502 21:27:35.670109 25524 leader_election.cc:288] T f84bbe304597450c978ae28e0c4e7a3e P 009d3e7e8a8b4e2497de3a73d01ced49 [CANDIDATE]: Term 1 election: Requested vote from peers 038c5d9d1d7a49c981c4c34fbc00d9e6 (127.15.255.132:46813), 6c194a6db6c145acafc0f3068b8ce1fd (127.15.255.130:41235)
I20240502 21:27:35.670456 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f84bbe304597450c978ae28e0c4e7a3e" candidate_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6"
I20240502 21:27:35.670539 24049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f84bbe304597450c978ae28e0c4e7a3e" candidate_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c194a6db6c145acafc0f3068b8ce1fd"
I20240502 21:27:35.670667 24316 raft_consensus.cc:3037] T f84bbe304597450c978ae28e0c4e7a3e P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:35.670706 24049 raft_consensus.cc:3037] T f84bbe304597450c978ae28e0c4e7a3e P 6c194a6db6c145acafc0f3068b8ce1fd [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:35.673346 24316 raft_consensus.cc:2454] T f84bbe304597450c978ae28e0c4e7a3e P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 009d3e7e8a8b4e2497de3a73d01ced49 in term 1.
I20240502 21:27:35.673357 24049 raft_consensus.cc:2454] T f84bbe304597450c978ae28e0c4e7a3e P 6c194a6db6c145acafc0f3068b8ce1fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 009d3e7e8a8b4e2497de3a73d01ced49 in term 1.
I20240502 21:27:35.673693 24116 leader_election.cc:302] T f84bbe304597450c978ae28e0c4e7a3e P 009d3e7e8a8b4e2497de3a73d01ced49 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 009d3e7e8a8b4e2497de3a73d01ced49, 038c5d9d1d7a49c981c4c34fbc00d9e6; no voters:
I20240502 21:27:35.673940 25524 raft_consensus.cc:2781] T f84bbe304597450c978ae28e0c4e7a3e P 009d3e7e8a8b4e2497de3a73d01ced49 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:35.674098 25524 raft_consensus.cc:686] T f84bbe304597450c978ae28e0c4e7a3e P 009d3e7e8a8b4e2497de3a73d01ced49 [term 1 LEADER]: Becoming Leader. State: Replica: 009d3e7e8a8b4e2497de3a73d01ced49, State: Running, Role: LEADER
I20240502 21:27:35.674340 25524 consensus_queue.cc:237] T f84bbe304597450c978ae28e0c4e7a3e P 009d3e7e8a8b4e2497de3a73d01ced49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } }
I20240502 21:27:35.676923 23765 catalog_manager.cc:5484] T f84bbe304597450c978ae28e0c4e7a3e P 009d3e7e8a8b4e2497de3a73d01ced49 reported cstate change: term changed from 0 to 1, leader changed from <none> to 009d3e7e8a8b4e2497de3a73d01ced49 (127.15.255.131). New cstate: current_term: 1 leader_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } health_report { overall_health: HEALTHY } } }
I20240502 21:27:35.719739 25429 raft_consensus.cc:482] T 77fc7226878f44d3884ccb1fc41c44a4 P 6c194a6db6c145acafc0f3068b8ce1fd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:35.719879 25429 raft_consensus.cc:504] T 77fc7226878f44d3884ccb1fc41c44a4 P 6c194a6db6c145acafc0f3068b8ce1fd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } }
I20240502 21:27:35.720450 25429 leader_election.cc:288] T 77fc7226878f44d3884ccb1fc41c44a4 P 6c194a6db6c145acafc0f3068b8ce1fd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 038c5d9d1d7a49c981c4c34fbc00d9e6 (127.15.255.132:46813), 6810a866decb4f5089f7d35c870e4204 (127.15.255.129:40435)
I20240502 21:27:35.720837 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77fc7226878f44d3884ccb1fc41c44a4" candidate_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" is_pre_election: true
I20240502 21:27:35.720889 23915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77fc7226878f44d3884ccb1fc41c44a4" candidate_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6810a866decb4f5089f7d35c870e4204" is_pre_election: true
I20240502 21:27:35.721046 24316 raft_consensus.cc:2454] T 77fc7226878f44d3884ccb1fc41c44a4 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6c194a6db6c145acafc0f3068b8ce1fd in term 0.
I20240502 21:27:35.721084 23915 raft_consensus.cc:2454] T 77fc7226878f44d3884ccb1fc41c44a4 P 6810a866decb4f5089f7d35c870e4204 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6c194a6db6c145acafc0f3068b8ce1fd in term 0.
I20240502 21:27:35.721421 23981 leader_election.cc:302] T 77fc7226878f44d3884ccb1fc41c44a4 P 6c194a6db6c145acafc0f3068b8ce1fd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 038c5d9d1d7a49c981c4c34fbc00d9e6, 6810a866decb4f5089f7d35c870e4204, 6c194a6db6c145acafc0f3068b8ce1fd; no voters:
I20240502 21:27:35.721701 25429 raft_consensus.cc:2781] T 77fc7226878f44d3884ccb1fc41c44a4 P 6c194a6db6c145acafc0f3068b8ce1fd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:35.721808 25429 raft_consensus.cc:482] T 77fc7226878f44d3884ccb1fc41c44a4 P 6c194a6db6c145acafc0f3068b8ce1fd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:35.721884 25429 raft_consensus.cc:3037] T 77fc7226878f44d3884ccb1fc41c44a4 P 6c194a6db6c145acafc0f3068b8ce1fd [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:35.723654 25429 raft_consensus.cc:504] T 77fc7226878f44d3884ccb1fc41c44a4 P 6c194a6db6c145acafc0f3068b8ce1fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } }
I20240502 21:27:35.724143 25429 leader_election.cc:288] T 77fc7226878f44d3884ccb1fc41c44a4 P 6c194a6db6c145acafc0f3068b8ce1fd [CANDIDATE]: Term 1 election: Requested vote from peers 038c5d9d1d7a49c981c4c34fbc00d9e6 (127.15.255.132:46813), 6810a866decb4f5089f7d35c870e4204 (127.15.255.129:40435)
I20240502 21:27:35.724529 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77fc7226878f44d3884ccb1fc41c44a4" candidate_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6"
I20240502 21:27:35.724573 23915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77fc7226878f44d3884ccb1fc41c44a4" candidate_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6810a866decb4f5089f7d35c870e4204"
I20240502 21:27:35.724690 24316 raft_consensus.cc:3037] T 77fc7226878f44d3884ccb1fc41c44a4 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:35.724763 23915 raft_consensus.cc:3037] T 77fc7226878f44d3884ccb1fc41c44a4 P 6810a866decb4f5089f7d35c870e4204 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:35.726472 24316 raft_consensus.cc:2454] T 77fc7226878f44d3884ccb1fc41c44a4 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c194a6db6c145acafc0f3068b8ce1fd in term 1.
I20240502 21:27:35.726568 23915 raft_consensus.cc:2454] T 77fc7226878f44d3884ccb1fc41c44a4 P 6810a866decb4f5089f7d35c870e4204 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c194a6db6c145acafc0f3068b8ce1fd in term 1.
I20240502 21:27:35.726838 23981 leader_election.cc:302] T 77fc7226878f44d3884ccb1fc41c44a4 P 6c194a6db6c145acafc0f3068b8ce1fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 038c5d9d1d7a49c981c4c34fbc00d9e6, 6c194a6db6c145acafc0f3068b8ce1fd; no voters:
I20240502 21:27:35.727126 25429 raft_consensus.cc:2781] T 77fc7226878f44d3884ccb1fc41c44a4 P 6c194a6db6c145acafc0f3068b8ce1fd [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:35.727484 25429 raft_consensus.cc:686] T 77fc7226878f44d3884ccb1fc41c44a4 P 6c194a6db6c145acafc0f3068b8ce1fd [term 1 LEADER]: Becoming Leader. State: Replica: 6c194a6db6c145acafc0f3068b8ce1fd, State: Running, Role: LEADER
I20240502 21:27:35.727754 25429 consensus_queue.cc:237] T 77fc7226878f44d3884ccb1fc41c44a4 P 6c194a6db6c145acafc0f3068b8ce1fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } }
I20240502 21:27:35.730516 23765 catalog_manager.cc:5484] T 77fc7226878f44d3884ccb1fc41c44a4 P 6c194a6db6c145acafc0f3068b8ce1fd reported cstate change: term changed from 0 to 1, leader changed from <none> to 6c194a6db6c145acafc0f3068b8ce1fd (127.15.255.130). New cstate: current_term: 1 leader_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:36.004838 25524 consensus_queue.cc:1035] T 6bd759d700a24e9f8ad11f243cf7ebfa P 009d3e7e8a8b4e2497de3a73d01ced49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:36.026167 25524 consensus_queue.cc:1035] T 6bd759d700a24e9f8ad11f243cf7ebfa P 009d3e7e8a8b4e2497de3a73d01ced49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:36.058869 25524 consensus_queue.cc:1035] T f84bbe304597450c978ae28e0c4e7a3e P 009d3e7e8a8b4e2497de3a73d01ced49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:36.062944 25564 consensus_queue.cc:1035] T f84bbe304597450c978ae28e0c4e7a3e P 009d3e7e8a8b4e2497de3a73d01ced49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 21:27:36.095330 25558 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:27:36.095722 25558 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:27:36.107553 25558 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 21:27:36.132196 25429 consensus_queue.cc:1035] T 77fc7226878f44d3884ccb1fc41c44a4 P 6c194a6db6c145acafc0f3068b8ce1fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:36.138496 25429 consensus_queue.cc:1035] T 77fc7226878f44d3884ccb1fc41c44a4 P 6c194a6db6c145acafc0f3068b8ce1fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_7'
INSERT report
rows total: 2000
time total: 192.279 ms
time per row: 0.0961394 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240502 21:27:37.299229 23771 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41618:
name: "rebalance_test_table_8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 21:27:37.308243 24027 tablet_service.cc:1467] Processing CreateTablet for tablet 83b95f3ae7564b81b31a2f5673ee22bb (DEFAULT_TABLE table=rebalance_test_table_8 [id=9a4e7bd31db34cf3aff2242876f0de47]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:37.308741 24027 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83b95f3ae7564b81b31a2f5673ee22bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:37.309124 23894 tablet_service.cc:1467] Processing CreateTablet for tablet 16bb47f03a2f46fdb7344e018f07fdcb (DEFAULT_TABLE table=rebalance_test_table_8 [id=9a4e7bd31db34cf3aff2242876f0de47]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:37.309541 23894 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 16bb47f03a2f46fdb7344e018f07fdcb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:37.309509 24029 tablet_service.cc:1467] Processing CreateTablet for tablet 16bb47f03a2f46fdb7344e018f07fdcb (DEFAULT_TABLE table=rebalance_test_table_8 [id=9a4e7bd31db34cf3aff2242876f0de47]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:37.309883 24029 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 16bb47f03a2f46fdb7344e018f07fdcb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:37.309852 23895 tablet_service.cc:1467] Processing CreateTablet for tablet 83b95f3ae7564b81b31a2f5673ee22bb (DEFAULT_TABLE table=rebalance_test_table_8 [id=9a4e7bd31db34cf3aff2242876f0de47]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:37.310209 23895 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83b95f3ae7564b81b31a2f5673ee22bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:37.313293 24296 tablet_service.cc:1467] Processing CreateTablet for tablet fef55863beb6435dbb25bfb6d71e3cd7 (DEFAULT_TABLE table=rebalance_test_table_8 [id=9a4e7bd31db34cf3aff2242876f0de47]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:37.313724 24296 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fef55863beb6435dbb25bfb6d71e3cd7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:37.313889 24297 tablet_service.cc:1467] Processing CreateTablet for tablet 83b95f3ae7564b81b31a2f5673ee22bb (DEFAULT_TABLE table=rebalance_test_table_8 [id=9a4e7bd31db34cf3aff2242876f0de47]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:37.314229 24297 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83b95f3ae7564b81b31a2f5673ee22bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:37.318210 24162 tablet_service.cc:1467] Processing CreateTablet for tablet 16bb47f03a2f46fdb7344e018f07fdcb (DEFAULT_TABLE table=rebalance_test_table_8 [id=9a4e7bd31db34cf3aff2242876f0de47]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:37.318281 23896 tablet_service.cc:1467] Processing CreateTablet for tablet fef55863beb6435dbb25bfb6d71e3cd7 (DEFAULT_TABLE table=rebalance_test_table_8 [id=9a4e7bd31db34cf3aff2242876f0de47]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:37.318308 24164 tablet_service.cc:1467] Processing CreateTablet for tablet fef55863beb6435dbb25bfb6d71e3cd7 (DEFAULT_TABLE table=rebalance_test_table_8 [id=9a4e7bd31db34cf3aff2242876f0de47]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:37.318648 23896 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fef55863beb6435dbb25bfb6d71e3cd7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:37.318665 24162 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 16bb47f03a2f46fdb7344e018f07fdcb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:37.324662 24164 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fef55863beb6435dbb25bfb6d71e3cd7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:37.332840 25627 tablet_bootstrap.cc:492] T fef55863beb6435dbb25bfb6d71e3cd7 P 009d3e7e8a8b4e2497de3a73d01ced49: Bootstrap starting.
I20240502 21:27:37.334774 25627 tablet_bootstrap.cc:654] T fef55863beb6435dbb25bfb6d71e3cd7 P 009d3e7e8a8b4e2497de3a73d01ced49: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:37.339232 25624 tablet_bootstrap.cc:492] T 83b95f3ae7564b81b31a2f5673ee22bb P 6c194a6db6c145acafc0f3068b8ce1fd: Bootstrap starting.
I20240502 21:27:37.339648 25625 tablet_bootstrap.cc:492] T 16bb47f03a2f46fdb7344e018f07fdcb P 6810a866decb4f5089f7d35c870e4204: Bootstrap starting.
I20240502 21:27:37.341053 25624 tablet_bootstrap.cc:654] T 83b95f3ae7564b81b31a2f5673ee22bb P 6c194a6db6c145acafc0f3068b8ce1fd: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:37.342202 25626 tablet_bootstrap.cc:492] T 83b95f3ae7564b81b31a2f5673ee22bb P 038c5d9d1d7a49c981c4c34fbc00d9e6: Bootstrap starting.
I20240502 21:27:37.342402 25625 tablet_bootstrap.cc:654] T 16bb47f03a2f46fdb7344e018f07fdcb P 6810a866decb4f5089f7d35c870e4204: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:37.343487 25627 tablet_bootstrap.cc:492] T fef55863beb6435dbb25bfb6d71e3cd7 P 009d3e7e8a8b4e2497de3a73d01ced49: No bootstrap required, opened a new log
I20240502 21:27:37.343698 25627 ts_tablet_manager.cc:1367] T fef55863beb6435dbb25bfb6d71e3cd7 P 009d3e7e8a8b4e2497de3a73d01ced49: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.003s
I20240502 21:27:37.344620 25627 raft_consensus.cc:348] T fef55863beb6435dbb25bfb6d71e3cd7 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } }
I20240502 21:27:37.345504 25627 raft_consensus.cc:374] T fef55863beb6435dbb25bfb6d71e3cd7 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:37.345597 25627 raft_consensus.cc:729] T fef55863beb6435dbb25bfb6d71e3cd7 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 009d3e7e8a8b4e2497de3a73d01ced49, State: Initialized, Role: FOLLOWER
I20240502 21:27:37.346210 25624 tablet_bootstrap.cc:492] T 83b95f3ae7564b81b31a2f5673ee22bb P 6c194a6db6c145acafc0f3068b8ce1fd: No bootstrap required, opened a new log
I20240502 21:27:37.346352 25624 ts_tablet_manager.cc:1367] T 83b95f3ae7564b81b31a2f5673ee22bb P 6c194a6db6c145acafc0f3068b8ce1fd: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.003s
I20240502 21:27:37.346549 25627 consensus_queue.cc:260] T fef55863beb6435dbb25bfb6d71e3cd7 P 009d3e7e8a8b4e2497de3a73d01ced49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } }
I20240502 21:27:37.347059 25624 raft_consensus.cc:348] T 83b95f3ae7564b81b31a2f5673ee22bb P 6c194a6db6c145acafc0f3068b8ce1fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } }
I20240502 21:27:37.347314 25624 raft_consensus.cc:374] T 83b95f3ae7564b81b31a2f5673ee22bb P 6c194a6db6c145acafc0f3068b8ce1fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:37.347440 25624 raft_consensus.cc:729] T 83b95f3ae7564b81b31a2f5673ee22bb P 6c194a6db6c145acafc0f3068b8ce1fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c194a6db6c145acafc0f3068b8ce1fd, State: Initialized, Role: FOLLOWER
I20240502 21:27:37.347721 25624 consensus_queue.cc:260] T 83b95f3ae7564b81b31a2f5673ee22bb P 6c194a6db6c145acafc0f3068b8ce1fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } }
I20240502 21:27:37.348326 25625 tablet_bootstrap.cc:492] T 16bb47f03a2f46fdb7344e018f07fdcb P 6810a866decb4f5089f7d35c870e4204: No bootstrap required, opened a new log
I20240502 21:27:37.348500 25625 ts_tablet_manager.cc:1367] T 16bb47f03a2f46fdb7344e018f07fdcb P 6810a866decb4f5089f7d35c870e4204: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.003s
I20240502 21:27:37.349061 25624 ts_tablet_manager.cc:1398] T 83b95f3ae7564b81b31a2f5673ee22bb P 6c194a6db6c145acafc0f3068b8ce1fd: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240502 21:27:37.349373 25624 tablet_bootstrap.cc:492] T 16bb47f03a2f46fdb7344e018f07fdcb P 6c194a6db6c145acafc0f3068b8ce1fd: Bootstrap starting.
I20240502 21:27:37.349315 25625 raft_consensus.cc:348] T 16bb47f03a2f46fdb7344e018f07fdcb P 6810a866decb4f5089f7d35c870e4204 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } }
I20240502 21:27:37.349642 25625 raft_consensus.cc:374] T 16bb47f03a2f46fdb7344e018f07fdcb P 6810a866decb4f5089f7d35c870e4204 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:37.349792 25625 raft_consensus.cc:729] T 16bb47f03a2f46fdb7344e018f07fdcb P 6810a866decb4f5089f7d35c870e4204 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6810a866decb4f5089f7d35c870e4204, State: Initialized, Role: FOLLOWER
I20240502 21:27:37.350106 25625 consensus_queue.cc:260] T 16bb47f03a2f46fdb7344e018f07fdcb P 6810a866decb4f5089f7d35c870e4204 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } }
I20240502 21:27:37.351074 25625 ts_tablet_manager.cc:1398] T 16bb47f03a2f46fdb7344e018f07fdcb P 6810a866decb4f5089f7d35c870e4204: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:27:37.351303 25627 ts_tablet_manager.cc:1398] T fef55863beb6435dbb25bfb6d71e3cd7 P 009d3e7e8a8b4e2497de3a73d01ced49: Time spent starting tablet: real 0.007s user 0.001s sys 0.000s
I20240502 21:27:37.351397 25625 tablet_bootstrap.cc:492] T 83b95f3ae7564b81b31a2f5673ee22bb P 6810a866decb4f5089f7d35c870e4204: Bootstrap starting.
I20240502 21:27:37.352581 25624 tablet_bootstrap.cc:654] T 16bb47f03a2f46fdb7344e018f07fdcb P 6c194a6db6c145acafc0f3068b8ce1fd: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:37.352859 25625 tablet_bootstrap.cc:654] T 83b95f3ae7564b81b31a2f5673ee22bb P 6810a866decb4f5089f7d35c870e4204: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:37.353264 25627 tablet_bootstrap.cc:492] T 16bb47f03a2f46fdb7344e018f07fdcb P 009d3e7e8a8b4e2497de3a73d01ced49: Bootstrap starting.
I20240502 21:27:37.354663 25626 tablet_bootstrap.cc:654] T 83b95f3ae7564b81b31a2f5673ee22bb P 038c5d9d1d7a49c981c4c34fbc00d9e6: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:37.356458 25627 tablet_bootstrap.cc:654] T 16bb47f03a2f46fdb7344e018f07fdcb P 009d3e7e8a8b4e2497de3a73d01ced49: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:37.358778 25626 tablet_bootstrap.cc:492] T 83b95f3ae7564b81b31a2f5673ee22bb P 038c5d9d1d7a49c981c4c34fbc00d9e6: No bootstrap required, opened a new log
I20240502 21:27:37.358922 25626 ts_tablet_manager.cc:1367] T 83b95f3ae7564b81b31a2f5673ee22bb P 038c5d9d1d7a49c981c4c34fbc00d9e6: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.000s
I20240502 21:27:37.359673 25626 raft_consensus.cc:348] T 83b95f3ae7564b81b31a2f5673ee22bb P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } }
I20240502 21:27:37.359933 25626 raft_consensus.cc:374] T 83b95f3ae7564b81b31a2f5673ee22bb P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:37.360024 25626 raft_consensus.cc:729] T 83b95f3ae7564b81b31a2f5673ee22bb P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 038c5d9d1d7a49c981c4c34fbc00d9e6, State: Initialized, Role: FOLLOWER
I20240502 21:27:37.360265 25626 consensus_queue.cc:260] T 83b95f3ae7564b81b31a2f5673ee22bb P 038c5d9d1d7a49c981c4c34fbc00d9e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } }
I20240502 21:27:37.360819 25626 ts_tablet_manager.cc:1398] T 83b95f3ae7564b81b31a2f5673ee22bb P 038c5d9d1d7a49c981c4c34fbc00d9e6: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:27:37.361660 25626 tablet_bootstrap.cc:492] T fef55863beb6435dbb25bfb6d71e3cd7 P 038c5d9d1d7a49c981c4c34fbc00d9e6: Bootstrap starting.
I20240502 21:27:37.363152 25626 tablet_bootstrap.cc:654] T fef55863beb6435dbb25bfb6d71e3cd7 P 038c5d9d1d7a49c981c4c34fbc00d9e6: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:37.367740 25625 tablet_bootstrap.cc:492] T 83b95f3ae7564b81b31a2f5673ee22bb P 6810a866decb4f5089f7d35c870e4204: No bootstrap required, opened a new log
I20240502 21:27:37.367967 25625 ts_tablet_manager.cc:1367] T 83b95f3ae7564b81b31a2f5673ee22bb P 6810a866decb4f5089f7d35c870e4204: Time spent bootstrapping tablet: real 0.017s user 0.007s sys 0.007s
I20240502 21:27:37.368108 25626 tablet_bootstrap.cc:492] T fef55863beb6435dbb25bfb6d71e3cd7 P 038c5d9d1d7a49c981c4c34fbc00d9e6: No bootstrap required, opened a new log
I20240502 21:27:37.368225 25627 tablet_bootstrap.cc:492] T 16bb47f03a2f46fdb7344e018f07fdcb P 009d3e7e8a8b4e2497de3a73d01ced49: No bootstrap required, opened a new log
I20240502 21:27:37.368301 25626 ts_tablet_manager.cc:1367] T fef55863beb6435dbb25bfb6d71e3cd7 P 038c5d9d1d7a49c981c4c34fbc00d9e6: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20240502 21:27:37.368441 25627 ts_tablet_manager.cc:1367] T 16bb47f03a2f46fdb7344e018f07fdcb P 009d3e7e8a8b4e2497de3a73d01ced49: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.009s
I20240502 21:27:37.369089 25624 tablet_bootstrap.cc:492] T 16bb47f03a2f46fdb7344e018f07fdcb P 6c194a6db6c145acafc0f3068b8ce1fd: No bootstrap required, opened a new log
I20240502 21:27:37.368961 25625 raft_consensus.cc:348] T 83b95f3ae7564b81b31a2f5673ee22bb P 6810a866decb4f5089f7d35c870e4204 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } }
I20240502 21:27:37.369084 25626 raft_consensus.cc:348] T fef55863beb6435dbb25bfb6d71e3cd7 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } }
I20240502 21:27:37.369251 25625 raft_consensus.cc:374] T 83b95f3ae7564b81b31a2f5673ee22bb P 6810a866decb4f5089f7d35c870e4204 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:37.369160 25627 raft_consensus.cc:348] T 16bb47f03a2f46fdb7344e018f07fdcb P 009d3e7e8a8b4e2497de3a73d01ced49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } }
I20240502 21:27:37.369305 25624 ts_tablet_manager.cc:1367] T 16bb47f03a2f46fdb7344e018f07fdcb P 6c194a6db6c145acafc0f3068b8ce1fd: Time spent bootstrapping tablet: real 0.020s user 0.001s sys 0.008s
I20240502 21:27:37.369364 25627 raft_consensus.cc:374] T 16bb47f03a2f46fdb7344e018f07fdcb P 009d3e7e8a8b4e2497de3a73d01ced49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:37.369367 25626 raft_consensus.cc:374] T fef55863beb6435dbb25bfb6d71e3cd7 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:37.369391 25625 raft_consensus.cc:729] T 83b95f3ae7564b81b31a2f5673ee22bb P 6810a866decb4f5089f7d35c870e4204 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6810a866decb4f5089f7d35c870e4204, State: Initialized, Role: FOLLOWER
I20240502 21:27:37.369450 25626 raft_consensus.cc:729] T fef55863beb6435dbb25bfb6d71e3cd7 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 038c5d9d1d7a49c981c4c34fbc00d9e6, State: Initialized, Role: FOLLOWER
I20240502 21:27:37.369468 25627 raft_consensus.cc:729] T 16bb47f03a2f46fdb7344e018f07fdcb P 009d3e7e8a8b4e2497de3a73d01ced49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 009d3e7e8a8b4e2497de3a73d01ced49, State: Initialized, Role: FOLLOWER
I20240502 21:27:37.369702 25627 consensus_queue.cc:260] T 16bb47f03a2f46fdb7344e018f07fdcb P 009d3e7e8a8b4e2497de3a73d01ced49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } }
I20240502 21:27:37.369719 25625 consensus_queue.cc:260] T 83b95f3ae7564b81b31a2f5673ee22bb P 6810a866decb4f5089f7d35c870e4204 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } }
I20240502 21:27:37.369800 25626 consensus_queue.cc:260] T fef55863beb6435dbb25bfb6d71e3cd7 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } }
I20240502 21:27:37.370328 25624 raft_consensus.cc:348] T 16bb47f03a2f46fdb7344e018f07fdcb P 6c194a6db6c145acafc0f3068b8ce1fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } }
I20240502 21:27:37.370502 25625 ts_tablet_manager.cc:1398] T 83b95f3ae7564b81b31a2f5673ee22bb P 6810a866decb4f5089f7d35c870e4204: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:27:37.370596 25624 raft_consensus.cc:374] T 16bb47f03a2f46fdb7344e018f07fdcb P 6c194a6db6c145acafc0f3068b8ce1fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:37.371104 25627 ts_tablet_manager.cc:1398] T 16bb47f03a2f46fdb7344e018f07fdcb P 009d3e7e8a8b4e2497de3a73d01ced49: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240502 21:27:37.371202 25626 ts_tablet_manager.cc:1398] T fef55863beb6435dbb25bfb6d71e3cd7 P 038c5d9d1d7a49c981c4c34fbc00d9e6: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 21:27:37.371299 25624 raft_consensus.cc:729] T 16bb47f03a2f46fdb7344e018f07fdcb P 6c194a6db6c145acafc0f3068b8ce1fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c194a6db6c145acafc0f3068b8ce1fd, State: Initialized, Role: FOLLOWER
I20240502 21:27:37.371594 25624 consensus_queue.cc:260] T 16bb47f03a2f46fdb7344e018f07fdcb P 6c194a6db6c145acafc0f3068b8ce1fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } }
I20240502 21:27:37.372171 25624 ts_tablet_manager.cc:1398] T 16bb47f03a2f46fdb7344e018f07fdcb P 6c194a6db6c145acafc0f3068b8ce1fd: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240502 21:27:37.374100 25625 tablet_bootstrap.cc:492] T fef55863beb6435dbb25bfb6d71e3cd7 P 6810a866decb4f5089f7d35c870e4204: Bootstrap starting.
I20240502 21:27:37.376065 25625 tablet_bootstrap.cc:654] T fef55863beb6435dbb25bfb6d71e3cd7 P 6810a866decb4f5089f7d35c870e4204: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:37.377965 25625 tablet_bootstrap.cc:492] T fef55863beb6435dbb25bfb6d71e3cd7 P 6810a866decb4f5089f7d35c870e4204: No bootstrap required, opened a new log
I20240502 21:27:37.378099 25625 ts_tablet_manager.cc:1367] T fef55863beb6435dbb25bfb6d71e3cd7 P 6810a866decb4f5089f7d35c870e4204: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240502 21:27:37.378811 25625 raft_consensus.cc:348] T fef55863beb6435dbb25bfb6d71e3cd7 P 6810a866decb4f5089f7d35c870e4204 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } }
I20240502 21:27:37.379009 25625 raft_consensus.cc:374] T fef55863beb6435dbb25bfb6d71e3cd7 P 6810a866decb4f5089f7d35c870e4204 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:37.379115 25625 raft_consensus.cc:729] T fef55863beb6435dbb25bfb6d71e3cd7 P 6810a866decb4f5089f7d35c870e4204 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6810a866decb4f5089f7d35c870e4204, State: Initialized, Role: FOLLOWER
I20240502 21:27:37.379339 25625 consensus_queue.cc:260] T fef55863beb6435dbb25bfb6d71e3cd7 P 6810a866decb4f5089f7d35c870e4204 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } }
I20240502 21:27:37.379868 25625 ts_tablet_manager.cc:1398] T fef55863beb6435dbb25bfb6d71e3cd7 P 6810a866decb4f5089f7d35c870e4204: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 21:27:37.381938 25524 raft_consensus.cc:482] T 16bb47f03a2f46fdb7344e018f07fdcb P 009d3e7e8a8b4e2497de3a73d01ced49 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:37.382123 25524 raft_consensus.cc:504] T 16bb47f03a2f46fdb7344e018f07fdcb P 009d3e7e8a8b4e2497de3a73d01ced49 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } }
I20240502 21:27:37.382867 25524 leader_election.cc:288] T 16bb47f03a2f46fdb7344e018f07fdcb P 009d3e7e8a8b4e2497de3a73d01ced49 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6810a866decb4f5089f7d35c870e4204 (127.15.255.129:40435), 6c194a6db6c145acafc0f3068b8ce1fd (127.15.255.130:41235)
I20240502 21:27:37.383297 24049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16bb47f03a2f46fdb7344e018f07fdcb" candidate_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" is_pre_election: true
I20240502 21:27:37.383498 24049 raft_consensus.cc:2454] T 16bb47f03a2f46fdb7344e018f07fdcb P 6c194a6db6c145acafc0f3068b8ce1fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 009d3e7e8a8b4e2497de3a73d01ced49 in term 0.
I20240502 21:27:37.383688 23915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16bb47f03a2f46fdb7344e018f07fdcb" candidate_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6810a866decb4f5089f7d35c870e4204" is_pre_election: true
I20240502 21:27:37.383903 23915 raft_consensus.cc:2454] T 16bb47f03a2f46fdb7344e018f07fdcb P 6810a866decb4f5089f7d35c870e4204 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 009d3e7e8a8b4e2497de3a73d01ced49 in term 0.
I20240502 21:27:37.383919 24118 leader_election.cc:302] T 16bb47f03a2f46fdb7344e018f07fdcb P 009d3e7e8a8b4e2497de3a73d01ced49 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 009d3e7e8a8b4e2497de3a73d01ced49, 6c194a6db6c145acafc0f3068b8ce1fd; no voters:
I20240502 21:27:37.384454 25524 raft_consensus.cc:2781] T 16bb47f03a2f46fdb7344e018f07fdcb P 009d3e7e8a8b4e2497de3a73d01ced49 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:37.384563 25524 raft_consensus.cc:482] T 16bb47f03a2f46fdb7344e018f07fdcb P 009d3e7e8a8b4e2497de3a73d01ced49 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:37.384689 25524 raft_consensus.cc:3037] T 16bb47f03a2f46fdb7344e018f07fdcb P 009d3e7e8a8b4e2497de3a73d01ced49 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:37.386658 25524 raft_consensus.cc:504] T 16bb47f03a2f46fdb7344e018f07fdcb P 009d3e7e8a8b4e2497de3a73d01ced49 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } }
I20240502 21:27:37.387174 25524 leader_election.cc:288] T 16bb47f03a2f46fdb7344e018f07fdcb P 009d3e7e8a8b4e2497de3a73d01ced49 [CANDIDATE]: Term 1 election: Requested vote from peers 6810a866decb4f5089f7d35c870e4204 (127.15.255.129:40435), 6c194a6db6c145acafc0f3068b8ce1fd (127.15.255.130:41235)
I20240502 21:27:37.387586 24049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16bb47f03a2f46fdb7344e018f07fdcb" candidate_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c194a6db6c145acafc0f3068b8ce1fd"
I20240502 21:27:37.387645 23916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16bb47f03a2f46fdb7344e018f07fdcb" candidate_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6810a866decb4f5089f7d35c870e4204"
I20240502 21:27:37.387779 24049 raft_consensus.cc:3037] T 16bb47f03a2f46fdb7344e018f07fdcb P 6c194a6db6c145acafc0f3068b8ce1fd [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:37.387801 23916 raft_consensus.cc:3037] T 16bb47f03a2f46fdb7344e018f07fdcb P 6810a866decb4f5089f7d35c870e4204 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:37.390308 23916 raft_consensus.cc:2454] T 16bb47f03a2f46fdb7344e018f07fdcb P 6810a866decb4f5089f7d35c870e4204 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 009d3e7e8a8b4e2497de3a73d01ced49 in term 1.
I20240502 21:27:37.390326 24049 raft_consensus.cc:2454] T 16bb47f03a2f46fdb7344e018f07fdcb P 6c194a6db6c145acafc0f3068b8ce1fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 009d3e7e8a8b4e2497de3a73d01ced49 in term 1.
I20240502 21:27:37.390666 24118 leader_election.cc:302] T 16bb47f03a2f46fdb7344e018f07fdcb P 009d3e7e8a8b4e2497de3a73d01ced49 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 009d3e7e8a8b4e2497de3a73d01ced49, 6810a866decb4f5089f7d35c870e4204; no voters:
I20240502 21:27:37.390950 25524 raft_consensus.cc:2781] T 16bb47f03a2f46fdb7344e018f07fdcb P 009d3e7e8a8b4e2497de3a73d01ced49 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:37.391115 25524 raft_consensus.cc:686] T 16bb47f03a2f46fdb7344e018f07fdcb P 009d3e7e8a8b4e2497de3a73d01ced49 [term 1 LEADER]: Becoming Leader. State: Replica: 009d3e7e8a8b4e2497de3a73d01ced49, State: Running, Role: LEADER
I20240502 21:27:37.391350 25524 consensus_queue.cc:237] T 16bb47f03a2f46fdb7344e018f07fdcb P 009d3e7e8a8b4e2497de3a73d01ced49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } }
I20240502 21:27:37.393900 23768 catalog_manager.cc:5484] T 16bb47f03a2f46fdb7344e018f07fdcb P 009d3e7e8a8b4e2497de3a73d01ced49 reported cstate change: term changed from 0 to 1, leader changed from <none> to 009d3e7e8a8b4e2497de3a73d01ced49 (127.15.255.131). New cstate: current_term: 1 leader_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:37.425496 25390 raft_consensus.cc:482] T fef55863beb6435dbb25bfb6d71e3cd7 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:37.425630 25390 raft_consensus.cc:504] T fef55863beb6435dbb25bfb6d71e3cd7 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } }
I20240502 21:27:37.426180 25390 leader_election.cc:288] T fef55863beb6435dbb25bfb6d71e3cd7 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 009d3e7e8a8b4e2497de3a73d01ced49 (127.15.255.131:37895), 6810a866decb4f5089f7d35c870e4204 (127.15.255.129:40435)
I20240502 21:27:37.426519 24185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fef55863beb6435dbb25bfb6d71e3cd7" candidate_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" is_pre_election: true
I20240502 21:27:37.426643 23916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fef55863beb6435dbb25bfb6d71e3cd7" candidate_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6810a866decb4f5089f7d35c870e4204" is_pre_election: true
I20240502 21:27:37.426749 24185 raft_consensus.cc:2454] T fef55863beb6435dbb25bfb6d71e3cd7 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 038c5d9d1d7a49c981c4c34fbc00d9e6 in term 0.
I20240502 21:27:37.426879 23916 raft_consensus.cc:2454] T fef55863beb6435dbb25bfb6d71e3cd7 P 6810a866decb4f5089f7d35c870e4204 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 038c5d9d1d7a49c981c4c34fbc00d9e6 in term 0.
I20240502 21:27:37.427150 24252 leader_election.cc:302] T fef55863beb6435dbb25bfb6d71e3cd7 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 009d3e7e8a8b4e2497de3a73d01ced49, 038c5d9d1d7a49c981c4c34fbc00d9e6; no voters:
I20240502 21:27:37.427404 25390 raft_consensus.cc:2781] T fef55863beb6435dbb25bfb6d71e3cd7 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:37.427534 25390 raft_consensus.cc:482] T fef55863beb6435dbb25bfb6d71e3cd7 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:37.427635 25390 raft_consensus.cc:3037] T fef55863beb6435dbb25bfb6d71e3cd7 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:37.429729 25390 raft_consensus.cc:504] T fef55863beb6435dbb25bfb6d71e3cd7 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } }
I20240502 21:27:37.430238 25390 leader_election.cc:288] T fef55863beb6435dbb25bfb6d71e3cd7 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [CANDIDATE]: Term 1 election: Requested vote from peers 009d3e7e8a8b4e2497de3a73d01ced49 (127.15.255.131:37895), 6810a866decb4f5089f7d35c870e4204 (127.15.255.129:40435)
I20240502 21:27:37.430554 24185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fef55863beb6435dbb25bfb6d71e3cd7" candidate_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "009d3e7e8a8b4e2497de3a73d01ced49"
I20240502 21:27:37.430660 23916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fef55863beb6435dbb25bfb6d71e3cd7" candidate_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6810a866decb4f5089f7d35c870e4204"
I20240502 21:27:37.430732 24185 raft_consensus.cc:3037] T fef55863beb6435dbb25bfb6d71e3cd7 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:37.430879 23916 raft_consensus.cc:3037] T fef55863beb6435dbb25bfb6d71e3cd7 P 6810a866decb4f5089f7d35c870e4204 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:37.432826 24185 raft_consensus.cc:2454] T fef55863beb6435dbb25bfb6d71e3cd7 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 038c5d9d1d7a49c981c4c34fbc00d9e6 in term 1.
I20240502 21:27:37.433156 23916 raft_consensus.cc:2454] T fef55863beb6435dbb25bfb6d71e3cd7 P 6810a866decb4f5089f7d35c870e4204 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 038c5d9d1d7a49c981c4c34fbc00d9e6 in term 1.
I20240502 21:27:37.433159 24252 leader_election.cc:302] T fef55863beb6435dbb25bfb6d71e3cd7 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 009d3e7e8a8b4e2497de3a73d01ced49, 038c5d9d1d7a49c981c4c34fbc00d9e6; no voters:
I20240502 21:27:37.433421 25390 raft_consensus.cc:2781] T fef55863beb6435dbb25bfb6d71e3cd7 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:37.433760 25390 raft_consensus.cc:686] T fef55863beb6435dbb25bfb6d71e3cd7 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 1 LEADER]: Becoming Leader. State: Replica: 038c5d9d1d7a49c981c4c34fbc00d9e6, State: Running, Role: LEADER
I20240502 21:27:37.434064 25390 consensus_queue.cc:237] T fef55863beb6435dbb25bfb6d71e3cd7 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } }
I20240502 21:27:37.436694 23768 catalog_manager.cc:5484] T fef55863beb6435dbb25bfb6d71e3cd7 P 038c5d9d1d7a49c981c4c34fbc00d9e6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 038c5d9d1d7a49c981c4c34fbc00d9e6 (127.15.255.132). New cstate: current_term: 1 leader_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:37.513104 25472 raft_consensus.cc:482] T 83b95f3ae7564b81b31a2f5673ee22bb P 6810a866decb4f5089f7d35c870e4204 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:37.513268 25472 raft_consensus.cc:504] T 83b95f3ae7564b81b31a2f5673ee22bb P 6810a866decb4f5089f7d35c870e4204 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } }
I20240502 21:27:37.513932 25472 leader_election.cc:288] T 83b95f3ae7564b81b31a2f5673ee22bb P 6810a866decb4f5089f7d35c870e4204 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6c194a6db6c145acafc0f3068b8ce1fd (127.15.255.130:41235), 038c5d9d1d7a49c981c4c34fbc00d9e6 (127.15.255.132:46813)
I20240502 21:27:37.514338 24049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83b95f3ae7564b81b31a2f5673ee22bb" candidate_uuid: "6810a866decb4f5089f7d35c870e4204" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" is_pre_election: true
I20240502 21:27:37.514420 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83b95f3ae7564b81b31a2f5673ee22bb" candidate_uuid: "6810a866decb4f5089f7d35c870e4204" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" is_pre_election: true
I20240502 21:27:37.514596 24049 raft_consensus.cc:2454] T 83b95f3ae7564b81b31a2f5673ee22bb P 6c194a6db6c145acafc0f3068b8ce1fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6810a866decb4f5089f7d35c870e4204 in term 0.
I20240502 21:27:37.514647 24316 raft_consensus.cc:2454] T 83b95f3ae7564b81b31a2f5673ee22bb P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6810a866decb4f5089f7d35c870e4204 in term 0.
I20240502 21:27:37.515024 23851 leader_election.cc:302] T 83b95f3ae7564b81b31a2f5673ee22bb P 6810a866decb4f5089f7d35c870e4204 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6810a866decb4f5089f7d35c870e4204, 6c194a6db6c145acafc0f3068b8ce1fd; no voters:
I20240502 21:27:37.515301 25472 raft_consensus.cc:2781] T 83b95f3ae7564b81b31a2f5673ee22bb P 6810a866decb4f5089f7d35c870e4204 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:37.515412 25472 raft_consensus.cc:482] T 83b95f3ae7564b81b31a2f5673ee22bb P 6810a866decb4f5089f7d35c870e4204 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:37.515494 25472 raft_consensus.cc:3037] T 83b95f3ae7564b81b31a2f5673ee22bb P 6810a866decb4f5089f7d35c870e4204 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:37.517515 25472 raft_consensus.cc:504] T 83b95f3ae7564b81b31a2f5673ee22bb P 6810a866decb4f5089f7d35c870e4204 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } }
I20240502 21:27:37.518054 25472 leader_election.cc:288] T 83b95f3ae7564b81b31a2f5673ee22bb P 6810a866decb4f5089f7d35c870e4204 [CANDIDATE]: Term 1 election: Requested vote from peers 6c194a6db6c145acafc0f3068b8ce1fd (127.15.255.130:41235), 038c5d9d1d7a49c981c4c34fbc00d9e6 (127.15.255.132:46813)
I20240502 21:27:37.518474 24049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83b95f3ae7564b81b31a2f5673ee22bb" candidate_uuid: "6810a866decb4f5089f7d35c870e4204" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c194a6db6c145acafc0f3068b8ce1fd"
I20240502 21:27:37.518446 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83b95f3ae7564b81b31a2f5673ee22bb" candidate_uuid: "6810a866decb4f5089f7d35c870e4204" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6"
I20240502 21:27:37.518664 24049 raft_consensus.cc:3037] T 83b95f3ae7564b81b31a2f5673ee22bb P 6c194a6db6c145acafc0f3068b8ce1fd [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:37.518675 24316 raft_consensus.cc:3037] T 83b95f3ae7564b81b31a2f5673ee22bb P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:37.521332 24316 raft_consensus.cc:2454] T 83b95f3ae7564b81b31a2f5673ee22bb P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6810a866decb4f5089f7d35c870e4204 in term 1.
I20240502 21:27:37.521464 24049 raft_consensus.cc:2454] T 83b95f3ae7564b81b31a2f5673ee22bb P 6c194a6db6c145acafc0f3068b8ce1fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6810a866decb4f5089f7d35c870e4204 in term 1.
I20240502 21:27:37.521770 23849 leader_election.cc:302] T 83b95f3ae7564b81b31a2f5673ee22bb P 6810a866decb4f5089f7d35c870e4204 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 038c5d9d1d7a49c981c4c34fbc00d9e6, 6810a866decb4f5089f7d35c870e4204; no voters:
I20240502 21:27:37.522037 25472 raft_consensus.cc:2781] T 83b95f3ae7564b81b31a2f5673ee22bb P 6810a866decb4f5089f7d35c870e4204 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:37.522393 25472 raft_consensus.cc:686] T 83b95f3ae7564b81b31a2f5673ee22bb P 6810a866decb4f5089f7d35c870e4204 [term 1 LEADER]: Becoming Leader. State: Replica: 6810a866decb4f5089f7d35c870e4204, State: Running, Role: LEADER
I20240502 21:27:37.522691 25472 consensus_queue.cc:237] T 83b95f3ae7564b81b31a2f5673ee22bb P 6810a866decb4f5089f7d35c870e4204 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } }
I20240502 21:27:37.525385 23768 catalog_manager.cc:5484] T 83b95f3ae7564b81b31a2f5673ee22bb P 6810a866decb4f5089f7d35c870e4204 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6810a866decb4f5089f7d35c870e4204 (127.15.255.129). New cstate: current_term: 1 leader_uuid: "6810a866decb4f5089f7d35c870e4204" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:37.845330 25390 consensus_queue.cc:1035] T fef55863beb6435dbb25bfb6d71e3cd7 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:37.851909 25390 consensus_queue.cc:1035] T fef55863beb6435dbb25bfb6d71e3cd7 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 21:27:37.894188 25644 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:27:37.894429 25644 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:27:37.901463 25644 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_8'
I20240502 21:27:37.932287 25524 consensus_queue.cc:1035] T 16bb47f03a2f46fdb7344e018f07fdcb P 009d3e7e8a8b4e2497de3a73d01ced49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:37.933163 25524 consensus_queue.cc:1035] T 16bb47f03a2f46fdb7344e018f07fdcb P 009d3e7e8a8b4e2497de3a73d01ced49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:38.035245 25472 consensus_queue.cc:1035] T 83b95f3ae7564b81b31a2f5673ee22bb P 6810a866decb4f5089f7d35c870e4204 [LEADER]: Connected to new peer: Peer: permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:38.040791 25687 consensus_queue.cc:1035] T 83b95f3ae7564b81b31a2f5673ee22bb P 6810a866decb4f5089f7d35c870e4204 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 230.676 ms
time per row: 0.115338 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240502 21:27:39.149171 23770 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41618:
name: "rebalance_test_table_9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 21:27:39.158828 24164 tablet_service.cc:1467] Processing CreateTablet for tablet d5697d8b06094211b261f144de9d46f8 (DEFAULT_TABLE table=rebalance_test_table_9 [id=188ff51c46164f47803c423463c2e6e2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:39.159051 24027 tablet_service.cc:1467] Processing CreateTablet for tablet d5697d8b06094211b261f144de9d46f8 (DEFAULT_TABLE table=rebalance_test_table_9 [id=188ff51c46164f47803c423463c2e6e2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:39.159283 24164 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d5697d8b06094211b261f144de9d46f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:39.159436 24027 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d5697d8b06094211b261f144de9d46f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:39.159950 24296 tablet_service.cc:1467] Processing CreateTablet for tablet fcc9cbeae6f343af9f84c73349f3dadb (DEFAULT_TABLE table=rebalance_test_table_9 [id=188ff51c46164f47803c423463c2e6e2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:39.160390 24296 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fcc9cbeae6f343af9f84c73349f3dadb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:39.160315 24297 tablet_service.cc:1467] Processing CreateTablet for tablet d5697d8b06094211b261f144de9d46f8 (DEFAULT_TABLE table=rebalance_test_table_9 [id=188ff51c46164f47803c423463c2e6e2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:39.160708 24297 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d5697d8b06094211b261f144de9d46f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:39.162071 24028 tablet_service.cc:1467] Processing CreateTablet for tablet 8eba3872a77046ecb9c9477d4cc0e18b (DEFAULT_TABLE table=rebalance_test_table_9 [id=188ff51c46164f47803c423463c2e6e2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:39.162447 24028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8eba3872a77046ecb9c9477d4cc0e18b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:39.164381 24029 tablet_service.cc:1467] Processing CreateTablet for tablet fcc9cbeae6f343af9f84c73349f3dadb (DEFAULT_TABLE table=rebalance_test_table_9 [id=188ff51c46164f47803c423463c2e6e2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:39.164741 24029 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fcc9cbeae6f343af9f84c73349f3dadb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:39.164984 24295 tablet_service.cc:1467] Processing CreateTablet for tablet 8eba3872a77046ecb9c9477d4cc0e18b (DEFAULT_TABLE table=rebalance_test_table_9 [id=188ff51c46164f47803c423463c2e6e2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:39.165356 24295 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8eba3872a77046ecb9c9477d4cc0e18b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:39.168810 23894 tablet_service.cc:1467] Processing CreateTablet for tablet 8eba3872a77046ecb9c9477d4cc0e18b (DEFAULT_TABLE table=rebalance_test_table_9 [id=188ff51c46164f47803c423463c2e6e2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:39.169219 23894 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8eba3872a77046ecb9c9477d4cc0e18b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:39.173425 24162 tablet_service.cc:1467] Processing CreateTablet for tablet fcc9cbeae6f343af9f84c73349f3dadb (DEFAULT_TABLE table=rebalance_test_table_9 [id=188ff51c46164f47803c423463c2e6e2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:27:39.173835 24162 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fcc9cbeae6f343af9f84c73349f3dadb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:39.185104 25716 tablet_bootstrap.cc:492] T d5697d8b06094211b261f144de9d46f8 P 6c194a6db6c145acafc0f3068b8ce1fd: Bootstrap starting.
I20240502 21:27:39.186836 25716 tablet_bootstrap.cc:654] T d5697d8b06094211b261f144de9d46f8 P 6c194a6db6c145acafc0f3068b8ce1fd: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:39.189872 25717 tablet_bootstrap.cc:492] T fcc9cbeae6f343af9f84c73349f3dadb P 009d3e7e8a8b4e2497de3a73d01ced49: Bootstrap starting.
I20240502 21:27:39.191597 25717 tablet_bootstrap.cc:654] T fcc9cbeae6f343af9f84c73349f3dadb P 009d3e7e8a8b4e2497de3a73d01ced49: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:39.191962 25715 tablet_bootstrap.cc:492] T d5697d8b06094211b261f144de9d46f8 P 038c5d9d1d7a49c981c4c34fbc00d9e6: Bootstrap starting.
I20240502 21:27:39.193627 25715 tablet_bootstrap.cc:654] T d5697d8b06094211b261f144de9d46f8 P 038c5d9d1d7a49c981c4c34fbc00d9e6: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:39.195676 25718 tablet_bootstrap.cc:492] T 8eba3872a77046ecb9c9477d4cc0e18b P 6810a866decb4f5089f7d35c870e4204: Bootstrap starting.
I20240502 21:27:39.197451 25718 tablet_bootstrap.cc:654] T 8eba3872a77046ecb9c9477d4cc0e18b P 6810a866decb4f5089f7d35c870e4204: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:39.199294 25716 tablet_bootstrap.cc:492] T d5697d8b06094211b261f144de9d46f8 P 6c194a6db6c145acafc0f3068b8ce1fd: No bootstrap required, opened a new log
I20240502 21:27:39.199452 25716 ts_tablet_manager.cc:1367] T d5697d8b06094211b261f144de9d46f8 P 6c194a6db6c145acafc0f3068b8ce1fd: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.004s
I20240502 21:27:39.199592 25717 tablet_bootstrap.cc:492] T fcc9cbeae6f343af9f84c73349f3dadb P 009d3e7e8a8b4e2497de3a73d01ced49: No bootstrap required, opened a new log
I20240502 21:27:39.199636 25715 tablet_bootstrap.cc:492] T d5697d8b06094211b261f144de9d46f8 P 038c5d9d1d7a49c981c4c34fbc00d9e6: No bootstrap required, opened a new log
I20240502 21:27:39.199746 25715 ts_tablet_manager.cc:1367] T d5697d8b06094211b261f144de9d46f8 P 038c5d9d1d7a49c981c4c34fbc00d9e6: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.000s
I20240502 21:27:39.199740 25717 ts_tablet_manager.cc:1367] T fcc9cbeae6f343af9f84c73349f3dadb P 009d3e7e8a8b4e2497de3a73d01ced49: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20240502 21:27:39.200168 25716 raft_consensus.cc:348] T d5697d8b06094211b261f144de9d46f8 P 6c194a6db6c145acafc0f3068b8ce1fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } }
I20240502 21:27:39.200448 25716 raft_consensus.cc:374] T d5697d8b06094211b261f144de9d46f8 P 6c194a6db6c145acafc0f3068b8ce1fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:39.200445 25717 raft_consensus.cc:348] T fcc9cbeae6f343af9f84c73349f3dadb P 009d3e7e8a8b4e2497de3a73d01ced49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } }
I20240502 21:27:39.200505 25715 raft_consensus.cc:348] T d5697d8b06094211b261f144de9d46f8 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } }
I20240502 21:27:39.200588 25716 raft_consensus.cc:729] T d5697d8b06094211b261f144de9d46f8 P 6c194a6db6c145acafc0f3068b8ce1fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c194a6db6c145acafc0f3068b8ce1fd, State: Initialized, Role: FOLLOWER
I20240502 21:27:39.200696 25717 raft_consensus.cc:374] T fcc9cbeae6f343af9f84c73349f3dadb P 009d3e7e8a8b4e2497de3a73d01ced49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:39.200691 25715 raft_consensus.cc:374] T d5697d8b06094211b261f144de9d46f8 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:39.200798 25717 raft_consensus.cc:729] T fcc9cbeae6f343af9f84c73349f3dadb P 009d3e7e8a8b4e2497de3a73d01ced49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 009d3e7e8a8b4e2497de3a73d01ced49, State: Initialized, Role: FOLLOWER
I20240502 21:27:39.200829 25715 raft_consensus.cc:729] T d5697d8b06094211b261f144de9d46f8 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 038c5d9d1d7a49c981c4c34fbc00d9e6, State: Initialized, Role: FOLLOWER
I20240502 21:27:39.200831 25716 consensus_queue.cc:260] T d5697d8b06094211b261f144de9d46f8 P 6c194a6db6c145acafc0f3068b8ce1fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } }
I20240502 21:27:39.201030 25717 consensus_queue.cc:260] T fcc9cbeae6f343af9f84c73349f3dadb P 009d3e7e8a8b4e2497de3a73d01ced49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } }
I20240502 21:27:39.201028 25715 consensus_queue.cc:260] T d5697d8b06094211b261f144de9d46f8 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } }
I20240502 21:27:39.201766 25717 ts_tablet_manager.cc:1398] T fcc9cbeae6f343af9f84c73349f3dadb P 009d3e7e8a8b4e2497de3a73d01ced49: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:27:39.202090 25717 tablet_bootstrap.cc:492] T d5697d8b06094211b261f144de9d46f8 P 009d3e7e8a8b4e2497de3a73d01ced49: Bootstrap starting.
I20240502 21:27:39.201794 25715 ts_tablet_manager.cc:1398] T d5697d8b06094211b261f144de9d46f8 P 038c5d9d1d7a49c981c4c34fbc00d9e6: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 21:27:39.202971 25718 tablet_bootstrap.cc:492] T 8eba3872a77046ecb9c9477d4cc0e18b P 6810a866decb4f5089f7d35c870e4204: No bootstrap required, opened a new log
I20240502 21:27:39.203226 25716 ts_tablet_manager.cc:1398] T d5697d8b06094211b261f144de9d46f8 P 6c194a6db6c145acafc0f3068b8ce1fd: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20240502 21:27:39.203490 25715 tablet_bootstrap.cc:492] T 8eba3872a77046ecb9c9477d4cc0e18b P 038c5d9d1d7a49c981c4c34fbc00d9e6: Bootstrap starting.
I20240502 21:27:39.203574 25716 tablet_bootstrap.cc:492] T 8eba3872a77046ecb9c9477d4cc0e18b P 6c194a6db6c145acafc0f3068b8ce1fd: Bootstrap starting.
I20240502 21:27:39.205081 25715 tablet_bootstrap.cc:654] T 8eba3872a77046ecb9c9477d4cc0e18b P 038c5d9d1d7a49c981c4c34fbc00d9e6: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:39.205982 25718 ts_tablet_manager.cc:1367] T 8eba3872a77046ecb9c9477d4cc0e18b P 6810a866decb4f5089f7d35c870e4204: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.000s
I20240502 21:27:39.206856 25718 raft_consensus.cc:348] T 8eba3872a77046ecb9c9477d4cc0e18b P 6810a866decb4f5089f7d35c870e4204 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } }
I20240502 21:27:39.207118 25716 tablet_bootstrap.cc:654] T 8eba3872a77046ecb9c9477d4cc0e18b P 6c194a6db6c145acafc0f3068b8ce1fd: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:39.207120 25718 raft_consensus.cc:374] T 8eba3872a77046ecb9c9477d4cc0e18b P 6810a866decb4f5089f7d35c870e4204 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:39.207219 25718 raft_consensus.cc:729] T 8eba3872a77046ecb9c9477d4cc0e18b P 6810a866decb4f5089f7d35c870e4204 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6810a866decb4f5089f7d35c870e4204, State: Initialized, Role: FOLLOWER
I20240502 21:27:39.207501 25718 consensus_queue.cc:260] T 8eba3872a77046ecb9c9477d4cc0e18b P 6810a866decb4f5089f7d35c870e4204 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } }
I20240502 21:27:39.208170 25718 ts_tablet_manager.cc:1398] T 8eba3872a77046ecb9c9477d4cc0e18b P 6810a866decb4f5089f7d35c870e4204: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 21:27:39.208367 25717 tablet_bootstrap.cc:654] T d5697d8b06094211b261f144de9d46f8 P 009d3e7e8a8b4e2497de3a73d01ced49: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:39.211400 25716 tablet_bootstrap.cc:492] T 8eba3872a77046ecb9c9477d4cc0e18b P 6c194a6db6c145acafc0f3068b8ce1fd: No bootstrap required, opened a new log
I20240502 21:27:39.211575 25716 ts_tablet_manager.cc:1367] T 8eba3872a77046ecb9c9477d4cc0e18b P 6c194a6db6c145acafc0f3068b8ce1fd: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20240502 21:27:39.212364 25716 raft_consensus.cc:348] T 8eba3872a77046ecb9c9477d4cc0e18b P 6c194a6db6c145acafc0f3068b8ce1fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } }
I20240502 21:27:39.212648 25716 raft_consensus.cc:374] T 8eba3872a77046ecb9c9477d4cc0e18b P 6c194a6db6c145acafc0f3068b8ce1fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:39.212733 25716 raft_consensus.cc:729] T 8eba3872a77046ecb9c9477d4cc0e18b P 6c194a6db6c145acafc0f3068b8ce1fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c194a6db6c145acafc0f3068b8ce1fd, State: Initialized, Role: FOLLOWER
I20240502 21:27:39.213097 25717 tablet_bootstrap.cc:492] T d5697d8b06094211b261f144de9d46f8 P 009d3e7e8a8b4e2497de3a73d01ced49: No bootstrap required, opened a new log
I20240502 21:27:39.213047 25716 consensus_queue.cc:260] T 8eba3872a77046ecb9c9477d4cc0e18b P 6c194a6db6c145acafc0f3068b8ce1fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } }
I20240502 21:27:39.213222 25717 ts_tablet_manager.cc:1367] T d5697d8b06094211b261f144de9d46f8 P 009d3e7e8a8b4e2497de3a73d01ced49: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.001s
I20240502 21:27:39.213588 25716 ts_tablet_manager.cc:1398] T 8eba3872a77046ecb9c9477d4cc0e18b P 6c194a6db6c145acafc0f3068b8ce1fd: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:27:39.213877 25716 tablet_bootstrap.cc:492] T fcc9cbeae6f343af9f84c73349f3dadb P 6c194a6db6c145acafc0f3068b8ce1fd: Bootstrap starting.
I20240502 21:27:39.213853 25717 raft_consensus.cc:348] T d5697d8b06094211b261f144de9d46f8 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } }
I20240502 21:27:39.214030 25717 raft_consensus.cc:374] T d5697d8b06094211b261f144de9d46f8 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:39.214118 25717 raft_consensus.cc:729] T d5697d8b06094211b261f144de9d46f8 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 009d3e7e8a8b4e2497de3a73d01ced49, State: Initialized, Role: FOLLOWER
I20240502 21:27:39.214350 25717 consensus_queue.cc:260] T d5697d8b06094211b261f144de9d46f8 P 009d3e7e8a8b4e2497de3a73d01ced49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } }
I20240502 21:27:39.215441 25716 tablet_bootstrap.cc:654] T fcc9cbeae6f343af9f84c73349f3dadb P 6c194a6db6c145acafc0f3068b8ce1fd: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:39.216574 25715 tablet_bootstrap.cc:492] T 8eba3872a77046ecb9c9477d4cc0e18b P 038c5d9d1d7a49c981c4c34fbc00d9e6: No bootstrap required, opened a new log
I20240502 21:27:39.216748 25715 ts_tablet_manager.cc:1367] T 8eba3872a77046ecb9c9477d4cc0e18b P 038c5d9d1d7a49c981c4c34fbc00d9e6: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.000s
I20240502 21:27:39.216989 25717 ts_tablet_manager.cc:1398] T d5697d8b06094211b261f144de9d46f8 P 009d3e7e8a8b4e2497de3a73d01ced49: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 21:27:39.217489 25716 tablet_bootstrap.cc:492] T fcc9cbeae6f343af9f84c73349f3dadb P 6c194a6db6c145acafc0f3068b8ce1fd: No bootstrap required, opened a new log
I20240502 21:27:39.217510 25715 raft_consensus.cc:348] T 8eba3872a77046ecb9c9477d4cc0e18b P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } }
I20240502 21:27:39.217727 25715 raft_consensus.cc:374] T 8eba3872a77046ecb9c9477d4cc0e18b P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:39.217818 25715 raft_consensus.cc:729] T 8eba3872a77046ecb9c9477d4cc0e18b P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 038c5d9d1d7a49c981c4c34fbc00d9e6, State: Initialized, Role: FOLLOWER
I20240502 21:27:39.217916 25716 ts_tablet_manager.cc:1367] T fcc9cbeae6f343af9f84c73349f3dadb P 6c194a6db6c145acafc0f3068b8ce1fd: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20240502 21:27:39.218076 25715 consensus_queue.cc:260] T 8eba3872a77046ecb9c9477d4cc0e18b P 038c5d9d1d7a49c981c4c34fbc00d9e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } }
I20240502 21:27:39.218676 25715 ts_tablet_manager.cc:1398] T 8eba3872a77046ecb9c9477d4cc0e18b P 038c5d9d1d7a49c981c4c34fbc00d9e6: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:27:39.218633 25716 raft_consensus.cc:348] T fcc9cbeae6f343af9f84c73349f3dadb P 6c194a6db6c145acafc0f3068b8ce1fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } }
I20240502 21:27:39.218818 25716 raft_consensus.cc:374] T fcc9cbeae6f343af9f84c73349f3dadb P 6c194a6db6c145acafc0f3068b8ce1fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:39.218906 25716 raft_consensus.cc:729] T fcc9cbeae6f343af9f84c73349f3dadb P 6c194a6db6c145acafc0f3068b8ce1fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c194a6db6c145acafc0f3068b8ce1fd, State: Initialized, Role: FOLLOWER
I20240502 21:27:39.218978 25715 tablet_bootstrap.cc:492] T fcc9cbeae6f343af9f84c73349f3dadb P 038c5d9d1d7a49c981c4c34fbc00d9e6: Bootstrap starting.
I20240502 21:27:39.219424 25716 consensus_queue.cc:260] T fcc9cbeae6f343af9f84c73349f3dadb P 6c194a6db6c145acafc0f3068b8ce1fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } }
I20240502 21:27:39.220103 25716 ts_tablet_manager.cc:1398] T fcc9cbeae6f343af9f84c73349f3dadb P 6c194a6db6c145acafc0f3068b8ce1fd: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 21:27:39.221668 25715 tablet_bootstrap.cc:654] T fcc9cbeae6f343af9f84c73349f3dadb P 038c5d9d1d7a49c981c4c34fbc00d9e6: Neither blocks nor log segments found. Creating new log.
I20240502 21:27:39.223847 25715 tablet_bootstrap.cc:492] T fcc9cbeae6f343af9f84c73349f3dadb P 038c5d9d1d7a49c981c4c34fbc00d9e6: No bootstrap required, opened a new log
I20240502 21:27:39.224031 25715 ts_tablet_manager.cc:1367] T fcc9cbeae6f343af9f84c73349f3dadb P 038c5d9d1d7a49c981c4c34fbc00d9e6: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240502 21:27:39.224655 25715 raft_consensus.cc:348] T fcc9cbeae6f343af9f84c73349f3dadb P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } }
I20240502 21:27:39.224808 25715 raft_consensus.cc:374] T fcc9cbeae6f343af9f84c73349f3dadb P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:27:39.224916 25715 raft_consensus.cc:729] T fcc9cbeae6f343af9f84c73349f3dadb P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 038c5d9d1d7a49c981c4c34fbc00d9e6, State: Initialized, Role: FOLLOWER
I20240502 21:27:39.225082 25715 consensus_queue.cc:260] T fcc9cbeae6f343af9f84c73349f3dadb P 038c5d9d1d7a49c981c4c34fbc00d9e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } }
I20240502 21:27:39.225648 25715 ts_tablet_manager.cc:1398] T fcc9cbeae6f343af9f84c73349f3dadb P 038c5d9d1d7a49c981c4c34fbc00d9e6: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 21:27:39.373294 25390 raft_consensus.cc:482] T d5697d8b06094211b261f144de9d46f8 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:39.373471 25390 raft_consensus.cc:504] T d5697d8b06094211b261f144de9d46f8 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } }
I20240502 21:27:39.374051 25390 leader_election.cc:288] T d5697d8b06094211b261f144de9d46f8 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 009d3e7e8a8b4e2497de3a73d01ced49 (127.15.255.131:37895), 6c194a6db6c145acafc0f3068b8ce1fd (127.15.255.130:41235)
I20240502 21:27:39.374402 24185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5697d8b06094211b261f144de9d46f8" candidate_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" is_pre_election: true
I20240502 21:27:39.374491 24047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5697d8b06094211b261f144de9d46f8" candidate_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" is_pre_election: true
I20240502 21:27:39.374615 24185 raft_consensus.cc:2454] T d5697d8b06094211b261f144de9d46f8 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 038c5d9d1d7a49c981c4c34fbc00d9e6 in term 0.
I20240502 21:27:39.374701 24047 raft_consensus.cc:2454] T d5697d8b06094211b261f144de9d46f8 P 6c194a6db6c145acafc0f3068b8ce1fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 038c5d9d1d7a49c981c4c34fbc00d9e6 in term 0.
I20240502 21:27:39.375025 24252 leader_election.cc:302] T d5697d8b06094211b261f144de9d46f8 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 009d3e7e8a8b4e2497de3a73d01ced49, 038c5d9d1d7a49c981c4c34fbc00d9e6; no voters:
I20240502 21:27:39.375320 25390 raft_consensus.cc:2781] T d5697d8b06094211b261f144de9d46f8 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:39.375406 25390 raft_consensus.cc:482] T d5697d8b06094211b261f144de9d46f8 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:39.375504 25390 raft_consensus.cc:3037] T d5697d8b06094211b261f144de9d46f8 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:39.377302 25390 raft_consensus.cc:504] T d5697d8b06094211b261f144de9d46f8 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } }
I20240502 21:27:39.377821 25390 leader_election.cc:288] T d5697d8b06094211b261f144de9d46f8 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [CANDIDATE]: Term 1 election: Requested vote from peers 009d3e7e8a8b4e2497de3a73d01ced49 (127.15.255.131:37895), 6c194a6db6c145acafc0f3068b8ce1fd (127.15.255.130:41235)
I20240502 21:27:39.378226 24185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5697d8b06094211b261f144de9d46f8" candidate_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "009d3e7e8a8b4e2497de3a73d01ced49"
I20240502 21:27:39.378289 24047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5697d8b06094211b261f144de9d46f8" candidate_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c194a6db6c145acafc0f3068b8ce1fd"
I20240502 21:27:39.378432 24185 raft_consensus.cc:3037] T d5697d8b06094211b261f144de9d46f8 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:39.378461 24047 raft_consensus.cc:3037] T d5697d8b06094211b261f144de9d46f8 P 6c194a6db6c145acafc0f3068b8ce1fd [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:39.381091 24047 raft_consensus.cc:2454] T d5697d8b06094211b261f144de9d46f8 P 6c194a6db6c145acafc0f3068b8ce1fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 038c5d9d1d7a49c981c4c34fbc00d9e6 in term 1.
I20240502 21:27:39.381078 24185 raft_consensus.cc:2454] T d5697d8b06094211b261f144de9d46f8 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 038c5d9d1d7a49c981c4c34fbc00d9e6 in term 1.
I20240502 21:27:39.381457 24252 leader_election.cc:302] T d5697d8b06094211b261f144de9d46f8 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 009d3e7e8a8b4e2497de3a73d01ced49, 038c5d9d1d7a49c981c4c34fbc00d9e6; no voters:
I20240502 21:27:39.381704 25390 raft_consensus.cc:2781] T d5697d8b06094211b261f144de9d46f8 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:39.382061 25390 raft_consensus.cc:686] T d5697d8b06094211b261f144de9d46f8 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 1 LEADER]: Becoming Leader. State: Replica: 038c5d9d1d7a49c981c4c34fbc00d9e6, State: Running, Role: LEADER
I20240502 21:27:39.382302 25390 consensus_queue.cc:237] T d5697d8b06094211b261f144de9d46f8 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } }
I20240502 21:27:39.385113 23770 catalog_manager.cc:5484] T d5697d8b06094211b261f144de9d46f8 P 038c5d9d1d7a49c981c4c34fbc00d9e6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 038c5d9d1d7a49c981c4c34fbc00d9e6 (127.15.255.132). New cstate: current_term: 1 leader_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } health_report { overall_health: HEALTHY } } }
I20240502 21:27:39.525389 25390 raft_consensus.cc:482] T 8eba3872a77046ecb9c9477d4cc0e18b P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:39.525570 25390 raft_consensus.cc:504] T 8eba3872a77046ecb9c9477d4cc0e18b P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } }
I20240502 21:27:39.526255 25390 leader_election.cc:288] T 8eba3872a77046ecb9c9477d4cc0e18b P 038c5d9d1d7a49c981c4c34fbc00d9e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6810a866decb4f5089f7d35c870e4204 (127.15.255.129:40435), 6c194a6db6c145acafc0f3068b8ce1fd (127.15.255.130:41235)
I20240502 21:27:39.526822 24047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8eba3872a77046ecb9c9477d4cc0e18b" candidate_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" is_pre_election: true
I20240502 21:27:39.527029 24047 raft_consensus.cc:2454] T 8eba3872a77046ecb9c9477d4cc0e18b P 6c194a6db6c145acafc0f3068b8ce1fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 038c5d9d1d7a49c981c4c34fbc00d9e6 in term 0.
I20240502 21:27:39.527199 23915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8eba3872a77046ecb9c9477d4cc0e18b" candidate_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6810a866decb4f5089f7d35c870e4204" is_pre_election: true
I20240502 21:27:39.527413 23915 raft_consensus.cc:2454] T 8eba3872a77046ecb9c9477d4cc0e18b P 6810a866decb4f5089f7d35c870e4204 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 038c5d9d1d7a49c981c4c34fbc00d9e6 in term 0.
I20240502 21:27:39.527587 24253 leader_election.cc:302] T 8eba3872a77046ecb9c9477d4cc0e18b P 038c5d9d1d7a49c981c4c34fbc00d9e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 038c5d9d1d7a49c981c4c34fbc00d9e6, 6c194a6db6c145acafc0f3068b8ce1fd; no voters:
I20240502 21:27:39.527844 25390 raft_consensus.cc:2781] T 8eba3872a77046ecb9c9477d4cc0e18b P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:39.527966 25390 raft_consensus.cc:482] T 8eba3872a77046ecb9c9477d4cc0e18b P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:39.528064 25390 raft_consensus.cc:3037] T 8eba3872a77046ecb9c9477d4cc0e18b P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:39.530113 25390 raft_consensus.cc:504] T 8eba3872a77046ecb9c9477d4cc0e18b P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } }
I20240502 21:27:39.530725 25390 leader_election.cc:288] T 8eba3872a77046ecb9c9477d4cc0e18b P 038c5d9d1d7a49c981c4c34fbc00d9e6 [CANDIDATE]: Term 1 election: Requested vote from peers 6810a866decb4f5089f7d35c870e4204 (127.15.255.129:40435), 6c194a6db6c145acafc0f3068b8ce1fd (127.15.255.130:41235)
I20240502 21:27:39.531100 23915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8eba3872a77046ecb9c9477d4cc0e18b" candidate_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6810a866decb4f5089f7d35c870e4204"
I20240502 21:27:39.531286 23915 raft_consensus.cc:3037] T 8eba3872a77046ecb9c9477d4cc0e18b P 6810a866decb4f5089f7d35c870e4204 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:39.531256 24047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8eba3872a77046ecb9c9477d4cc0e18b" candidate_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c194a6db6c145acafc0f3068b8ce1fd"
I20240502 21:27:39.531487 24047 raft_consensus.cc:3037] T 8eba3872a77046ecb9c9477d4cc0e18b P 6c194a6db6c145acafc0f3068b8ce1fd [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:39.533846 24047 raft_consensus.cc:2454] T 8eba3872a77046ecb9c9477d4cc0e18b P 6c194a6db6c145acafc0f3068b8ce1fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 038c5d9d1d7a49c981c4c34fbc00d9e6 in term 1.
I20240502 21:27:39.533990 23915 raft_consensus.cc:2454] T 8eba3872a77046ecb9c9477d4cc0e18b P 6810a866decb4f5089f7d35c870e4204 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 038c5d9d1d7a49c981c4c34fbc00d9e6 in term 1.
I20240502 21:27:39.534313 24253 leader_election.cc:302] T 8eba3872a77046ecb9c9477d4cc0e18b P 038c5d9d1d7a49c981c4c34fbc00d9e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 038c5d9d1d7a49c981c4c34fbc00d9e6, 6c194a6db6c145acafc0f3068b8ce1fd; no voters:
I20240502 21:27:39.535087 25390 raft_consensus.cc:2781] T 8eba3872a77046ecb9c9477d4cc0e18b P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:39.535229 25390 raft_consensus.cc:686] T 8eba3872a77046ecb9c9477d4cc0e18b P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 1 LEADER]: Becoming Leader. State: Replica: 038c5d9d1d7a49c981c4c34fbc00d9e6, State: Running, Role: LEADER
I20240502 21:27:39.535472 25390 consensus_queue.cc:237] T 8eba3872a77046ecb9c9477d4cc0e18b P 038c5d9d1d7a49c981c4c34fbc00d9e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } }
I20240502 21:27:39.538297 23770 catalog_manager.cc:5484] T 8eba3872a77046ecb9c9477d4cc0e18b P 038c5d9d1d7a49c981c4c34fbc00d9e6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 038c5d9d1d7a49c981c4c34fbc00d9e6 (127.15.255.132). New cstate: current_term: 1 leader_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } health_report { overall_health: HEALTHY } } }
I20240502 21:27:39.581836 25390 raft_consensus.cc:482] T fcc9cbeae6f343af9f84c73349f3dadb P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:27:39.582005 25390 raft_consensus.cc:504] T fcc9cbeae6f343af9f84c73349f3dadb P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } }
I20240502 21:27:39.582576 25390 leader_election.cc:288] T fcc9cbeae6f343af9f84c73349f3dadb P 038c5d9d1d7a49c981c4c34fbc00d9e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6c194a6db6c145acafc0f3068b8ce1fd (127.15.255.130:41235), 009d3e7e8a8b4e2497de3a73d01ced49 (127.15.255.131:37895)
I20240502 21:27:39.582971 24047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcc9cbeae6f343af9f84c73349f3dadb" candidate_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" is_pre_election: true
I20240502 21:27:39.583014 24185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcc9cbeae6f343af9f84c73349f3dadb" candidate_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" is_pre_election: true
I20240502 21:27:39.583154 24047 raft_consensus.cc:2454] T fcc9cbeae6f343af9f84c73349f3dadb P 6c194a6db6c145acafc0f3068b8ce1fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 038c5d9d1d7a49c981c4c34fbc00d9e6 in term 0.
I20240502 21:27:39.583194 24185 raft_consensus.cc:2454] T fcc9cbeae6f343af9f84c73349f3dadb P 009d3e7e8a8b4e2497de3a73d01ced49 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 038c5d9d1d7a49c981c4c34fbc00d9e6 in term 0.
I20240502 21:27:39.583535 24253 leader_election.cc:302] T fcc9cbeae6f343af9f84c73349f3dadb P 038c5d9d1d7a49c981c4c34fbc00d9e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 038c5d9d1d7a49c981c4c34fbc00d9e6, 6c194a6db6c145acafc0f3068b8ce1fd; no voters:
I20240502 21:27:39.583786 25390 raft_consensus.cc:2781] T fcc9cbeae6f343af9f84c73349f3dadb P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:27:39.583863 25390 raft_consensus.cc:482] T fcc9cbeae6f343af9f84c73349f3dadb P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:27:39.583952 25390 raft_consensus.cc:3037] T fcc9cbeae6f343af9f84c73349f3dadb P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:39.586114 25390 raft_consensus.cc:504] T fcc9cbeae6f343af9f84c73349f3dadb P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } }
I20240502 21:27:39.586622 25390 leader_election.cc:288] T fcc9cbeae6f343af9f84c73349f3dadb P 038c5d9d1d7a49c981c4c34fbc00d9e6 [CANDIDATE]: Term 1 election: Requested vote from peers 6c194a6db6c145acafc0f3068b8ce1fd (127.15.255.130:41235), 009d3e7e8a8b4e2497de3a73d01ced49 (127.15.255.131:37895)
I20240502 21:27:39.586952 24047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcc9cbeae6f343af9f84c73349f3dadb" candidate_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c194a6db6c145acafc0f3068b8ce1fd"
I20240502 21:27:39.587008 24185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcc9cbeae6f343af9f84c73349f3dadb" candidate_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "009d3e7e8a8b4e2497de3a73d01ced49"
I20240502 21:27:39.587122 24047 raft_consensus.cc:3037] T fcc9cbeae6f343af9f84c73349f3dadb P 6c194a6db6c145acafc0f3068b8ce1fd [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:39.587173 24185 raft_consensus.cc:3037] T fcc9cbeae6f343af9f84c73349f3dadb P 009d3e7e8a8b4e2497de3a73d01ced49 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:27:39.588928 24047 raft_consensus.cc:2454] T fcc9cbeae6f343af9f84c73349f3dadb P 6c194a6db6c145acafc0f3068b8ce1fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 038c5d9d1d7a49c981c4c34fbc00d9e6 in term 1.
I20240502 21:27:39.588963 24185 raft_consensus.cc:2454] T fcc9cbeae6f343af9f84c73349f3dadb P 009d3e7e8a8b4e2497de3a73d01ced49 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 038c5d9d1d7a49c981c4c34fbc00d9e6 in term 1.
I20240502 21:27:39.589242 24253 leader_election.cc:302] T fcc9cbeae6f343af9f84c73349f3dadb P 038c5d9d1d7a49c981c4c34fbc00d9e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 038c5d9d1d7a49c981c4c34fbc00d9e6, 6c194a6db6c145acafc0f3068b8ce1fd; no voters:
I20240502 21:27:39.589473 25390 raft_consensus.cc:2781] T fcc9cbeae6f343af9f84c73349f3dadb P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:27:39.589622 25390 raft_consensus.cc:686] T fcc9cbeae6f343af9f84c73349f3dadb P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 1 LEADER]: Becoming Leader. State: Replica: 038c5d9d1d7a49c981c4c34fbc00d9e6, State: Running, Role: LEADER
I20240502 21:27:39.589844 25390 consensus_queue.cc:237] T fcc9cbeae6f343af9f84c73349f3dadb P 038c5d9d1d7a49c981c4c34fbc00d9e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } }
I20240502 21:27:39.592494 23770 catalog_manager.cc:5484] T fcc9cbeae6f343af9f84c73349f3dadb P 038c5d9d1d7a49c981c4c34fbc00d9e6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 038c5d9d1d7a49c981c4c34fbc00d9e6 (127.15.255.132). New cstate: current_term: 1 leader_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } health_report { overall_health: HEALTHY } } }
I20240502 21:27:39.765681 25390 consensus_queue.cc:1035] T d5697d8b06094211b261f144de9d46f8 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:39.771888 25390 consensus_queue.cc:1035] T d5697d8b06094211b261f144de9d46f8 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:39.953125 25390 consensus_queue.cc:1035] T 8eba3872a77046ecb9c9477d4cc0e18b P 038c5d9d1d7a49c981c4c34fbc00d9e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:39.961822 25390 consensus_queue.cc:1035] T 8eba3872a77046ecb9c9477d4cc0e18b P 038c5d9d1d7a49c981c4c34fbc00d9e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 21:27:39.987289 25734 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:27:39.987589 25734 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:27:39.995182 25734 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_9'
I20240502 21:27:40.068729 25742 consensus_queue.cc:1035] T fcc9cbeae6f343af9f84c73349f3dadb P 038c5d9d1d7a49c981c4c34fbc00d9e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:27:40.090037 25730 consensus_queue.cc:1035] T fcc9cbeae6f343af9f84c73349f3dadb P 038c5d9d1d7a49c981c4c34fbc00d9e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 235.932 ms
time per row: 0.117966 ms
I20240502 21:27:40.368575 16382 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvNciQZ/build/asan/bin/kudu
/tmp/dist-test-taskvNciQZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.255.133:40841
--local_ip_for_outbound_sockets=127.15.255.133
--tserver_master_addrs=127.15.255.190:33301
--webserver_port=43285
--webserver_interface=127.15.255.133
--builtin_ntp_servers=127.15.255.148:46015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
W20240502 21:27:40.698048 25795 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:27:40.698323 25795 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:27:40.698464 25795 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:27:40.705523 25795 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 21:27:40.705631 25795 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:27:40.705770 25795 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.255.133
I20240502 21:27:40.715343 25795 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.255.148:46015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.255.133:40841
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.15.255.133
--webserver_port=43285
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.15.255.190:33301
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.255.133
--log_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:27:40.716549 25795 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:27:40.718120 25795 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:27:40.729141 25802 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:27:40.729254 25801 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:27:40.729645 25804 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:27:40.730051 25795 server_base.cc:1034] running on GCE node
I20240502 21:27:40.730552 25795 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:27:40.731406 25795 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:27:40.732596 25795 hybrid_clock.cc:648] HybridClock initialized: now 1714685260732551 us; error 35 us; skew 500 ppm
I20240502 21:27:40.732913 25795 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:27:40.735026 25795 webserver.cc:460] Webserver started at http://127.15.255.133:43285/ using document root <none> and password file <none>
I20240502 21:27:40.735697 25795 fs_manager.cc:359] Metadata directory not provided
I20240502 21:27:40.735843 25795 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:27:40.740873 25795 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20240502 21:27:40.743412 25810 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:27:40.744679 25795 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240502 21:27:40.744804 25795 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "ee27c3f9e0ec46caa742ac386d6b7c20"
format_stamp: "Formatted at 2024-05-02 21:27:20 on dist-test-slave-jdfx"
I20240502 21:27:40.745549 25795 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:27:40.769095 25795 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:27:40.769762 25795 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:27:40.769989 25795 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:27:40.770763 25795 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:27:40.772102 25795 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:27:40.772177 25795 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:27:40.772266 25795 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:27:40.772310 25795 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 21:27:40.804872 25795 rpc_server.cc:307] RPC server started. Bound to: 127.15.255.133:40841
I20240502 21:27:40.804920 25923 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.255.133:40841 every 8 connection(s)
I20240502 21:27:40.806241 25795 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240502 21:27:40.806777 16382 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvNciQZ/build/asan/bin/kudu as pid 25795
I20240502 21:27:40.807574 16382 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvNciQZ/build/asan/bin/kudu
/tmp/dist-test-taskvNciQZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.255.134:32825
--local_ip_for_outbound_sockets=127.15.255.134
--tserver_master_addrs=127.15.255.190:33301
--webserver_port=43073
--webserver_interface=127.15.255.134
--builtin_ntp_servers=127.15.255.148:46015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240502 21:27:40.819145 25924 heartbeater.cc:347] Connected to a master server at 127.15.255.190:33301
I20240502 21:27:40.819445 25924 heartbeater.cc:464] Registering TS with master...
I20240502 21:27:40.820251 25924 heartbeater.cc:511] Master 127.15.255.190:33301 requested a full tablet report, sending...
I20240502 21:27:40.821947 23770 ts_manager.cc:194] Re-registered known tserver with Master: ee27c3f9e0ec46caa742ac386d6b7c20 (127.15.255.133:40841)
I20240502 21:27:40.822760 23770 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.255.133:51813
W20240502 21:27:41.158067 25928 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:27:41.158336 25928 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:27:41.158493 25928 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:27:41.166663 25928 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 21:27:41.166798 25928 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:27:41.166972 25928 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.255.134
I20240502 21:27:41.176179 25928 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.255.148:46015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.255.134:32825
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.15.255.134
--webserver_port=43073
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.15.255.190:33301
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.255.134
--log_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:27:41.177429 25928 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:27:41.179134 25928 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:27:41.191607 25933 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:27:41.192029 25934 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:27:41.192479 25936 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:27:41.192829 25928 server_base.cc:1034] running on GCE node
I20240502 21:27:41.193405 25928 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:27:41.194321 25928 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:27:41.195531 25928 hybrid_clock.cc:648] HybridClock initialized: now 1714685261195465 us; error 56 us; skew 500 ppm
I20240502 21:27:41.195829 25928 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:27:41.198004 25928 webserver.cc:460] Webserver started at http://127.15.255.134:43073/ using document root <none> and password file <none>
I20240502 21:27:41.198691 25928 fs_manager.cc:359] Metadata directory not provided
I20240502 21:27:41.198823 25928 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:27:41.204373 25928 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20240502 21:27:41.207473 25942 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:27:41.208916 25928 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20240502 21:27:41.209067 25928 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "fb6004b8640a4906aa0cfe63e383d40e"
format_stamp: "Formatted at 2024-05-02 21:27:21 on dist-test-slave-jdfx"
I20240502 21:27:41.210129 25928 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:27:41.233757 25928 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:27:41.234498 25928 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:27:41.234758 25928 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:27:41.235603 25928 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:27:41.237068 25928 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:27:41.237143 25928 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:27:41.237267 25928 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:27:41.237316 25928 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 21:27:41.266516 25928 rpc_server.cc:307] RPC server started. Bound to: 127.15.255.134:32825
I20240502 21:27:41.266561 26057 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.255.134:32825 every 8 connection(s)
I20240502 21:27:41.267910 25928 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240502 21:27:41.270375 16382 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvNciQZ/build/asan/bin/kudu as pid 25928
I20240502 21:27:41.271096 16382 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvNciQZ/build/asan/bin/kudu
/tmp/dist-test-taskvNciQZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.255.135:32775
--local_ip_for_outbound_sockets=127.15.255.135
--tserver_master_addrs=127.15.255.190:33301
--webserver_port=34771
--webserver_interface=127.15.255.135
--builtin_ntp_servers=127.15.255.148:46015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240502 21:27:41.280714 26058 heartbeater.cc:347] Connected to a master server at 127.15.255.190:33301
I20240502 21:27:41.280982 26058 heartbeater.cc:464] Registering TS with master...
I20240502 21:27:41.281735 26058 heartbeater.cc:511] Master 127.15.255.190:33301 requested a full tablet report, sending...
I20240502 21:27:41.283067 23770 ts_manager.cc:194] Re-registered known tserver with Master: fb6004b8640a4906aa0cfe63e383d40e (127.15.255.134:32825)
I20240502 21:27:41.284096 23770 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.255.134:50667
W20240502 21:27:41.606205 26061 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:27:41.606454 26061 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:27:41.606590 26061 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:27:41.613951 26061 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 21:27:41.614049 26061 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:27:41.614188 26061 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.255.135
I20240502 21:27:41.623054 26061 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.255.148:46015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.255.135:32775
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.15.255.135
--webserver_port=34771
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.15.255.190:33301
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.255.135
--log_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:27:41.624225 26061 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:27:41.625913 26061 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:27:41.637130 26069 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:27:41.637207 26068 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:27:41.637648 26071 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:27:41.637953 26061 server_base.cc:1034] running on GCE node
I20240502 21:27:41.638455 26061 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:27:41.639248 26061 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:27:41.640472 26061 hybrid_clock.cc:648] HybridClock initialized: now 1714685261640395 us; error 34 us; skew 500 ppm
I20240502 21:27:41.640750 26061 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:27:41.642756 26061 webserver.cc:460] Webserver started at http://127.15.255.135:34771/ using document root <none> and password file <none>
I20240502 21:27:41.643535 26061 fs_manager.cc:359] Metadata directory not provided
I20240502 21:27:41.643679 26061 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:27:41.648748 26061 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20240502 21:27:41.651340 26077 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:27:41.652624 26061 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240502 21:27:41.652760 26061 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "e117afbf8819424e8a98476453dfde2a"
format_stamp: "Formatted at 2024-05-02 21:27:21 on dist-test-slave-jdfx"
I20240502 21:27:41.653556 26061 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:27:41.677528 26061 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:27:41.678361 26061 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:27:41.678617 26061 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:27:41.679574 26061 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:27:41.680986 26061 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:27:41.681057 26061 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:27:41.681143 26061 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:27:41.681185 26061 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 21:27:41.712342 26061 rpc_server.cc:307] RPC server started. Bound to: 127.15.255.135:32775
I20240502 21:27:41.712441 26192 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.255.135:32775 every 8 connection(s)
I20240502 21:27:41.713728 26061 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240502 21:27:41.723457 16382 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvNciQZ/build/asan/bin/kudu as pid 26061
I20240502 21:27:41.724224 16382 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvNciQZ/build/asan/bin/kudu
/tmp/dist-test-taskvNciQZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.255.136:43595
--local_ip_for_outbound_sockets=127.15.255.136
--tserver_master_addrs=127.15.255.190:33301
--webserver_port=41673
--webserver_interface=127.15.255.136
--builtin_ntp_servers=127.15.255.148:46015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240502 21:27:41.725462 26193 heartbeater.cc:347] Connected to a master server at 127.15.255.190:33301
I20240502 21:27:41.725744 26193 heartbeater.cc:464] Registering TS with master...
I20240502 21:27:41.726657 26193 heartbeater.cc:511] Master 127.15.255.190:33301 requested a full tablet report, sending...
I20240502 21:27:41.728039 23770 ts_manager.cc:194] Re-registered known tserver with Master: e117afbf8819424e8a98476453dfde2a (127.15.255.135:32775)
I20240502 21:27:41.728833 23770 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.255.135:39143
I20240502 21:27:41.824982 25924 heartbeater.cc:503] Master 127.15.255.190:33301 was elected leader, sending a full tablet report...
W20240502 21:27:42.063491 26196 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:27:42.063757 26196 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:27:42.063910 26196 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:27:42.071077 26196 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 21:27:42.071192 26196 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:27:42.071341 26196 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.255.136
I20240502 21:27:42.083621 26196 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.255.148:46015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.255.136:43595
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.15.255.136
--webserver_port=41673
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.15.255.190:33301
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.255.136
--log_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:27:42.085099 26196 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:27:42.087183 26196 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:27:42.099776 26201 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:27:42.100103 26202 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:27:42.100557 26204 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:27:42.100765 26196 server_base.cc:1034] running on GCE node
I20240502 21:27:42.101255 26196 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:27:42.102103 26196 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:27:42.103294 26196 hybrid_clock.cc:648] HybridClock initialized: now 1714685262103237 us; error 38 us; skew 500 ppm
I20240502 21:27:42.103633 26196 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:27:42.106559 26196 webserver.cc:460] Webserver started at http://127.15.255.136:41673/ using document root <none> and password file <none>
I20240502 21:27:42.107276 26196 fs_manager.cc:359] Metadata directory not provided
I20240502 21:27:42.107386 26196 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:27:42.112491 26196 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20240502 21:27:42.115128 26210 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:27:42.116535 26196 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240502 21:27:42.116678 26196 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "a3e500d7077a48acb5edb33f7e178c81"
format_stamp: "Formatted at 2024-05-02 21:27:22 on dist-test-slave-jdfx"
I20240502 21:27:42.117424 26196 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:27:42.146521 26196 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:27:42.147282 26196 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:27:42.147543 26196 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:27:42.148630 26196 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:27:42.150449 26196 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:27:42.150540 26196 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:27:42.150665 26196 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:27:42.150740 26196 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 21:27:42.187088 26196 rpc_server.cc:307] RPC server started. Bound to: 127.15.255.136:43595
I20240502 21:27:42.187094 26325 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.255.136:43595 every 8 connection(s)
I20240502 21:27:42.188465 26196 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240502 21:27:42.195837 16382 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvNciQZ/build/asan/bin/kudu as pid 26196
I20240502 21:27:42.200825 26326 heartbeater.cc:347] Connected to a master server at 127.15.255.190:33301
I20240502 21:27:42.201088 26326 heartbeater.cc:464] Registering TS with master...
I20240502 21:27:42.201843 26326 heartbeater.cc:511] Master 127.15.255.190:33301 requested a full tablet report, sending...
I20240502 21:27:42.203411 23771 ts_manager.cc:194] Re-registered known tserver with Master: a3e500d7077a48acb5edb33f7e178c81 (127.15.255.136:43595)
I20240502 21:27:42.204479 23771 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.255.136:48775
I20240502 21:27:42.286870 26058 heartbeater.cc:503] Master 127.15.255.190:33301 was elected leader, sending a full tablet report...
W20240502 21:27:42.594374 26331 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:27:42.594653 26331 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:27:42.602231 26331 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 21:27:42.731395 26193 heartbeater.cc:503] Master 127.15.255.190:33301 was elected leader, sending a full tablet report...
I20240502 21:27:42.830991 24297 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 21:27:42.890633 23894 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 21:27:42.911625 26127 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:42.918102 24162 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:27:42.940554 24027 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:27:42.947886 26260 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:42.965663 25992 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:42.971668 25859 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:43.002545 24162 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:27:43.051051 24297 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 21:27:43.118480 23894 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 21:27:43.127424 26127 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:43.129312 26260 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:43.133101 24027 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:27:43.164492 25992 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:43.171577 25859 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:43.206912 26326 heartbeater.cc:503] Master 127.15.255.190:33301 was elected leader, sending a full tablet report...
I20240502 21:27:43.441498 24162 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:27:43.459376 23894 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 21:27:43.470055 24297 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 21:27:43.512162 24027 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:27:43.528322 26260 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:43.579846 25992 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:43.580844 25859 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:43.581193 26127 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240502 21:27:43.797024 26447 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:27:43.797499 26447 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:27:43.809063 26447 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 21:27:43.847976 24162 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:27:43.881781 23894 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 21:27:43.889880 26260 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:43.895063 24027 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:27:43.902058 24297 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 21:27:43.936540 25859 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:43.937392 25992 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:43.958663 26127 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:44.080178 24184 consensus_queue.cc:237] T cca45c89b9fe40e7800d95c1598a9e52 P 009d3e7e8a8b4e2497de3a73d01ced49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } attrs { replace: true } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: NON_VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: true } }
I20240502 21:27:44.104050 24048 raft_consensus.cc:1261] T cca45c89b9fe40e7800d95c1598a9e52 P 6c194a6db6c145acafc0f3068b8ce1fd [term 1 FOLLOWER]: Refusing update from remote peer 009d3e7e8a8b4e2497de3a73d01ced49: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:27:44.104871 24315 raft_consensus.cc:1261] T cca45c89b9fe40e7800d95c1598a9e52 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 1 FOLLOWER]: Refusing update from remote peer 009d3e7e8a8b4e2497de3a73d01ced49: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:27:44.104876 26475 consensus_queue.cc:1035] T cca45c89b9fe40e7800d95c1598a9e52 P 009d3e7e8a8b4e2497de3a73d01ced49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:27:44.105887 26554 consensus_queue.cc:1035] T cca45c89b9fe40e7800d95c1598a9e52 P 009d3e7e8a8b4e2497de3a73d01ced49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:27:44.120956 26475 raft_consensus.cc:2932] T cca45c89b9fe40e7800d95c1598a9e52 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ee27c3f9e0ec46caa742ac386d6b7c20 (127.15.255.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } attrs { replace: true } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: NON_VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: true } } }
I20240502 21:27:44.122995 24048 raft_consensus.cc:2932] T cca45c89b9fe40e7800d95c1598a9e52 P 6c194a6db6c145acafc0f3068b8ce1fd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ee27c3f9e0ec46caa742ac386d6b7c20 (127.15.255.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } attrs { replace: true } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: NON_VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: true } } }
I20240502 21:27:44.133428 23768 catalog_manager.cc:5484] T cca45c89b9fe40e7800d95c1598a9e52 P 009d3e7e8a8b4e2497de3a73d01ced49 reported cstate change: config changed from index -1 to 4, NON_VOTER ee27c3f9e0ec46caa742ac386d6b7c20 (127.15.255.133) added. New cstate: current_term: 1 leader_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: NON_VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240502 21:27:44.134899 24115 consensus_peers.cc:485] T cca45c89b9fe40e7800d95c1598a9e52 P 009d3e7e8a8b4e2497de3a73d01ced49 -> Peer ee27c3f9e0ec46caa742ac386d6b7c20 (127.15.255.133:40841): Couldn't send request to peer ee27c3f9e0ec46caa742ac386d6b7c20. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cca45c89b9fe40e7800d95c1598a9e52. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:44.136492 24315 raft_consensus.cc:2932] T cca45c89b9fe40e7800d95c1598a9e52 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ee27c3f9e0ec46caa742ac386d6b7c20 (127.15.255.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } attrs { replace: true } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: NON_VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: true } } }
I20240502 21:27:44.165333 24162 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:27:44.170655 24297 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 21:27:44.195463 24185 consensus_queue.cc:237] T 0b3c514213bb40ca98955d6409a3b372 P 009d3e7e8a8b4e2497de3a73d01ced49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } attrs { replace: true } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: NON_VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: true } }
I20240502 21:27:44.218737 23894 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 21:27:44.232051 24027 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:27:44.240025 24315 raft_consensus.cc:1261] T 0b3c514213bb40ca98955d6409a3b372 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 1 FOLLOWER]: Refusing update from remote peer 009d3e7e8a8b4e2497de3a73d01ced49: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:27:44.240615 26554 consensus_queue.cc:1035] T 0b3c514213bb40ca98955d6409a3b372 P 009d3e7e8a8b4e2497de3a73d01ced49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:27:44.244045 23916 raft_consensus.cc:1261] T 0b3c514213bb40ca98955d6409a3b372 P 6810a866decb4f5089f7d35c870e4204 [term 1 FOLLOWER]: Refusing update from remote peer 009d3e7e8a8b4e2497de3a73d01ced49: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:27:44.245218 26127 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:44.245311 26554 consensus_queue.cc:1035] T 0b3c514213bb40ca98955d6409a3b372 P 009d3e7e8a8b4e2497de3a73d01ced49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 21:27:44.247138 24115 consensus_peers.cc:485] T 0b3c514213bb40ca98955d6409a3b372 P 009d3e7e8a8b4e2497de3a73d01ced49 -> Peer fb6004b8640a4906aa0cfe63e383d40e (127.15.255.134:32825): Couldn't send request to peer fb6004b8640a4906aa0cfe63e383d40e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0b3c514213bb40ca98955d6409a3b372. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:44.261059 26260 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:44.262398 26554 raft_consensus.cc:2932] T 0b3c514213bb40ca98955d6409a3b372 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER fb6004b8640a4906aa0cfe63e383d40e (127.15.255.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } attrs { replace: true } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: NON_VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: true } } }
I20240502 21:27:44.265457 24315 raft_consensus.cc:2932] T 0b3c514213bb40ca98955d6409a3b372 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER fb6004b8640a4906aa0cfe63e383d40e (127.15.255.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } attrs { replace: true } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: NON_VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: true } } }
I20240502 21:27:44.271909 23766 catalog_manager.cc:5484] T 0b3c514213bb40ca98955d6409a3b372 P 038c5d9d1d7a49c981c4c34fbc00d9e6 reported cstate change: config changed from index -1 to 4, NON_VOTER fb6004b8640a4906aa0cfe63e383d40e (127.15.255.134) added. New cstate: current_term: 1 leader_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } attrs { replace: true } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: NON_VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: true } } }
I20240502 21:27:44.291170 25859 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:44.291921 23916 raft_consensus.cc:2932] T 0b3c514213bb40ca98955d6409a3b372 P 6810a866decb4f5089f7d35c870e4204 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER fb6004b8640a4906aa0cfe63e383d40e (127.15.255.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } attrs { replace: true } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: NON_VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: true } } }
I20240502 21:27:44.322140 25992 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:44.349889 23916 consensus_queue.cc:237] T 4eebf4508766421eb6c547a0fbbc28b1 P 6810a866decb4f5089f7d35c870e4204 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } attrs { replace: true } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: NON_VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: true } }
I20240502 21:27:44.353427 24315 raft_consensus.cc:1261] T 4eebf4508766421eb6c547a0fbbc28b1 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 1 FOLLOWER]: Refusing update from remote peer 6810a866decb4f5089f7d35c870e4204: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:27:44.353574 24185 raft_consensus.cc:1261] T 4eebf4508766421eb6c547a0fbbc28b1 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 1 FOLLOWER]: Refusing update from remote peer 6810a866decb4f5089f7d35c870e4204: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:27:44.354085 26471 consensus_queue.cc:1035] T 4eebf4508766421eb6c547a0fbbc28b1 P 6810a866decb4f5089f7d35c870e4204 [LEADER]: Connected to new peer: Peer: permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:27:44.354683 26471 consensus_queue.cc:1035] T 4eebf4508766421eb6c547a0fbbc28b1 P 6810a866decb4f5089f7d35c870e4204 [LEADER]: Connected to new peer: Peer: permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:27:44.365535 26471 raft_consensus.cc:2932] T 4eebf4508766421eb6c547a0fbbc28b1 P 6810a866decb4f5089f7d35c870e4204 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER e117afbf8819424e8a98476453dfde2a (127.15.255.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } attrs { replace: true } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: NON_VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: true } } }
I20240502 21:27:44.371856 23770 catalog_manager.cc:5484] T 4eebf4508766421eb6c547a0fbbc28b1 P 6810a866decb4f5089f7d35c870e4204 reported cstate change: config changed from index -1 to 4, NON_VOTER e117afbf8819424e8a98476453dfde2a (127.15.255.135) added. New cstate: current_term: 1 leader_uuid: "6810a866decb4f5089f7d35c870e4204" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: NON_VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:44.372401 24315 raft_consensus.cc:2932] T 4eebf4508766421eb6c547a0fbbc28b1 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER e117afbf8819424e8a98476453dfde2a (127.15.255.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } attrs { replace: true } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: NON_VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: true } } }
W20240502 21:27:44.376305 23850 consensus_peers.cc:485] T 4eebf4508766421eb6c547a0fbbc28b1 P 6810a866decb4f5089f7d35c870e4204 -> Peer e117afbf8819424e8a98476453dfde2a (127.15.255.135:32775): Couldn't send request to peer e117afbf8819424e8a98476453dfde2a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4eebf4508766421eb6c547a0fbbc28b1. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:44.384928 24185 raft_consensus.cc:2932] T 4eebf4508766421eb6c547a0fbbc28b1 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER e117afbf8819424e8a98476453dfde2a (127.15.255.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } attrs { replace: true } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: NON_VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: true } } }
Runtime error: ksck discovered errors
I20240502 21:27:44.397223 23916 consensus_queue.cc:237] T adf80b9d71314010a7a4cf052fd8a511 P 6810a866decb4f5089f7d35c870e4204 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } attrs { replace: true } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: true } }
I20240502 21:27:44.399803 24315 raft_consensus.cc:1261] T adf80b9d71314010a7a4cf052fd8a511 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 1 FOLLOWER]: Refusing update from remote peer 6810a866decb4f5089f7d35c870e4204: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:27:44.400014 24048 raft_consensus.cc:1261] T adf80b9d71314010a7a4cf052fd8a511 P 6c194a6db6c145acafc0f3068b8ce1fd [term 1 FOLLOWER]: Refusing update from remote peer 6810a866decb4f5089f7d35c870e4204: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:27:44.400331 26471 consensus_queue.cc:1035] T adf80b9d71314010a7a4cf052fd8a511 P 6810a866decb4f5089f7d35c870e4204 [LEADER]: Connected to new peer: Peer: permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:27:44.400702 26586 consensus_queue.cc:1035] T adf80b9d71314010a7a4cf052fd8a511 P 6810a866decb4f5089f7d35c870e4204 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:27:44.405397 26471 raft_consensus.cc:2932] T adf80b9d71314010a7a4cf052fd8a511 P 6810a866decb4f5089f7d35c870e4204 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a3e500d7077a48acb5edb33f7e178c81 (127.15.255.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } attrs { replace: true } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: true } } }
W20240502 21:27:44.406996 23850 consensus_peers.cc:485] T adf80b9d71314010a7a4cf052fd8a511 P 6810a866decb4f5089f7d35c870e4204 -> Peer a3e500d7077a48acb5edb33f7e178c81 (127.15.255.136:43595): Couldn't send request to peer a3e500d7077a48acb5edb33f7e178c81. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: adf80b9d71314010a7a4cf052fd8a511. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:44.407361 24315 raft_consensus.cc:2932] T adf80b9d71314010a7a4cf052fd8a511 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a3e500d7077a48acb5edb33f7e178c81 (127.15.255.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } attrs { replace: true } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: true } } }
I20240502 21:27:44.411844 23768 catalog_manager.cc:5484] T adf80b9d71314010a7a4cf052fd8a511 P 6810a866decb4f5089f7d35c870e4204 reported cstate change: config changed from index -1 to 4, NON_VOTER a3e500d7077a48acb5edb33f7e178c81 (127.15.255.136) added. New cstate: current_term: 1 leader_uuid: "6810a866decb4f5089f7d35c870e4204" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:44.414316 24048 raft_consensus.cc:2932] T adf80b9d71314010a7a4cf052fd8a511 P 6c194a6db6c145acafc0f3068b8ce1fd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a3e500d7077a48acb5edb33f7e178c81 (127.15.255.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } attrs { replace: true } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: true } } }
I20240502 21:27:44.454021 24185 consensus_queue.cc:237] T 6bd759d700a24e9f8ad11f243cf7ebfa P 009d3e7e8a8b4e2497de3a73d01ced49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } attrs { replace: true } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: NON_VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: true } }
I20240502 21:27:44.456127 24315 raft_consensus.cc:1261] T 6bd759d700a24e9f8ad11f243cf7ebfa P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 1 FOLLOWER]: Refusing update from remote peer 009d3e7e8a8b4e2497de3a73d01ced49: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:27:44.456279 23916 raft_consensus.cc:1261] T 6bd759d700a24e9f8ad11f243cf7ebfa P 6810a866decb4f5089f7d35c870e4204 [term 1 FOLLOWER]: Refusing update from remote peer 009d3e7e8a8b4e2497de3a73d01ced49: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:27:44.456831 25728 consensus_queue.cc:1035] T 6bd759d700a24e9f8ad11f243cf7ebfa P 009d3e7e8a8b4e2497de3a73d01ced49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:27:44.457206 26475 consensus_queue.cc:1035] T 6bd759d700a24e9f8ad11f243cf7ebfa P 009d3e7e8a8b4e2497de3a73d01ced49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 21:27:44.457731 24115 consensus_peers.cc:485] T 6bd759d700a24e9f8ad11f243cf7ebfa P 009d3e7e8a8b4e2497de3a73d01ced49 -> Peer ee27c3f9e0ec46caa742ac386d6b7c20 (127.15.255.133:40841): Couldn't send request to peer ee27c3f9e0ec46caa742ac386d6b7c20. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6bd759d700a24e9f8ad11f243cf7ebfa. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:44.460027 26475 raft_consensus.cc:2932] T 6bd759d700a24e9f8ad11f243cf7ebfa P 009d3e7e8a8b4e2497de3a73d01ced49 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ee27c3f9e0ec46caa742ac386d6b7c20 (127.15.255.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } attrs { replace: true } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: NON_VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: true } } }
I20240502 21:27:44.460995 24315 raft_consensus.cc:2932] T 6bd759d700a24e9f8ad11f243cf7ebfa P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ee27c3f9e0ec46caa742ac386d6b7c20 (127.15.255.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } attrs { replace: true } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: NON_VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: true } } }
I20240502 21:27:44.463811 23916 raft_consensus.cc:2932] T 6bd759d700a24e9f8ad11f243cf7ebfa P 6810a866decb4f5089f7d35c870e4204 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ee27c3f9e0ec46caa742ac386d6b7c20 (127.15.255.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } attrs { replace: true } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: NON_VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: true } } }
I20240502 21:27:44.467401 23766 catalog_manager.cc:5484] T 6bd759d700a24e9f8ad11f243cf7ebfa P 038c5d9d1d7a49c981c4c34fbc00d9e6 reported cstate change: config changed from index -1 to 4, NON_VOTER ee27c3f9e0ec46caa742ac386d6b7c20 (127.15.255.133) added. New cstate: current_term: 1 leader_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } attrs { replace: true } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: NON_VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: true } } }
I20240502 21:27:44.479733 23916 consensus_queue.cc:237] T 89f86d58fe414b579845214534d338d8 P 6810a866decb4f5089f7d35c870e4204 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } attrs { replace: true } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: NON_VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: true } }
I20240502 21:27:44.482115 24315 raft_consensus.cc:1261] T 89f86d58fe414b579845214534d338d8 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 1 FOLLOWER]: Refusing update from remote peer 6810a866decb4f5089f7d35c870e4204: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:27:44.482266 24185 raft_consensus.cc:1261] T 89f86d58fe414b579845214534d338d8 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 1 FOLLOWER]: Refusing update from remote peer 6810a866decb4f5089f7d35c870e4204: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:27:44.482724 26588 consensus_queue.cc:1035] T 89f86d58fe414b579845214534d338d8 P 6810a866decb4f5089f7d35c870e4204 [LEADER]: Connected to new peer: Peer: permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:27:44.483299 26588 consensus_queue.cc:1035] T 89f86d58fe414b579845214534d338d8 P 6810a866decb4f5089f7d35c870e4204 [LEADER]: Connected to new peer: Peer: permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:27:44.487932 26586 raft_consensus.cc:2932] T 89f86d58fe414b579845214534d338d8 P 6810a866decb4f5089f7d35c870e4204 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER fb6004b8640a4906aa0cfe63e383d40e (127.15.255.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } attrs { replace: true } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: NON_VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: true } } }
W20240502 21:27:44.489729 23847 consensus_peers.cc:485] T 89f86d58fe414b579845214534d338d8 P 6810a866decb4f5089f7d35c870e4204 -> Peer fb6004b8640a4906aa0cfe63e383d40e (127.15.255.134:32825): Couldn't send request to peer fb6004b8640a4906aa0cfe63e383d40e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 89f86d58fe414b579845214534d338d8. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:44.492122 24185 raft_consensus.cc:2932] T 89f86d58fe414b579845214534d338d8 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER fb6004b8640a4906aa0cfe63e383d40e (127.15.255.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } attrs { replace: true } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: NON_VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: true } } }
I20240502 21:27:44.493321 23770 catalog_manager.cc:5484] T 89f86d58fe414b579845214534d338d8 P 6810a866decb4f5089f7d35c870e4204 reported cstate change: config changed from index -1 to 4, NON_VOTER fb6004b8640a4906aa0cfe63e383d40e (127.15.255.134) added. New cstate: current_term: 1 leader_uuid: "6810a866decb4f5089f7d35c870e4204" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: NON_VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:44.502264 24315 raft_consensus.cc:2932] T 89f86d58fe414b579845214534d338d8 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER fb6004b8640a4906aa0cfe63e383d40e (127.15.255.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } attrs { replace: true } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: NON_VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: true } } }
I20240502 21:27:44.517771 23916 consensus_queue.cc:237] T 549666d19e724a96801db183a07809d8 P 6810a866decb4f5089f7d35c870e4204 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } attrs { replace: true } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: true } }
I20240502 21:27:44.520282 24048 raft_consensus.cc:1261] T 549666d19e724a96801db183a07809d8 P 6c194a6db6c145acafc0f3068b8ce1fd [term 1 FOLLOWER]: Refusing update from remote peer 6810a866decb4f5089f7d35c870e4204: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:27:44.521139 26588 consensus_queue.cc:1035] T 549666d19e724a96801db183a07809d8 P 6810a866decb4f5089f7d35c870e4204 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:27:44.521746 24185 raft_consensus.cc:1261] T 549666d19e724a96801db183a07809d8 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 1 FOLLOWER]: Refusing update from remote peer 6810a866decb4f5089f7d35c870e4204: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 21:27:44.521946 23850 consensus_peers.cc:485] T 549666d19e724a96801db183a07809d8 P 6810a866decb4f5089f7d35c870e4204 -> Peer a3e500d7077a48acb5edb33f7e178c81 (127.15.255.136:43595): Couldn't send request to peer a3e500d7077a48acb5edb33f7e178c81. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 549666d19e724a96801db183a07809d8. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:44.522392 26588 consensus_queue.cc:1035] T 549666d19e724a96801db183a07809d8 P 6810a866decb4f5089f7d35c870e4204 [LEADER]: Connected to new peer: Peer: permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:27:44.525586 26586 raft_consensus.cc:2932] T 549666d19e724a96801db183a07809d8 P 6810a866decb4f5089f7d35c870e4204 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a3e500d7077a48acb5edb33f7e178c81 (127.15.255.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } attrs { replace: true } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: true } } }
I20240502 21:27:44.526113 24048 raft_consensus.cc:2932] T 549666d19e724a96801db183a07809d8 P 6c194a6db6c145acafc0f3068b8ce1fd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a3e500d7077a48acb5edb33f7e178c81 (127.15.255.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } attrs { replace: true } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: true } } }
I20240502 21:27:44.531373 23770 catalog_manager.cc:5484] T 549666d19e724a96801db183a07809d8 P 6810a866decb4f5089f7d35c870e4204 reported cstate change: config changed from index -1 to 4, NON_VOTER a3e500d7077a48acb5edb33f7e178c81 (127.15.255.136) added. New cstate: current_term: 1 leader_uuid: "6810a866decb4f5089f7d35c870e4204" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:44.535698 24185 raft_consensus.cc:2932] T 549666d19e724a96801db183a07809d8 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a3e500d7077a48acb5edb33f7e178c81 (127.15.255.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } attrs { replace: true } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: true } } }
I20240502 21:27:44.566221 24315 consensus_queue.cc:237] T 11cb9a2fbda946fb9f303f2c465bccb6 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } attrs { replace: true } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: NON_VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: true } }
I20240502 21:27:44.568951 24048 raft_consensus.cc:1261] T 11cb9a2fbda946fb9f303f2c465bccb6 P 6c194a6db6c145acafc0f3068b8ce1fd [term 1 FOLLOWER]: Refusing update from remote peer 038c5d9d1d7a49c981c4c34fbc00d9e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:27:44.569599 26560 consensus_queue.cc:1035] T 11cb9a2fbda946fb9f303f2c465bccb6 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:27:44.578058 26633 ts_tablet_manager.cc:902] T cca45c89b9fe40e7800d95c1598a9e52 P ee27c3f9e0ec46caa742ac386d6b7c20: Initiating tablet copy from peer 009d3e7e8a8b4e2497de3a73d01ced49 (127.15.255.131:37895)
I20240502 21:27:44.587113 24185 raft_consensus.cc:1261] T 11cb9a2fbda946fb9f303f2c465bccb6 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 1 FOLLOWER]: Refusing update from remote peer 038c5d9d1d7a49c981c4c34fbc00d9e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:27:44.587159 26560 raft_consensus.cc:2932] T 11cb9a2fbda946fb9f303f2c465bccb6 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER e117afbf8819424e8a98476453dfde2a (127.15.255.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } attrs { replace: true } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: NON_VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: true } } }
I20240502 21:27:44.588027 24048 raft_consensus.cc:2932] T 11cb9a2fbda946fb9f303f2c465bccb6 P 6c194a6db6c145acafc0f3068b8ce1fd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER e117afbf8819424e8a98476453dfde2a (127.15.255.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } attrs { replace: true } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: NON_VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: true } } }
I20240502 21:27:44.592296 26560 consensus_queue.cc:1035] T 11cb9a2fbda946fb9f303f2c465bccb6 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:27:44.593148 23766 catalog_manager.cc:5484] T 11cb9a2fbda946fb9f303f2c465bccb6 P 038c5d9d1d7a49c981c4c34fbc00d9e6 reported cstate change: config changed from index -1 to 4, NON_VOTER e117afbf8819424e8a98476453dfde2a (127.15.255.135) added. New cstate: current_term: 1 leader_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: NON_VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240502 21:27:44.605229 24252 consensus_peers.cc:485] T 11cb9a2fbda946fb9f303f2c465bccb6 P 038c5d9d1d7a49c981c4c34fbc00d9e6 -> Peer e117afbf8819424e8a98476453dfde2a (127.15.255.135:32775): Couldn't send request to peer e117afbf8819424e8a98476453dfde2a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 11cb9a2fbda946fb9f303f2c465bccb6. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:44.604120 26633 tablet_copy_client.cc:308] T cca45c89b9fe40e7800d95c1598a9e52 P ee27c3f9e0ec46caa742ac386d6b7c20: tablet copy: Beginning tablet copy session from remote peer at address 127.15.255.131:37895
I20240502 21:27:44.609171 24185 raft_consensus.cc:2932] T 11cb9a2fbda946fb9f303f2c465bccb6 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER e117afbf8819424e8a98476453dfde2a (127.15.255.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } attrs { replace: true } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: NON_VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: true } } }
I20240502 21:27:44.624367 24205 tablet_copy_service.cc:140] P 009d3e7e8a8b4e2497de3a73d01ced49: Received BeginTabletCopySession request for tablet cca45c89b9fe40e7800d95c1598a9e52 from peer ee27c3f9e0ec46caa742ac386d6b7c20 ({username='slave'} at 127.15.255.133:44245)
I20240502 21:27:44.624699 24205 tablet_copy_service.cc:161] P 009d3e7e8a8b4e2497de3a73d01ced49: Beginning new tablet copy session on tablet cca45c89b9fe40e7800d95c1598a9e52 from peer ee27c3f9e0ec46caa742ac386d6b7c20 at {username='slave'} at 127.15.255.133:44245: session id = ee27c3f9e0ec46caa742ac386d6b7c20-cca45c89b9fe40e7800d95c1598a9e52
I20240502 21:27:44.628506 24205 tablet_copy_source_session.cc:205] T cca45c89b9fe40e7800d95c1598a9e52 P 009d3e7e8a8b4e2497de3a73d01ced49: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:27:44.633704 26633 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cca45c89b9fe40e7800d95c1598a9e52. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:44.640590 23916 consensus_queue.cc:237] T 8f14db150fda449e962be2466f1d2a0e P 6810a866decb4f5089f7d35c870e4204 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } attrs { replace: true } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: NON_VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: true } }
I20240502 21:27:44.646802 26633 tablet_copy_client.cc:783] T cca45c89b9fe40e7800d95c1598a9e52 P ee27c3f9e0ec46caa742ac386d6b7c20: tablet copy: Starting download of 0 data blocks...
I20240502 21:27:44.647148 26633 tablet_copy_client.cc:647] T cca45c89b9fe40e7800d95c1598a9e52 P ee27c3f9e0ec46caa742ac386d6b7c20: tablet copy: Starting download of 1 WAL segments...
I20240502 21:27:44.648726 24315 raft_consensus.cc:1261] T 8f14db150fda449e962be2466f1d2a0e P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 1 FOLLOWER]: Refusing update from remote peer 6810a866decb4f5089f7d35c870e4204: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:27:44.649538 26588 consensus_queue.cc:1035] T 8f14db150fda449e962be2466f1d2a0e P 6810a866decb4f5089f7d35c870e4204 [LEADER]: Connected to new peer: Peer: permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:27:44.651916 24185 raft_consensus.cc:1261] T 8f14db150fda449e962be2466f1d2a0e P 009d3e7e8a8b4e2497de3a73d01ced49 [term 1 FOLLOWER]: Refusing update from remote peer 6810a866decb4f5089f7d35c870e4204: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:27:44.652758 26638 tablet_copy_client.cc:1002] T cca45c89b9fe40e7800d95c1598a9e52 P ee27c3f9e0ec46caa742ac386d6b7c20: tablet copy: Injecting latency into file download: 1500
I20240502 21:27:44.652882 26588 consensus_queue.cc:1035] T 8f14db150fda449e962be2466f1d2a0e P 6810a866decb4f5089f7d35c870e4204 [LEADER]: Connected to new peer: Peer: permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 21:27:44.655104 23847 consensus_peers.cc:485] T 8f14db150fda449e962be2466f1d2a0e P 6810a866decb4f5089f7d35c870e4204 -> Peer ee27c3f9e0ec46caa742ac386d6b7c20 (127.15.255.133:40841): Couldn't send request to peer ee27c3f9e0ec46caa742ac386d6b7c20. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8f14db150fda449e962be2466f1d2a0e. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:44.654826 26588 raft_consensus.cc:2932] T 8f14db150fda449e962be2466f1d2a0e P 6810a866decb4f5089f7d35c870e4204 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ee27c3f9e0ec46caa742ac386d6b7c20 (127.15.255.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } attrs { replace: true } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: NON_VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: true } } }
I20240502 21:27:44.659253 24315 raft_consensus.cc:2932] T 8f14db150fda449e962be2466f1d2a0e P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ee27c3f9e0ec46caa742ac386d6b7c20 (127.15.255.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } attrs { replace: true } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: NON_VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: true } } }
I20240502 21:27:44.660797 23770 catalog_manager.cc:5484] T 8f14db150fda449e962be2466f1d2a0e P 6810a866decb4f5089f7d35c870e4204 reported cstate change: config changed from index -1 to 4, NON_VOTER ee27c3f9e0ec46caa742ac386d6b7c20 (127.15.255.133) added. New cstate: current_term: 1 leader_uuid: "6810a866decb4f5089f7d35c870e4204" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: NON_VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:44.677055 24184 raft_consensus.cc:2932] T 8f14db150fda449e962be2466f1d2a0e P 009d3e7e8a8b4e2497de3a73d01ced49 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ee27c3f9e0ec46caa742ac386d6b7c20 (127.15.255.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } attrs { replace: true } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: NON_VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: true } } }
I20240502 21:27:44.690402 24185 consensus_queue.cc:237] T a2a0537a60f849daae2061c2b0c4c89f P 009d3e7e8a8b4e2497de3a73d01ced49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } attrs { replace: true } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: true } }
I20240502 21:27:44.692935 24048 raft_consensus.cc:1261] T a2a0537a60f849daae2061c2b0c4c89f P 6c194a6db6c145acafc0f3068b8ce1fd [term 1 FOLLOWER]: Refusing update from remote peer 009d3e7e8a8b4e2497de3a73d01ced49: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:27:44.693753 26475 consensus_queue.cc:1035] T a2a0537a60f849daae2061c2b0c4c89f P 009d3e7e8a8b4e2497de3a73d01ced49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:27:44.693989 23916 raft_consensus.cc:1261] T a2a0537a60f849daae2061c2b0c4c89f P 6810a866decb4f5089f7d35c870e4204 [term 1 FOLLOWER]: Refusing update from remote peer 009d3e7e8a8b4e2497de3a73d01ced49: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:27:44.694505 26475 consensus_queue.cc:1035] T a2a0537a60f849daae2061c2b0c4c89f P 009d3e7e8a8b4e2497de3a73d01ced49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:27:44.697523 26475 raft_consensus.cc:2932] T a2a0537a60f849daae2061c2b0c4c89f P 009d3e7e8a8b4e2497de3a73d01ced49 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a3e500d7077a48acb5edb33f7e178c81 (127.15.255.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } attrs { replace: true } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: true } } }
W20240502 21:27:44.700616 24117 consensus_peers.cc:485] T a2a0537a60f849daae2061c2b0c4c89f P 009d3e7e8a8b4e2497de3a73d01ced49 -> Peer a3e500d7077a48acb5edb33f7e178c81 (127.15.255.136:43595): Couldn't send request to peer a3e500d7077a48acb5edb33f7e178c81. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a2a0537a60f849daae2061c2b0c4c89f. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:44.701634 23916 raft_consensus.cc:2932] T a2a0537a60f849daae2061c2b0c4c89f P 6810a866decb4f5089f7d35c870e4204 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a3e500d7077a48acb5edb33f7e178c81 (127.15.255.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } attrs { replace: true } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: true } } }
I20240502 21:27:44.703531 23766 catalog_manager.cc:5484] T a2a0537a60f849daae2061c2b0c4c89f P 009d3e7e8a8b4e2497de3a73d01ced49 reported cstate change: config changed from index -1 to 4, NON_VOTER a3e500d7077a48acb5edb33f7e178c81 (127.15.255.136) added. New cstate: current_term: 1 leader_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:44.716444 24048 raft_consensus.cc:2932] T a2a0537a60f849daae2061c2b0c4c89f P 6c194a6db6c145acafc0f3068b8ce1fd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a3e500d7077a48acb5edb33f7e178c81 (127.15.255.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } attrs { replace: true } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: true } } }
I20240502 21:27:44.731788 23916 consensus_queue.cc:237] T 906cfe7530754271bb1ee57c470109bd P 6810a866decb4f5089f7d35c870e4204 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } attrs { replace: true } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: NON_VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: true } }
I20240502 21:27:44.734232 24048 raft_consensus.cc:1261] T 906cfe7530754271bb1ee57c470109bd P 6c194a6db6c145acafc0f3068b8ce1fd [term 1 FOLLOWER]: Refusing update from remote peer 6810a866decb4f5089f7d35c870e4204: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:27:44.734531 24315 raft_consensus.cc:1261] T 906cfe7530754271bb1ee57c470109bd P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 1 FOLLOWER]: Refusing update from remote peer 6810a866decb4f5089f7d35c870e4204: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:27:44.735065 26586 consensus_queue.cc:1035] T 906cfe7530754271bb1ee57c470109bd P 6810a866decb4f5089f7d35c870e4204 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:27:44.735488 26588 consensus_queue.cc:1035] T 906cfe7530754271bb1ee57c470109bd P 6810a866decb4f5089f7d35c870e4204 [LEADER]: Connected to new peer: Peer: permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 21:27:44.736363 23847 consensus_peers.cc:485] T 906cfe7530754271bb1ee57c470109bd P 6810a866decb4f5089f7d35c870e4204 -> Peer fb6004b8640a4906aa0cfe63e383d40e (127.15.255.134:32825): Couldn't send request to peer fb6004b8640a4906aa0cfe63e383d40e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 906cfe7530754271bb1ee57c470109bd. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:44.747378 26664 ts_tablet_manager.cc:902] T 0b3c514213bb40ca98955d6409a3b372 P fb6004b8640a4906aa0cfe63e383d40e: Initiating tablet copy from peer 009d3e7e8a8b4e2497de3a73d01ced49 (127.15.255.131:37895)
I20240502 21:27:44.747784 26471 raft_consensus.cc:2932] T 906cfe7530754271bb1ee57c470109bd P 6810a866decb4f5089f7d35c870e4204 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER fb6004b8640a4906aa0cfe63e383d40e (127.15.255.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } attrs { replace: true } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: NON_VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: true } } }
I20240502 21:27:44.748579 24316 raft_consensus.cc:2932] T 906cfe7530754271bb1ee57c470109bd P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER fb6004b8640a4906aa0cfe63e383d40e (127.15.255.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } attrs { replace: true } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: NON_VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: true } } }
I20240502 21:27:44.755057 23766 catalog_manager.cc:5484] T 906cfe7530754271bb1ee57c470109bd P 6810a866decb4f5089f7d35c870e4204 reported cstate change: config changed from index -1 to 4, NON_VOTER fb6004b8640a4906aa0cfe63e383d40e (127.15.255.134) added. New cstate: current_term: 1 leader_uuid: "6810a866decb4f5089f7d35c870e4204" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: NON_VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:44.765115 26666 ts_tablet_manager.cc:902] T 4eebf4508766421eb6c547a0fbbc28b1 P e117afbf8819424e8a98476453dfde2a: Initiating tablet copy from peer 6810a866decb4f5089f7d35c870e4204 (127.15.255.129:40435)
I20240502 21:27:44.769255 24047 raft_consensus.cc:2932] T 906cfe7530754271bb1ee57c470109bd P 6c194a6db6c145acafc0f3068b8ce1fd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER fb6004b8640a4906aa0cfe63e383d40e (127.15.255.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } attrs { replace: true } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: NON_VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: true } } }
I20240502 21:27:44.776863 26664 tablet_copy_client.cc:308] T 0b3c514213bb40ca98955d6409a3b372 P fb6004b8640a4906aa0cfe63e383d40e: tablet copy: Beginning tablet copy session from remote peer at address 127.15.255.131:37895
I20240502 21:27:44.802582 24315 consensus_queue.cc:237] T fe044374ad7b4999afc72ab812466749 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } attrs { replace: true } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: NON_VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: true } }
I20240502 21:27:44.803280 24205 tablet_copy_service.cc:140] P 009d3e7e8a8b4e2497de3a73d01ced49: Received BeginTabletCopySession request for tablet 0b3c514213bb40ca98955d6409a3b372 from peer fb6004b8640a4906aa0cfe63e383d40e ({username='slave'} at 127.15.255.134:41395)
I20240502 21:27:44.803453 24205 tablet_copy_service.cc:161] P 009d3e7e8a8b4e2497de3a73d01ced49: Beginning new tablet copy session on tablet 0b3c514213bb40ca98955d6409a3b372 from peer fb6004b8640a4906aa0cfe63e383d40e at {username='slave'} at 127.15.255.134:41395: session id = fb6004b8640a4906aa0cfe63e383d40e-0b3c514213bb40ca98955d6409a3b372
I20240502 21:27:44.806543 24205 tablet_copy_source_session.cc:205] T 0b3c514213bb40ca98955d6409a3b372 P 009d3e7e8a8b4e2497de3a73d01ced49: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:27:44.806970 26666 tablet_copy_client.cc:308] T 4eebf4508766421eb6c547a0fbbc28b1 P e117afbf8819424e8a98476453dfde2a: tablet copy: Beginning tablet copy session from remote peer at address 127.15.255.129:40435
I20240502 21:27:44.817986 26664 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b3c514213bb40ca98955d6409a3b372. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:44.818462 24185 raft_consensus.cc:1261] T fe044374ad7b4999afc72ab812466749 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 1 FOLLOWER]: Refusing update from remote peer 038c5d9d1d7a49c981c4c34fbc00d9e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:27:44.819435 23916 raft_consensus.cc:1261] T fe044374ad7b4999afc72ab812466749 P 6810a866decb4f5089f7d35c870e4204 [term 1 FOLLOWER]: Refusing update from remote peer 038c5d9d1d7a49c981c4c34fbc00d9e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:27:44.820398 26560 consensus_queue.cc:1035] T fe044374ad7b4999afc72ab812466749 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:27:44.821086 26560 consensus_queue.cc:1035] T fe044374ad7b4999afc72ab812466749 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 21:27:44.823884 24250 consensus_peers.cc:485] T fe044374ad7b4999afc72ab812466749 P 038c5d9d1d7a49c981c4c34fbc00d9e6 -> Peer ee27c3f9e0ec46caa742ac386d6b7c20 (127.15.255.133:40841): Couldn't send request to peer ee27c3f9e0ec46caa742ac386d6b7c20. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fe044374ad7b4999afc72ab812466749. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:44.824643 23936 tablet_copy_service.cc:140] P 6810a866decb4f5089f7d35c870e4204: Received BeginTabletCopySession request for tablet 4eebf4508766421eb6c547a0fbbc28b1 from peer e117afbf8819424e8a98476453dfde2a ({username='slave'} at 127.15.255.135:39893)
I20240502 21:27:44.824950 23936 tablet_copy_service.cc:161] P 6810a866decb4f5089f7d35c870e4204: Beginning new tablet copy session on tablet 4eebf4508766421eb6c547a0fbbc28b1 from peer e117afbf8819424e8a98476453dfde2a at {username='slave'} at 127.15.255.135:39893: session id = e117afbf8819424e8a98476453dfde2a-4eebf4508766421eb6c547a0fbbc28b1
I20240502 21:27:44.826685 26630 raft_consensus.cc:2932] T fe044374ad7b4999afc72ab812466749 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ee27c3f9e0ec46caa742ac386d6b7c20 (127.15.255.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } attrs { replace: true } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: NON_VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: true } } }
I20240502 21:27:44.827112 24185 raft_consensus.cc:2932] T fe044374ad7b4999afc72ab812466749 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ee27c3f9e0ec46caa742ac386d6b7c20 (127.15.255.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } attrs { replace: true } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: NON_VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: true } } }
I20240502 21:27:44.828800 23936 tablet_copy_source_session.cc:205] T 4eebf4508766421eb6c547a0fbbc28b1 P 6810a866decb4f5089f7d35c870e4204: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:27:44.834374 26666 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4eebf4508766421eb6c547a0fbbc28b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:44.836057 23770 catalog_manager.cc:5484] T fe044374ad7b4999afc72ab812466749 P 009d3e7e8a8b4e2497de3a73d01ced49 reported cstate change: config changed from index -1 to 4, NON_VOTER ee27c3f9e0ec46caa742ac386d6b7c20 (127.15.255.133) added. New cstate: current_term: 1 leader_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } attrs { replace: true } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: NON_VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: true } } }
I20240502 21:27:44.850951 26664 tablet_copy_client.cc:783] T 0b3c514213bb40ca98955d6409a3b372 P fb6004b8640a4906aa0cfe63e383d40e: tablet copy: Starting download of 0 data blocks...
I20240502 21:27:44.851307 26664 tablet_copy_client.cc:647] T 0b3c514213bb40ca98955d6409a3b372 P fb6004b8640a4906aa0cfe63e383d40e: tablet copy: Starting download of 1 WAL segments...
I20240502 21:27:44.853628 26674 tablet_copy_client.cc:1002] T 0b3c514213bb40ca98955d6409a3b372 P fb6004b8640a4906aa0cfe63e383d40e: tablet copy: Injecting latency into file download: 1500
I20240502 21:27:44.859879 23916 raft_consensus.cc:2932] T fe044374ad7b4999afc72ab812466749 P 6810a866decb4f5089f7d35c870e4204 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ee27c3f9e0ec46caa742ac386d6b7c20 (127.15.255.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } attrs { replace: true } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: NON_VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: true } } }
I20240502 21:27:44.876099 26666 tablet_copy_client.cc:783] T 4eebf4508766421eb6c547a0fbbc28b1 P e117afbf8819424e8a98476453dfde2a: tablet copy: Starting download of 0 data blocks...
I20240502 21:27:44.876492 26666 tablet_copy_client.cc:647] T 4eebf4508766421eb6c547a0fbbc28b1 P e117afbf8819424e8a98476453dfde2a: tablet copy: Starting download of 1 WAL segments...
I20240502 21:27:44.879568 26681 tablet_copy_client.cc:1002] T 4eebf4508766421eb6c547a0fbbc28b1 P e117afbf8819424e8a98476453dfde2a: tablet copy: Injecting latency into file download: 1500
I20240502 21:27:44.880913 26689 ts_tablet_manager.cc:902] T 89f86d58fe414b579845214534d338d8 P fb6004b8640a4906aa0cfe63e383d40e: Initiating tablet copy from peer 6810a866decb4f5089f7d35c870e4204 (127.15.255.129:40435)
I20240502 21:27:44.882081 26687 ts_tablet_manager.cc:902] T adf80b9d71314010a7a4cf052fd8a511 P a3e500d7077a48acb5edb33f7e178c81: Initiating tablet copy from peer 6810a866decb4f5089f7d35c870e4204 (127.15.255.129:40435)
I20240502 21:27:44.884980 26689 tablet_copy_client.cc:308] T 89f86d58fe414b579845214534d338d8 P fb6004b8640a4906aa0cfe63e383d40e: tablet copy: Beginning tablet copy session from remote peer at address 127.15.255.129:40435
I20240502 21:27:44.885450 26687 tablet_copy_client.cc:308] T adf80b9d71314010a7a4cf052fd8a511 P a3e500d7077a48acb5edb33f7e178c81: tablet copy: Beginning tablet copy session from remote peer at address 127.15.255.129:40435
I20240502 21:27:44.893112 23936 tablet_copy_service.cc:140] P 6810a866decb4f5089f7d35c870e4204: Received BeginTabletCopySession request for tablet 89f86d58fe414b579845214534d338d8 from peer fb6004b8640a4906aa0cfe63e383d40e ({username='slave'} at 127.15.255.134:44215)
I20240502 21:27:44.893298 23936 tablet_copy_service.cc:161] P 6810a866decb4f5089f7d35c870e4204: Beginning new tablet copy session on tablet 89f86d58fe414b579845214534d338d8 from peer fb6004b8640a4906aa0cfe63e383d40e at {username='slave'} at 127.15.255.134:44215: session id = fb6004b8640a4906aa0cfe63e383d40e-89f86d58fe414b579845214534d338d8
I20240502 21:27:44.896239 23936 tablet_copy_source_session.cc:205] T 89f86d58fe414b579845214534d338d8 P 6810a866decb4f5089f7d35c870e4204: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:27:44.897598 26689 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89f86d58fe414b579845214534d338d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:44.904655 26689 tablet_copy_client.cc:783] T 89f86d58fe414b579845214534d338d8 P fb6004b8640a4906aa0cfe63e383d40e: tablet copy: Starting download of 0 data blocks...
I20240502 21:27:44.905005 26689 tablet_copy_client.cc:647] T 89f86d58fe414b579845214534d338d8 P fb6004b8640a4906aa0cfe63e383d40e: tablet copy: Starting download of 1 WAL segments...
I20240502 21:27:44.906947 26696 tablet_copy_client.cc:1002] T 89f86d58fe414b579845214534d338d8 P fb6004b8640a4906aa0cfe63e383d40e: tablet copy: Injecting latency into file download: 1500
I20240502 21:27:44.917035 23936 tablet_copy_service.cc:140] P 6810a866decb4f5089f7d35c870e4204: Received BeginTabletCopySession request for tablet adf80b9d71314010a7a4cf052fd8a511 from peer a3e500d7077a48acb5edb33f7e178c81 ({username='slave'} at 127.15.255.136:46499)
I20240502 21:27:44.917207 23936 tablet_copy_service.cc:161] P 6810a866decb4f5089f7d35c870e4204: Beginning new tablet copy session on tablet adf80b9d71314010a7a4cf052fd8a511 from peer a3e500d7077a48acb5edb33f7e178c81 at {username='slave'} at 127.15.255.136:46499: session id = a3e500d7077a48acb5edb33f7e178c81-adf80b9d71314010a7a4cf052fd8a511
I20240502 21:27:44.920279 23936 tablet_copy_source_session.cc:205] T adf80b9d71314010a7a4cf052fd8a511 P 6810a866decb4f5089f7d35c870e4204: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:27:44.925796 26687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet adf80b9d71314010a7a4cf052fd8a511. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:44.929203 26704 ts_tablet_manager.cc:902] T 549666d19e724a96801db183a07809d8 P a3e500d7077a48acb5edb33f7e178c81: Initiating tablet copy from peer 6810a866decb4f5089f7d35c870e4204 (127.15.255.129:40435)
I20240502 21:27:44.941283 24047 consensus_queue.cc:237] T a274546743354e79bffa26ae7000e895 P 6c194a6db6c145acafc0f3068b8ce1fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } attrs { replace: true } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: NON_VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: true } }
I20240502 21:27:44.941740 26687 tablet_copy_client.cc:783] T adf80b9d71314010a7a4cf052fd8a511 P a3e500d7077a48acb5edb33f7e178c81: tablet copy: Starting download of 0 data blocks...
I20240502 21:27:44.942308 26687 tablet_copy_client.cc:647] T adf80b9d71314010a7a4cf052fd8a511 P a3e500d7077a48acb5edb33f7e178c81: tablet copy: Starting download of 1 WAL segments...
I20240502 21:27:44.944702 26697 tablet_copy_client.cc:1002] T adf80b9d71314010a7a4cf052fd8a511 P a3e500d7077a48acb5edb33f7e178c81: tablet copy: Injecting latency into file download: 1500
I20240502 21:27:44.946622 24315 raft_consensus.cc:1261] T a274546743354e79bffa26ae7000e895 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 1 FOLLOWER]: Refusing update from remote peer 6c194a6db6c145acafc0f3068b8ce1fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:27:44.947408 26334 consensus_queue.cc:1035] T a274546743354e79bffa26ae7000e895 P 6c194a6db6c145acafc0f3068b8ce1fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 21:27:44.958746 23982 consensus_peers.cc:485] T a274546743354e79bffa26ae7000e895 P 6c194a6db6c145acafc0f3068b8ce1fd -> Peer e117afbf8819424e8a98476453dfde2a (127.15.255.135:32775): Couldn't send request to peer e117afbf8819424e8a98476453dfde2a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a274546743354e79bffa26ae7000e895. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:44.960412 26704 tablet_copy_client.cc:308] T 549666d19e724a96801db183a07809d8 P a3e500d7077a48acb5edb33f7e178c81: tablet copy: Beginning tablet copy session from remote peer at address 127.15.255.129:40435
I20240502 21:27:44.961328 23936 tablet_copy_service.cc:140] P 6810a866decb4f5089f7d35c870e4204: Received BeginTabletCopySession request for tablet 549666d19e724a96801db183a07809d8 from peer a3e500d7077a48acb5edb33f7e178c81 ({username='slave'} at 127.15.255.136:46499)
I20240502 21:27:44.961503 23936 tablet_copy_service.cc:161] P 6810a866decb4f5089f7d35c870e4204: Beginning new tablet copy session on tablet 549666d19e724a96801db183a07809d8 from peer a3e500d7077a48acb5edb33f7e178c81 at {username='slave'} at 127.15.255.136:46499: session id = a3e500d7077a48acb5edb33f7e178c81-549666d19e724a96801db183a07809d8
I20240502 21:27:44.961917 26334 raft_consensus.cc:2932] T a274546743354e79bffa26ae7000e895 P 6c194a6db6c145acafc0f3068b8ce1fd [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER e117afbf8819424e8a98476453dfde2a (127.15.255.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } attrs { replace: true } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: NON_VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: true } } }
I20240502 21:27:44.964368 23936 tablet_copy_source_session.cc:205] T 549666d19e724a96801db183a07809d8 P 6810a866decb4f5089f7d35c870e4204: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:27:44.965221 24316 raft_consensus.cc:2932] T a274546743354e79bffa26ae7000e895 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER e117afbf8819424e8a98476453dfde2a (127.15.255.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } attrs { replace: true } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: NON_VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: true } } }
I20240502 21:27:44.969808 26704 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 549666d19e724a96801db183a07809d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:44.978654 26704 tablet_copy_client.cc:783] T 549666d19e724a96801db183a07809d8 P a3e500d7077a48acb5edb33f7e178c81: tablet copy: Starting download of 0 data blocks...
I20240502 21:27:44.978452 23770 catalog_manager.cc:5484] T a274546743354e79bffa26ae7000e895 P 038c5d9d1d7a49c981c4c34fbc00d9e6 reported cstate change: config changed from index -1 to 4, NON_VOTER e117afbf8819424e8a98476453dfde2a (127.15.255.135) added. New cstate: current_term: 1 leader_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } attrs { replace: true } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: NON_VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: true } } }
I20240502 21:27:44.978997 26704 tablet_copy_client.cc:647] T 549666d19e724a96801db183a07809d8 P a3e500d7077a48acb5edb33f7e178c81: tablet copy: Starting download of 1 WAL segments...
I20240502 21:27:44.980914 26712 tablet_copy_client.cc:1002] T 549666d19e724a96801db183a07809d8 P a3e500d7077a48acb5edb33f7e178c81: tablet copy: Injecting latency into file download: 1500
I20240502 21:27:45.006104 26713 ts_tablet_manager.cc:902] T 6bd759d700a24e9f8ad11f243cf7ebfa P ee27c3f9e0ec46caa742ac386d6b7c20: Initiating tablet copy from peer 009d3e7e8a8b4e2497de3a73d01ced49 (127.15.255.131:37895)
I20240502 21:27:45.010957 24185 raft_consensus.cc:1261] T a274546743354e79bffa26ae7000e895 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 1 FOLLOWER]: Refusing update from remote peer 6c194a6db6c145acafc0f3068b8ce1fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:27:45.011801 26714 consensus_queue.cc:1035] T a274546743354e79bffa26ae7000e895 P 6c194a6db6c145acafc0f3068b8ce1fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 21:27:45.018254 23981 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.132:46813, user_credentials={real_user=slave}} blocked reactor thread for 46497.1us
I20240502 21:27:45.026896 24185 raft_consensus.cc:2932] T a274546743354e79bffa26ae7000e895 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER e117afbf8819424e8a98476453dfde2a (127.15.255.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } attrs { replace: true } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: NON_VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: true } } }
I20240502 21:27:45.049381 26713 tablet_copy_client.cc:308] T 6bd759d700a24e9f8ad11f243cf7ebfa P ee27c3f9e0ec46caa742ac386d6b7c20: tablet copy: Beginning tablet copy session from remote peer at address 127.15.255.131:37895
I20240502 21:27:45.050580 24205 tablet_copy_service.cc:140] P 009d3e7e8a8b4e2497de3a73d01ced49: Received BeginTabletCopySession request for tablet 6bd759d700a24e9f8ad11f243cf7ebfa from peer ee27c3f9e0ec46caa742ac386d6b7c20 ({username='slave'} at 127.15.255.133:44245)
I20240502 21:27:45.050756 24205 tablet_copy_service.cc:161] P 009d3e7e8a8b4e2497de3a73d01ced49: Beginning new tablet copy session on tablet 6bd759d700a24e9f8ad11f243cf7ebfa from peer ee27c3f9e0ec46caa742ac386d6b7c20 at {username='slave'} at 127.15.255.133:44245: session id = ee27c3f9e0ec46caa742ac386d6b7c20-6bd759d700a24e9f8ad11f243cf7ebfa
I20240502 21:27:45.053712 24205 tablet_copy_source_session.cc:205] T 6bd759d700a24e9f8ad11f243cf7ebfa P 009d3e7e8a8b4e2497de3a73d01ced49: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:27:45.059381 24047 consensus_queue.cc:237] T 3cb3e62ca0e94e659efb3b7b0c2c9c91 P 6c194a6db6c145acafc0f3068b8ce1fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } attrs { replace: true } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: NON_VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: true } }
I20240502 21:27:45.061256 26713 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6bd759d700a24e9f8ad11f243cf7ebfa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:45.068337 26713 tablet_copy_client.cc:783] T 6bd759d700a24e9f8ad11f243cf7ebfa P ee27c3f9e0ec46caa742ac386d6b7c20: tablet copy: Starting download of 0 data blocks...
I20240502 21:27:45.068683 26713 tablet_copy_client.cc:647] T 6bd759d700a24e9f8ad11f243cf7ebfa P ee27c3f9e0ec46caa742ac386d6b7c20: tablet copy: Starting download of 1 WAL segments...
W20240502 21:27:45.070252 26624 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:27:45.070641 26624 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240502 21:27:45.071329 26728 tablet_copy_client.cc:1002] T 6bd759d700a24e9f8ad11f243cf7ebfa P ee27c3f9e0ec46caa742ac386d6b7c20: tablet copy: Injecting latency into file download: 1500
I20240502 21:27:45.073765 24185 raft_consensus.cc:1261] T 3cb3e62ca0e94e659efb3b7b0c2c9c91 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 1 FOLLOWER]: Refusing update from remote peer 6c194a6db6c145acafc0f3068b8ce1fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 21:27:45.074072 23982 consensus_peers.cc:485] T 3cb3e62ca0e94e659efb3b7b0c2c9c91 P 6c194a6db6c145acafc0f3068b8ce1fd -> Peer e117afbf8819424e8a98476453dfde2a (127.15.255.135:32775): Couldn't send request to peer e117afbf8819424e8a98476453dfde2a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3cb3e62ca0e94e659efb3b7b0c2c9c91. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:45.074079 23916 raft_consensus.cc:1261] T 3cb3e62ca0e94e659efb3b7b0c2c9c91 P 6810a866decb4f5089f7d35c870e4204 [term 1 FOLLOWER]: Refusing update from remote peer 6c194a6db6c145acafc0f3068b8ce1fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:27:45.074386 26717 consensus_queue.cc:1035] T 3cb3e62ca0e94e659efb3b7b0c2c9c91 P 6c194a6db6c145acafc0f3068b8ce1fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:27:45.074738 26715 consensus_queue.cc:1035] T 3cb3e62ca0e94e659efb3b7b0c2c9c91 P 6c194a6db6c145acafc0f3068b8ce1fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 21:27:45.081907 26624 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 21:27:45.081912 26717 raft_consensus.cc:2932] T 3cb3e62ca0e94e659efb3b7b0c2c9c91 P 6c194a6db6c145acafc0f3068b8ce1fd [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER e117afbf8819424e8a98476453dfde2a (127.15.255.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } attrs { replace: true } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: NON_VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: true } } }
I20240502 21:27:45.083556 23916 raft_consensus.cc:2932] T 3cb3e62ca0e94e659efb3b7b0c2c9c91 P 6810a866decb4f5089f7d35c870e4204 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER e117afbf8819424e8a98476453dfde2a (127.15.255.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } attrs { replace: true } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: NON_VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: true } } }
I20240502 21:27:45.083601 24185 raft_consensus.cc:2932] T 3cb3e62ca0e94e659efb3b7b0c2c9c91 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER e117afbf8819424e8a98476453dfde2a (127.15.255.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } attrs { replace: true } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: NON_VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: true } } }
I20240502 21:27:45.089766 23770 catalog_manager.cc:5484] T 3cb3e62ca0e94e659efb3b7b0c2c9c91 P 6c194a6db6c145acafc0f3068b8ce1fd reported cstate change: config changed from index -1 to 4, NON_VOTER e117afbf8819424e8a98476453dfde2a (127.15.255.135) added. New cstate: current_term: 1 leader_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: NON_VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:45.124413 24316 consensus_queue.cc:237] T d5697d8b06094211b261f144de9d46f8 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } attrs { replace: true } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: true } }
I20240502 21:27:45.129011 24047 raft_consensus.cc:1261] T d5697d8b06094211b261f144de9d46f8 P 6c194a6db6c145acafc0f3068b8ce1fd [term 1 FOLLOWER]: Refusing update from remote peer 038c5d9d1d7a49c981c4c34fbc00d9e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:27:45.130437 24185 raft_consensus.cc:1261] T d5697d8b06094211b261f144de9d46f8 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 1 FOLLOWER]: Refusing update from remote peer 038c5d9d1d7a49c981c4c34fbc00d9e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:27:45.131271 26560 consensus_queue.cc:1035] T d5697d8b06094211b261f144de9d46f8 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:27:45.132715 26630 consensus_queue.cc:1035] T d5697d8b06094211b261f144de9d46f8 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:27:45.141382 26630 raft_consensus.cc:2932] T d5697d8b06094211b261f144de9d46f8 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a3e500d7077a48acb5edb33f7e178c81 (127.15.255.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } attrs { replace: true } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: true } } }
W20240502 21:27:45.142982 24252 consensus_peers.cc:485] T d5697d8b06094211b261f144de9d46f8 P 038c5d9d1d7a49c981c4c34fbc00d9e6 -> Peer a3e500d7077a48acb5edb33f7e178c81 (127.15.255.136:43595): Couldn't send request to peer a3e500d7077a48acb5edb33f7e178c81. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d5697d8b06094211b261f144de9d46f8. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:45.144742 24047 raft_consensus.cc:2932] T d5697d8b06094211b261f144de9d46f8 P 6c194a6db6c145acafc0f3068b8ce1fd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a3e500d7077a48acb5edb33f7e178c81 (127.15.255.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } attrs { replace: true } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: true } } }
I20240502 21:27:45.145855 26745 ts_tablet_manager.cc:902] T 11cb9a2fbda946fb9f303f2c465bccb6 P e117afbf8819424e8a98476453dfde2a: Initiating tablet copy from peer 038c5d9d1d7a49c981c4c34fbc00d9e6 (127.15.255.132:46813)
I20240502 21:27:45.144196 24185 raft_consensus.cc:2932] T d5697d8b06094211b261f144de9d46f8 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a3e500d7077a48acb5edb33f7e178c81 (127.15.255.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } attrs { replace: true } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: true } } }
I20240502 21:27:45.150460 23766 catalog_manager.cc:5484] T d5697d8b06094211b261f144de9d46f8 P 038c5d9d1d7a49c981c4c34fbc00d9e6 reported cstate change: config changed from index -1 to 4, NON_VOTER a3e500d7077a48acb5edb33f7e178c81 (127.15.255.136) added. New cstate: current_term: 1 leader_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:45.161069 26752 ts_tablet_manager.cc:902] T 8f14db150fda449e962be2466f1d2a0e P ee27c3f9e0ec46caa742ac386d6b7c20: Initiating tablet copy from peer 6810a866decb4f5089f7d35c870e4204 (127.15.255.129:40435)
I20240502 21:27:45.165820 26752 tablet_copy_client.cc:308] T 8f14db150fda449e962be2466f1d2a0e P ee27c3f9e0ec46caa742ac386d6b7c20: tablet copy: Beginning tablet copy session from remote peer at address 127.15.255.129:40435
I20240502 21:27:45.197494 26761 ts_tablet_manager.cc:902] T 906cfe7530754271bb1ee57c470109bd P fb6004b8640a4906aa0cfe63e383d40e: Initiating tablet copy from peer 6810a866decb4f5089f7d35c870e4204 (127.15.255.129:40435)
I20240502 21:27:45.200834 26745 tablet_copy_client.cc:308] T 11cb9a2fbda946fb9f303f2c465bccb6 P e117afbf8819424e8a98476453dfde2a: tablet copy: Beginning tablet copy session from remote peer at address 127.15.255.132:46813
I20240502 21:27:45.208617 24338 tablet_copy_service.cc:140] P 038c5d9d1d7a49c981c4c34fbc00d9e6: Received BeginTabletCopySession request for tablet 11cb9a2fbda946fb9f303f2c465bccb6 from peer e117afbf8819424e8a98476453dfde2a ({username='slave'} at 127.15.255.135:44869)
I20240502 21:27:45.208894 24338 tablet_copy_service.cc:161] P 038c5d9d1d7a49c981c4c34fbc00d9e6: Beginning new tablet copy session on tablet 11cb9a2fbda946fb9f303f2c465bccb6 from peer e117afbf8819424e8a98476453dfde2a at {username='slave'} at 127.15.255.135:44869: session id = e117afbf8819424e8a98476453dfde2a-11cb9a2fbda946fb9f303f2c465bccb6
I20240502 21:27:45.210201 23936 tablet_copy_service.cc:140] P 6810a866decb4f5089f7d35c870e4204: Received BeginTabletCopySession request for tablet 8f14db150fda449e962be2466f1d2a0e from peer ee27c3f9e0ec46caa742ac386d6b7c20 ({username='slave'} at 127.15.255.133:36415)
I20240502 21:27:45.210348 23936 tablet_copy_service.cc:161] P 6810a866decb4f5089f7d35c870e4204: Beginning new tablet copy session on tablet 8f14db150fda449e962be2466f1d2a0e from peer ee27c3f9e0ec46caa742ac386d6b7c20 at {username='slave'} at 127.15.255.133:36415: session id = ee27c3f9e0ec46caa742ac386d6b7c20-8f14db150fda449e962be2466f1d2a0e
I20240502 21:27:45.212772 24338 tablet_copy_source_session.cc:205] T 11cb9a2fbda946fb9f303f2c465bccb6 P 038c5d9d1d7a49c981c4c34fbc00d9e6: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:27:45.213181 23936 tablet_copy_source_session.cc:205] T 8f14db150fda449e962be2466f1d2a0e P 6810a866decb4f5089f7d35c870e4204: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:27:45.215058 26745 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11cb9a2fbda946fb9f303f2c465bccb6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:45.218864 26752 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f14db150fda449e962be2466f1d2a0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:45.221410 26745 tablet_copy_client.cc:783] T 11cb9a2fbda946fb9f303f2c465bccb6 P e117afbf8819424e8a98476453dfde2a: tablet copy: Starting download of 0 data blocks...
I20240502 21:27:45.221671 26745 tablet_copy_client.cc:647] T 11cb9a2fbda946fb9f303f2c465bccb6 P e117afbf8819424e8a98476453dfde2a: tablet copy: Starting download of 1 WAL segments...
I20240502 21:27:45.223627 26759 tablet_copy_client.cc:1002] T 11cb9a2fbda946fb9f303f2c465bccb6 P e117afbf8819424e8a98476453dfde2a: tablet copy: Injecting latency into file download: 1500
I20240502 21:27:45.225114 26752 tablet_copy_client.cc:783] T 8f14db150fda449e962be2466f1d2a0e P ee27c3f9e0ec46caa742ac386d6b7c20: tablet copy: Starting download of 0 data blocks...
I20240502 21:27:45.225355 26752 tablet_copy_client.cc:647] T 8f14db150fda449e962be2466f1d2a0e P ee27c3f9e0ec46caa742ac386d6b7c20: tablet copy: Starting download of 1 WAL segments...
I20240502 21:27:45.227489 26761 tablet_copy_client.cc:308] T 906cfe7530754271bb1ee57c470109bd P fb6004b8640a4906aa0cfe63e383d40e: tablet copy: Beginning tablet copy session from remote peer at address 127.15.255.129:40435
I20240502 21:27:45.228477 23935 tablet_copy_service.cc:140] P 6810a866decb4f5089f7d35c870e4204: Received BeginTabletCopySession request for tablet 906cfe7530754271bb1ee57c470109bd from peer fb6004b8640a4906aa0cfe63e383d40e ({username='slave'} at 127.15.255.134:44215)
I20240502 21:27:45.228680 23935 tablet_copy_service.cc:161] P 6810a866decb4f5089f7d35c870e4204: Beginning new tablet copy session on tablet 906cfe7530754271bb1ee57c470109bd from peer fb6004b8640a4906aa0cfe63e383d40e at {username='slave'} at 127.15.255.134:44215: session id = fb6004b8640a4906aa0cfe63e383d40e-906cfe7530754271bb1ee57c470109bd
I20240502 21:27:45.228998 26756 tablet_copy_client.cc:1002] T 8f14db150fda449e962be2466f1d2a0e P ee27c3f9e0ec46caa742ac386d6b7c20: tablet copy: Injecting latency into file download: 1500
I20240502 21:27:45.230998 23935 tablet_copy_source_session.cc:205] T 906cfe7530754271bb1ee57c470109bd P 6810a866decb4f5089f7d35c870e4204: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:27:45.232319 26761 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 906cfe7530754271bb1ee57c470109bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:45.233409 24316 consensus_queue.cc:237] T fcc9cbeae6f343af9f84c73349f3dadb P 038c5d9d1d7a49c981c4c34fbc00d9e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } attrs { replace: true } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: NON_VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: true } }
I20240502 21:27:45.236052 24048 raft_consensus.cc:1261] T fcc9cbeae6f343af9f84c73349f3dadb P 6c194a6db6c145acafc0f3068b8ce1fd [term 1 FOLLOWER]: Refusing update from remote peer 038c5d9d1d7a49c981c4c34fbc00d9e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:27:45.236825 26630 consensus_queue.cc:1035] T fcc9cbeae6f343af9f84c73349f3dadb P 038c5d9d1d7a49c981c4c34fbc00d9e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:27:45.236397 24185 raft_consensus.cc:1261] T fcc9cbeae6f343af9f84c73349f3dadb P 009d3e7e8a8b4e2497de3a73d01ced49 [term 1 FOLLOWER]: Refusing update from remote peer 038c5d9d1d7a49c981c4c34fbc00d9e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:27:45.238185 26761 tablet_copy_client.cc:783] T 906cfe7530754271bb1ee57c470109bd P fb6004b8640a4906aa0cfe63e383d40e: tablet copy: Starting download of 0 data blocks...
I20240502 21:27:45.238451 26761 tablet_copy_client.cc:647] T 906cfe7530754271bb1ee57c470109bd P fb6004b8640a4906aa0cfe63e383d40e: tablet copy: Starting download of 1 WAL segments...
I20240502 21:27:45.238687 26630 consensus_queue.cc:1035] T fcc9cbeae6f343af9f84c73349f3dadb P 038c5d9d1d7a49c981c4c34fbc00d9e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:27:45.241060 26770 tablet_copy_client.cc:1002] T 906cfe7530754271bb1ee57c470109bd P fb6004b8640a4906aa0cfe63e383d40e: tablet copy: Injecting latency into file download: 1500
I20240502 21:27:45.241485 26560 raft_consensus.cc:2932] T fcc9cbeae6f343af9f84c73349f3dadb P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER fb6004b8640a4906aa0cfe63e383d40e (127.15.255.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } attrs { replace: true } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: NON_VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: true } } }
I20240502 21:27:45.242453 24048 raft_consensus.cc:2932] T fcc9cbeae6f343af9f84c73349f3dadb P 6c194a6db6c145acafc0f3068b8ce1fd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER fb6004b8640a4906aa0cfe63e383d40e (127.15.255.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } attrs { replace: true } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: NON_VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: true } } }
I20240502 21:27:45.242755 24185 raft_consensus.cc:2932] T fcc9cbeae6f343af9f84c73349f3dadb P 009d3e7e8a8b4e2497de3a73d01ced49 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER fb6004b8640a4906aa0cfe63e383d40e (127.15.255.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } attrs { replace: true } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: NON_VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: true } } }
I20240502 21:27:45.248067 23766 catalog_manager.cc:5484] T fcc9cbeae6f343af9f84c73349f3dadb P 038c5d9d1d7a49c981c4c34fbc00d9e6 reported cstate change: config changed from index -1 to 4, NON_VOTER fb6004b8640a4906aa0cfe63e383d40e (127.15.255.134) added. New cstate: current_term: 1 leader_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: NON_VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240502 21:27:45.250270 24250 consensus_peers.cc:485] T fcc9cbeae6f343af9f84c73349f3dadb P 038c5d9d1d7a49c981c4c34fbc00d9e6 -> Peer fb6004b8640a4906aa0cfe63e383d40e (127.15.255.134:32825): Couldn't send request to peer fb6004b8640a4906aa0cfe63e383d40e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fcc9cbeae6f343af9f84c73349f3dadb. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:45.266068 24185 consensus_queue.cc:237] T f84bbe304597450c978ae28e0c4e7a3e P 009d3e7e8a8b4e2497de3a73d01ced49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } attrs { replace: true } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: true } }
I20240502 21:27:45.269507 24048 raft_consensus.cc:1261] T f84bbe304597450c978ae28e0c4e7a3e P 6c194a6db6c145acafc0f3068b8ce1fd [term 1 FOLLOWER]: Refusing update from remote peer 009d3e7e8a8b4e2497de3a73d01ced49: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:27:45.269554 24316 raft_consensus.cc:1261] T f84bbe304597450c978ae28e0c4e7a3e P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 1 FOLLOWER]: Refusing update from remote peer 009d3e7e8a8b4e2497de3a73d01ced49: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:27:45.270072 25728 consensus_queue.cc:1035] T f84bbe304597450c978ae28e0c4e7a3e P 009d3e7e8a8b4e2497de3a73d01ced49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 21:27:45.271896 24117 consensus_peers.cc:485] T f84bbe304597450c978ae28e0c4e7a3e P 009d3e7e8a8b4e2497de3a73d01ced49 -> Peer a3e500d7077a48acb5edb33f7e178c81 (127.15.255.136:43595): Couldn't send request to peer a3e500d7077a48acb5edb33f7e178c81. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f84bbe304597450c978ae28e0c4e7a3e. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:45.272130 26554 consensus_queue.cc:1035] T f84bbe304597450c978ae28e0c4e7a3e P 009d3e7e8a8b4e2497de3a73d01ced49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:27:45.285578 25728 raft_consensus.cc:2932] T f84bbe304597450c978ae28e0c4e7a3e P 009d3e7e8a8b4e2497de3a73d01ced49 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a3e500d7077a48acb5edb33f7e178c81 (127.15.255.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } attrs { replace: true } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: true } } }
I20240502 21:27:45.286242 24048 raft_consensus.cc:2932] T f84bbe304597450c978ae28e0c4e7a3e P 6c194a6db6c145acafc0f3068b8ce1fd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a3e500d7077a48acb5edb33f7e178c81 (127.15.255.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } attrs { replace: true } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: true } } }
I20240502 21:27:45.286839 24315 raft_consensus.cc:2932] T f84bbe304597450c978ae28e0c4e7a3e P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a3e500d7077a48acb5edb33f7e178c81 (127.15.255.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } attrs { replace: true } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: true } } }
I20240502 21:27:45.291126 23766 catalog_manager.cc:5484] T f84bbe304597450c978ae28e0c4e7a3e P 009d3e7e8a8b4e2497de3a73d01ced49 reported cstate change: config changed from index -1 to 4, NON_VOTER a3e500d7077a48acb5edb33f7e178c81 (127.15.255.136) added. New cstate: current_term: 1 leader_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:45.296084 26790 ts_tablet_manager.cc:902] T a2a0537a60f849daae2061c2b0c4c89f P a3e500d7077a48acb5edb33f7e178c81: Initiating tablet copy from peer 009d3e7e8a8b4e2497de3a73d01ced49 (127.15.255.131:37895)
I20240502 21:27:45.320930 26790 tablet_copy_client.cc:308] T a2a0537a60f849daae2061c2b0c4c89f P a3e500d7077a48acb5edb33f7e178c81: tablet copy: Beginning tablet copy session from remote peer at address 127.15.255.131:37895
I20240502 21:27:45.340801 24205 tablet_copy_service.cc:140] P 009d3e7e8a8b4e2497de3a73d01ced49: Received BeginTabletCopySession request for tablet a2a0537a60f849daae2061c2b0c4c89f from peer a3e500d7077a48acb5edb33f7e178c81 ({username='slave'} at 127.15.255.136:51951)
I20240502 21:27:45.341003 24205 tablet_copy_service.cc:161] P 009d3e7e8a8b4e2497de3a73d01ced49: Beginning new tablet copy session on tablet a2a0537a60f849daae2061c2b0c4c89f from peer a3e500d7077a48acb5edb33f7e178c81 at {username='slave'} at 127.15.255.136:51951: session id = a3e500d7077a48acb5edb33f7e178c81-a2a0537a60f849daae2061c2b0c4c89f
I20240502 21:27:45.343413 24205 tablet_copy_source_session.cc:205] T a2a0537a60f849daae2061c2b0c4c89f P 009d3e7e8a8b4e2497de3a73d01ced49: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:27:45.345304 26790 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a2a0537a60f849daae2061c2b0c4c89f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:45.346616 24185 consensus_queue.cc:237] T 16bb47f03a2f46fdb7344e018f07fdcb P 009d3e7e8a8b4e2497de3a73d01ced49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } attrs { replace: true } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: NON_VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: true } }
I20240502 21:27:45.353478 26790 tablet_copy_client.cc:783] T a2a0537a60f849daae2061c2b0c4c89f P a3e500d7077a48acb5edb33f7e178c81: tablet copy: Starting download of 0 data blocks...
I20240502 21:27:45.353780 26790 tablet_copy_client.cc:647] T a2a0537a60f849daae2061c2b0c4c89f P a3e500d7077a48acb5edb33f7e178c81: tablet copy: Starting download of 1 WAL segments...
I20240502 21:27:45.355716 26805 tablet_copy_client.cc:1002] T a2a0537a60f849daae2061c2b0c4c89f P a3e500d7077a48acb5edb33f7e178c81: tablet copy: Injecting latency into file download: 1500
I20240502 21:27:45.359386 24048 raft_consensus.cc:1261] T 16bb47f03a2f46fdb7344e018f07fdcb P 6c194a6db6c145acafc0f3068b8ce1fd [term 1 FOLLOWER]: Refusing update from remote peer 009d3e7e8a8b4e2497de3a73d01ced49: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:27:45.359439 23916 raft_consensus.cc:1261] T 16bb47f03a2f46fdb7344e018f07fdcb P 6810a866decb4f5089f7d35c870e4204 [term 1 FOLLOWER]: Refusing update from remote peer 009d3e7e8a8b4e2497de3a73d01ced49: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 21:27:45.359872 24115 consensus_peers.cc:485] T 16bb47f03a2f46fdb7344e018f07fdcb P 009d3e7e8a8b4e2497de3a73d01ced49 -> Peer ee27c3f9e0ec46caa742ac386d6b7c20 (127.15.255.133:40841): Couldn't send request to peer ee27c3f9e0ec46caa742ac386d6b7c20. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 16bb47f03a2f46fdb7344e018f07fdcb. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:45.360913 26554 consensus_queue.cc:1035] T 16bb47f03a2f46fdb7344e018f07fdcb P 009d3e7e8a8b4e2497de3a73d01ced49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:27:45.361234 26475 consensus_queue.cc:1035] T 16bb47f03a2f46fdb7344e018f07fdcb P 009d3e7e8a8b4e2497de3a73d01ced49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:27:45.378814 26475 raft_consensus.cc:2932] T 16bb47f03a2f46fdb7344e018f07fdcb P 009d3e7e8a8b4e2497de3a73d01ced49 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ee27c3f9e0ec46caa742ac386d6b7c20 (127.15.255.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } attrs { replace: true } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: NON_VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: true } } }
I20240502 21:27:45.387387 23916 raft_consensus.cc:2932] T 16bb47f03a2f46fdb7344e018f07fdcb P 6810a866decb4f5089f7d35c870e4204 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ee27c3f9e0ec46caa742ac386d6b7c20 (127.15.255.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } attrs { replace: true } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: NON_VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: true } } }
I20240502 21:27:45.389619 23766 catalog_manager.cc:5484] T 16bb47f03a2f46fdb7344e018f07fdcb P 009d3e7e8a8b4e2497de3a73d01ced49 reported cstate change: config changed from index -1 to 4, NON_VOTER ee27c3f9e0ec46caa742ac386d6b7c20 (127.15.255.133) added. New cstate: current_term: 1 leader_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: NON_VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:45.392145 24048 raft_consensus.cc:2932] T 16bb47f03a2f46fdb7344e018f07fdcb P 6c194a6db6c145acafc0f3068b8ce1fd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ee27c3f9e0ec46caa742ac386d6b7c20 (127.15.255.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } attrs { replace: true } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: NON_VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: true } } }
I20240502 21:27:45.401078 26814 ts_tablet_manager.cc:902] T fe044374ad7b4999afc72ab812466749 P ee27c3f9e0ec46caa742ac386d6b7c20: Initiating tablet copy from peer 038c5d9d1d7a49c981c4c34fbc00d9e6 (127.15.255.132:46813)
I20240502 21:27:45.431861 24162 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:27:45.436971 24297 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 21:27:45.457037 23916 consensus_queue.cc:237] T 83b95f3ae7564b81b31a2f5673ee22bb P 6810a866decb4f5089f7d35c870e4204 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } attrs { replace: true } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: NON_VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: true } }
W20240502 21:27:45.481367 23850 consensus_peers.cc:485] T 83b95f3ae7564b81b31a2f5673ee22bb P 6810a866decb4f5089f7d35c870e4204 -> Peer e117afbf8819424e8a98476453dfde2a (127.15.255.135:32775): Couldn't send request to peer e117afbf8819424e8a98476453dfde2a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 83b95f3ae7564b81b31a2f5673ee22bb. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:45.482316 24048 raft_consensus.cc:1261] T 83b95f3ae7564b81b31a2f5673ee22bb P 6c194a6db6c145acafc0f3068b8ce1fd [term 1 FOLLOWER]: Refusing update from remote peer 6810a866decb4f5089f7d35c870e4204: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:27:45.482622 24316 raft_consensus.cc:1261] T 83b95f3ae7564b81b31a2f5673ee22bb P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 1 FOLLOWER]: Refusing update from remote peer 6810a866decb4f5089f7d35c870e4204: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:27:45.482863 26471 consensus_queue.cc:1035] T 83b95f3ae7564b81b31a2f5673ee22bb P 6810a866decb4f5089f7d35c870e4204 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:27:45.483217 26588 consensus_queue.cc:1035] T 83b95f3ae7564b81b31a2f5673ee22bb P 6810a866decb4f5089f7d35c870e4204 [LEADER]: Connected to new peer: Peer: permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:27:45.492316 23894 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 21:27:45.522320 26833 ts_tablet_manager.cc:902] T 3cb3e62ca0e94e659efb3b7b0c2c9c91 P e117afbf8819424e8a98476453dfde2a: Initiating tablet copy from peer 6c194a6db6c145acafc0f3068b8ce1fd (127.15.255.130:41235)
I20240502 21:27:45.535317 26260 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:45.543278 26833 tablet_copy_client.cc:308] T 3cb3e62ca0e94e659efb3b7b0c2c9c91 P e117afbf8819424e8a98476453dfde2a: tablet copy: Beginning tablet copy session from remote peer at address 127.15.255.130:41235
I20240502 21:27:45.557972 24070 tablet_copy_service.cc:140] P 6c194a6db6c145acafc0f3068b8ce1fd: Received BeginTabletCopySession request for tablet 3cb3e62ca0e94e659efb3b7b0c2c9c91 from peer e117afbf8819424e8a98476453dfde2a ({username='slave'} at 127.15.255.135:53383)
I20240502 21:27:45.558300 24070 tablet_copy_service.cc:161] P 6c194a6db6c145acafc0f3068b8ce1fd: Beginning new tablet copy session on tablet 3cb3e62ca0e94e659efb3b7b0c2c9c91 from peer e117afbf8819424e8a98476453dfde2a at {username='slave'} at 127.15.255.135:53383: session id = e117afbf8819424e8a98476453dfde2a-3cb3e62ca0e94e659efb3b7b0c2c9c91
I20240502 21:27:45.562037 26835 ts_tablet_manager.cc:902] T a274546743354e79bffa26ae7000e895 P e117afbf8819424e8a98476453dfde2a: Initiating tablet copy from peer 6c194a6db6c145acafc0f3068b8ce1fd (127.15.255.130:41235)
I20240502 21:27:45.562129 24070 tablet_copy_source_session.cc:205] T 3cb3e62ca0e94e659efb3b7b0c2c9c91 P 6c194a6db6c145acafc0f3068b8ce1fd: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:27:45.566785 26833 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3cb3e62ca0e94e659efb3b7b0c2c9c91. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:45.567274 26814 tablet_copy_client.cc:308] T fe044374ad7b4999afc72ab812466749 P ee27c3f9e0ec46caa742ac386d6b7c20: tablet copy: Beginning tablet copy session from remote peer at address 127.15.255.132:46813
I20240502 21:27:45.576478 24338 tablet_copy_service.cc:140] P 038c5d9d1d7a49c981c4c34fbc00d9e6: Received BeginTabletCopySession request for tablet fe044374ad7b4999afc72ab812466749 from peer ee27c3f9e0ec46caa742ac386d6b7c20 ({username='slave'} at 127.15.255.133:45805)
I20240502 21:27:45.576663 24338 tablet_copy_service.cc:161] P 038c5d9d1d7a49c981c4c34fbc00d9e6: Beginning new tablet copy session on tablet fe044374ad7b4999afc72ab812466749 from peer ee27c3f9e0ec46caa742ac386d6b7c20 at {username='slave'} at 127.15.255.133:45805: session id = ee27c3f9e0ec46caa742ac386d6b7c20-fe044374ad7b4999afc72ab812466749
I20240502 21:27:45.583987 24338 tablet_copy_source_session.cc:205] T fe044374ad7b4999afc72ab812466749 P 038c5d9d1d7a49c981c4c34fbc00d9e6: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:27:45.586531 26814 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe044374ad7b4999afc72ab812466749. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:45.587986 26586 raft_consensus.cc:2932] T 83b95f3ae7564b81b31a2f5673ee22bb P 6810a866decb4f5089f7d35c870e4204 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER e117afbf8819424e8a98476453dfde2a (127.15.255.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } attrs { replace: true } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: NON_VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: true } } }
I20240502 21:27:45.589553 24027 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:27:45.592186 24047 raft_consensus.cc:2932] T 83b95f3ae7564b81b31a2f5673ee22bb P 6c194a6db6c145acafc0f3068b8ce1fd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER e117afbf8819424e8a98476453dfde2a (127.15.255.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } attrs { replace: true } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: NON_VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: true } } }
I20240502 21:27:45.595317 26833 tablet_copy_client.cc:783] T 3cb3e62ca0e94e659efb3b7b0c2c9c91 P e117afbf8819424e8a98476453dfde2a: tablet copy: Starting download of 0 data blocks...
I20240502 21:27:45.595556 26833 tablet_copy_client.cc:647] T 3cb3e62ca0e94e659efb3b7b0c2c9c91 P e117afbf8819424e8a98476453dfde2a: tablet copy: Starting download of 1 WAL segments...
I20240502 21:27:45.598740 26839 tablet_copy_client.cc:1002] T 3cb3e62ca0e94e659efb3b7b0c2c9c91 P e117afbf8819424e8a98476453dfde2a: tablet copy: Injecting latency into file download: 1500
I20240502 21:27:45.607451 26814 tablet_copy_client.cc:783] T fe044374ad7b4999afc72ab812466749 P ee27c3f9e0ec46caa742ac386d6b7c20: tablet copy: Starting download of 0 data blocks...
I20240502 21:27:45.607748 26814 tablet_copy_client.cc:647] T fe044374ad7b4999afc72ab812466749 P ee27c3f9e0ec46caa742ac386d6b7c20: tablet copy: Starting download of 1 WAL segments...
I20240502 21:27:45.608882 24316 raft_consensus.cc:2932] T 83b95f3ae7564b81b31a2f5673ee22bb P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER e117afbf8819424e8a98476453dfde2a (127.15.255.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } attrs { replace: true } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: NON_VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: true } } }
I20240502 21:27:45.609506 26832 tablet_copy_client.cc:1002] T fe044374ad7b4999afc72ab812466749 P ee27c3f9e0ec46caa742ac386d6b7c20: tablet copy: Injecting latency into file download: 1500
I20240502 21:27:45.608157 23766 catalog_manager.cc:5484] T 83b95f3ae7564b81b31a2f5673ee22bb P 6c194a6db6c145acafc0f3068b8ce1fd reported cstate change: config changed from index -1 to 4, NON_VOTER e117afbf8819424e8a98476453dfde2a (127.15.255.135) added. New cstate: current_term: 1 leader_uuid: "6810a866decb4f5089f7d35c870e4204" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } attrs { replace: true } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: NON_VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: true } } }
I20240502 21:27:45.645565 25859 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:45.654596 26127 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:45.666627 25992 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:45.668648 26835 tablet_copy_client.cc:308] T a274546743354e79bffa26ae7000e895 P e117afbf8819424e8a98476453dfde2a: tablet copy: Beginning tablet copy session from remote peer at address 127.15.255.130:41235
I20240502 21:27:45.669297 24070 tablet_copy_service.cc:140] P 6c194a6db6c145acafc0f3068b8ce1fd: Received BeginTabletCopySession request for tablet a274546743354e79bffa26ae7000e895 from peer e117afbf8819424e8a98476453dfde2a ({username='slave'} at 127.15.255.135:53383)
I20240502 21:27:45.669471 24070 tablet_copy_service.cc:161] P 6c194a6db6c145acafc0f3068b8ce1fd: Beginning new tablet copy session on tablet a274546743354e79bffa26ae7000e895 from peer e117afbf8819424e8a98476453dfde2a at {username='slave'} at 127.15.255.135:53383: session id = e117afbf8819424e8a98476453dfde2a-a274546743354e79bffa26ae7000e895
I20240502 21:27:45.672453 24070 tablet_copy_source_session.cc:205] T a274546743354e79bffa26ae7000e895 P 6c194a6db6c145acafc0f3068b8ce1fd: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:27:45.674958 26835 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a274546743354e79bffa26ae7000e895. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:45.695922 26835 tablet_copy_client.cc:783] T a274546743354e79bffa26ae7000e895 P e117afbf8819424e8a98476453dfde2a: tablet copy: Starting download of 0 data blocks...
I20240502 21:27:45.696229 26835 tablet_copy_client.cc:647] T a274546743354e79bffa26ae7000e895 P e117afbf8819424e8a98476453dfde2a: tablet copy: Starting download of 1 WAL segments...
I20240502 21:27:45.697924 26848 tablet_copy_client.cc:1002] T a274546743354e79bffa26ae7000e895 P e117afbf8819424e8a98476453dfde2a: tablet copy: Injecting latency into file download: 1500
I20240502 21:27:45.699105 26849 ts_tablet_manager.cc:902] T fcc9cbeae6f343af9f84c73349f3dadb P fb6004b8640a4906aa0cfe63e383d40e: Initiating tablet copy from peer 038c5d9d1d7a49c981c4c34fbc00d9e6 (127.15.255.132:46813)
I20240502 21:27:45.723116 26852 ts_tablet_manager.cc:902] T d5697d8b06094211b261f144de9d46f8 P a3e500d7077a48acb5edb33f7e178c81: Initiating tablet copy from peer 038c5d9d1d7a49c981c4c34fbc00d9e6 (127.15.255.132:46813)
I20240502 21:27:45.738564 26849 tablet_copy_client.cc:308] T fcc9cbeae6f343af9f84c73349f3dadb P fb6004b8640a4906aa0cfe63e383d40e: tablet copy: Beginning tablet copy session from remote peer at address 127.15.255.132:46813
I20240502 21:27:45.739733 26852 tablet_copy_client.cc:308] T d5697d8b06094211b261f144de9d46f8 P a3e500d7077a48acb5edb33f7e178c81: tablet copy: Beginning tablet copy session from remote peer at address 127.15.255.132:46813
I20240502 21:27:45.758314 24338 tablet_copy_service.cc:140] P 038c5d9d1d7a49c981c4c34fbc00d9e6: Received BeginTabletCopySession request for tablet d5697d8b06094211b261f144de9d46f8 from peer a3e500d7077a48acb5edb33f7e178c81 ({username='slave'} at 127.15.255.136:38895)
I20240502 21:27:45.758457 24338 tablet_copy_service.cc:161] P 038c5d9d1d7a49c981c4c34fbc00d9e6: Beginning new tablet copy session on tablet d5697d8b06094211b261f144de9d46f8 from peer a3e500d7077a48acb5edb33f7e178c81 at {username='slave'} at 127.15.255.136:38895: session id = a3e500d7077a48acb5edb33f7e178c81-d5697d8b06094211b261f144de9d46f8
I20240502 21:27:45.760439 24338 tablet_copy_source_session.cc:205] T d5697d8b06094211b261f144de9d46f8 P 038c5d9d1d7a49c981c4c34fbc00d9e6: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:27:45.760797 24337 tablet_copy_service.cc:140] P 038c5d9d1d7a49c981c4c34fbc00d9e6: Received BeginTabletCopySession request for tablet fcc9cbeae6f343af9f84c73349f3dadb from peer fb6004b8640a4906aa0cfe63e383d40e ({username='slave'} at 127.15.255.134:51733)
I20240502 21:27:45.760963 24337 tablet_copy_service.cc:161] P 038c5d9d1d7a49c981c4c34fbc00d9e6: Beginning new tablet copy session on tablet fcc9cbeae6f343af9f84c73349f3dadb from peer fb6004b8640a4906aa0cfe63e383d40e at {username='slave'} at 127.15.255.134:51733: session id = fb6004b8640a4906aa0cfe63e383d40e-fcc9cbeae6f343af9f84c73349f3dadb
I20240502 21:27:45.763345 26852 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d5697d8b06094211b261f144de9d46f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:45.763955 24337 tablet_copy_source_session.cc:205] T fcc9cbeae6f343af9f84c73349f3dadb P 038c5d9d1d7a49c981c4c34fbc00d9e6: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:27:45.768198 26852 tablet_copy_client.cc:783] T d5697d8b06094211b261f144de9d46f8 P a3e500d7077a48acb5edb33f7e178c81: tablet copy: Starting download of 0 data blocks...
I20240502 21:27:45.769536 26849 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fcc9cbeae6f343af9f84c73349f3dadb. 1 dirs total, 0 dirs full, 0 dirs failed
Runtime error: ksck discovered errors
I20240502 21:27:45.774459 26849 tablet_copy_client.cc:783] T fcc9cbeae6f343af9f84c73349f3dadb P fb6004b8640a4906aa0cfe63e383d40e: tablet copy: Starting download of 0 data blocks...
I20240502 21:27:45.774726 26849 tablet_copy_client.cc:647] T fcc9cbeae6f343af9f84c73349f3dadb P fb6004b8640a4906aa0cfe63e383d40e: tablet copy: Starting download of 1 WAL segments...
I20240502 21:27:45.768406 26852 tablet_copy_client.cc:647] T d5697d8b06094211b261f144de9d46f8 P a3e500d7077a48acb5edb33f7e178c81: tablet copy: Starting download of 1 WAL segments...
I20240502 21:27:45.776474 26863 tablet_copy_client.cc:1002] T d5697d8b06094211b261f144de9d46f8 P a3e500d7077a48acb5edb33f7e178c81: tablet copy: Injecting latency into file download: 1500
I20240502 21:27:45.776628 26859 tablet_copy_client.cc:1002] T fcc9cbeae6f343af9f84c73349f3dadb P fb6004b8640a4906aa0cfe63e383d40e: tablet copy: Injecting latency into file download: 1500
I20240502 21:27:45.784304 26876 ts_tablet_manager.cc:902] T 16bb47f03a2f46fdb7344e018f07fdcb P ee27c3f9e0ec46caa742ac386d6b7c20: Initiating tablet copy from peer 009d3e7e8a8b4e2497de3a73d01ced49 (127.15.255.131:37895)
I20240502 21:27:45.803550 26879 ts_tablet_manager.cc:902] T f84bbe304597450c978ae28e0c4e7a3e P a3e500d7077a48acb5edb33f7e178c81: Initiating tablet copy from peer 009d3e7e8a8b4e2497de3a73d01ced49 (127.15.255.131:37895)
I20240502 21:27:45.817586 26876 tablet_copy_client.cc:308] T 16bb47f03a2f46fdb7344e018f07fdcb P ee27c3f9e0ec46caa742ac386d6b7c20: tablet copy: Beginning tablet copy session from remote peer at address 127.15.255.131:37895
I20240502 21:27:45.818300 24205 tablet_copy_service.cc:140] P 009d3e7e8a8b4e2497de3a73d01ced49: Received BeginTabletCopySession request for tablet 16bb47f03a2f46fdb7344e018f07fdcb from peer ee27c3f9e0ec46caa742ac386d6b7c20 ({username='slave'} at 127.15.255.133:44245)
I20240502 21:27:45.818468 24205 tablet_copy_service.cc:161] P 009d3e7e8a8b4e2497de3a73d01ced49: Beginning new tablet copy session on tablet 16bb47f03a2f46fdb7344e018f07fdcb from peer ee27c3f9e0ec46caa742ac386d6b7c20 at {username='slave'} at 127.15.255.133:44245: session id = ee27c3f9e0ec46caa742ac386d6b7c20-16bb47f03a2f46fdb7344e018f07fdcb
I20240502 21:27:45.821498 24205 tablet_copy_source_session.cc:205] T 16bb47f03a2f46fdb7344e018f07fdcb P 009d3e7e8a8b4e2497de3a73d01ced49: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:27:45.823100 26876 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 16bb47f03a2f46fdb7344e018f07fdcb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:45.834172 26876 tablet_copy_client.cc:783] T 16bb47f03a2f46fdb7344e018f07fdcb P ee27c3f9e0ec46caa742ac386d6b7c20: tablet copy: Starting download of 0 data blocks...
I20240502 21:27:45.834439 26876 tablet_copy_client.cc:647] T 16bb47f03a2f46fdb7344e018f07fdcb P ee27c3f9e0ec46caa742ac386d6b7c20: tablet copy: Starting download of 1 WAL segments...
I20240502 21:27:45.836030 26885 tablet_copy_client.cc:1002] T 16bb47f03a2f46fdb7344e018f07fdcb P ee27c3f9e0ec46caa742ac386d6b7c20: tablet copy: Injecting latency into file download: 1500
I20240502 21:27:45.851413 26879 tablet_copy_client.cc:308] T f84bbe304597450c978ae28e0c4e7a3e P a3e500d7077a48acb5edb33f7e178c81: tablet copy: Beginning tablet copy session from remote peer at address 127.15.255.131:37895
I20240502 21:27:45.852078 24205 tablet_copy_service.cc:140] P 009d3e7e8a8b4e2497de3a73d01ced49: Received BeginTabletCopySession request for tablet f84bbe304597450c978ae28e0c4e7a3e from peer a3e500d7077a48acb5edb33f7e178c81 ({username='slave'} at 127.15.255.136:51951)
I20240502 21:27:45.852245 24205 tablet_copy_service.cc:161] P 009d3e7e8a8b4e2497de3a73d01ced49: Beginning new tablet copy session on tablet f84bbe304597450c978ae28e0c4e7a3e from peer a3e500d7077a48acb5edb33f7e178c81 at {username='slave'} at 127.15.255.136:51951: session id = a3e500d7077a48acb5edb33f7e178c81-f84bbe304597450c978ae28e0c4e7a3e
I20240502 21:27:45.855218 24205 tablet_copy_source_session.cc:205] T f84bbe304597450c978ae28e0c4e7a3e P 009d3e7e8a8b4e2497de3a73d01ced49: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:27:45.856557 26879 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f84bbe304597450c978ae28e0c4e7a3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:45.862733 26879 tablet_copy_client.cc:783] T f84bbe304597450c978ae28e0c4e7a3e P a3e500d7077a48acb5edb33f7e178c81: tablet copy: Starting download of 0 data blocks...
I20240502 21:27:45.863010 26879 tablet_copy_client.cc:647] T f84bbe304597450c978ae28e0c4e7a3e P a3e500d7077a48acb5edb33f7e178c81: tablet copy: Starting download of 1 WAL segments...
I20240502 21:27:45.864696 26891 tablet_copy_client.cc:1002] T f84bbe304597450c978ae28e0c4e7a3e P a3e500d7077a48acb5edb33f7e178c81: tablet copy: Injecting latency into file download: 1500
I20240502 21:27:45.918946 16382 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvNciQZ/build/asan/bin/kudu
/tmp/dist-test-taskvNciQZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-8/wal
--fs_data_dirs=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-8/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-8/logs
--server_dump_info_path=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-8/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.255.137:0
--local_ip_for_outbound_sockets=127.15.255.137
--webserver_interface=127.15.255.137
--webserver_port=0
--tserver_master_addrs=127.15.255.190:33301
--builtin_ntp_servers=127.15.255.148:46015
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true
--tablet_copy_download_file_inject_latency_ms=1500
--follower_unavailable_considered_failed_sec=30 with env {}
I20240502 21:27:45.983657 26944 ts_tablet_manager.cc:902] T 83b95f3ae7564b81b31a2f5673ee22bb P e117afbf8819424e8a98476453dfde2a: Initiating tablet copy from peer 6810a866decb4f5089f7d35c870e4204 (127.15.255.129:40435)
I20240502 21:27:46.014551 26944 tablet_copy_client.cc:308] T 83b95f3ae7564b81b31a2f5673ee22bb P e117afbf8819424e8a98476453dfde2a: tablet copy: Beginning tablet copy session from remote peer at address 127.15.255.129:40435
I20240502 21:27:46.015422 23935 tablet_copy_service.cc:140] P 6810a866decb4f5089f7d35c870e4204: Received BeginTabletCopySession request for tablet 83b95f3ae7564b81b31a2f5673ee22bb from peer e117afbf8819424e8a98476453dfde2a ({username='slave'} at 127.15.255.135:39893)
I20240502 21:27:46.015679 23935 tablet_copy_service.cc:161] P 6810a866decb4f5089f7d35c870e4204: Beginning new tablet copy session on tablet 83b95f3ae7564b81b31a2f5673ee22bb from peer e117afbf8819424e8a98476453dfde2a at {username='slave'} at 127.15.255.135:39893: session id = e117afbf8819424e8a98476453dfde2a-83b95f3ae7564b81b31a2f5673ee22bb
I20240502 21:27:46.018025 23935 tablet_copy_source_session.cc:205] T 83b95f3ae7564b81b31a2f5673ee22bb P 6810a866decb4f5089f7d35c870e4204: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:27:46.019337 26944 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83b95f3ae7564b81b31a2f5673ee22bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:46.025324 26944 tablet_copy_client.cc:783] T 83b95f3ae7564b81b31a2f5673ee22bb P e117afbf8819424e8a98476453dfde2a: tablet copy: Starting download of 0 data blocks...
I20240502 21:27:46.025592 26944 tablet_copy_client.cc:647] T 83b95f3ae7564b81b31a2f5673ee22bb P e117afbf8819424e8a98476453dfde2a: tablet copy: Starting download of 1 WAL segments...
I20240502 21:27:46.027578 26952 tablet_copy_client.cc:1002] T 83b95f3ae7564b81b31a2f5673ee22bb P e117afbf8819424e8a98476453dfde2a: tablet copy: Injecting latency into file download: 1500
I20240502 21:27:46.153276 26633 tablet_copy_client.cc:523] T cca45c89b9fe40e7800d95c1598a9e52 P ee27c3f9e0ec46caa742ac386d6b7c20: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:27:46.157824 26633 tablet_bootstrap.cc:492] T cca45c89b9fe40e7800d95c1598a9e52 P ee27c3f9e0ec46caa742ac386d6b7c20: Bootstrap starting.
I20240502 21:27:46.166725 26633 log.cc:824] T cca45c89b9fe40e7800d95c1598a9e52 P ee27c3f9e0ec46caa742ac386d6b7c20: Log is configured to *not* fsync() on all Append() calls
I20240502 21:27:46.197005 26633 tablet_bootstrap.cc:492] T cca45c89b9fe40e7800d95c1598a9e52 P ee27c3f9e0ec46caa742ac386d6b7c20: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:27:46.197580 26633 tablet_bootstrap.cc:492] T cca45c89b9fe40e7800d95c1598a9e52 P ee27c3f9e0ec46caa742ac386d6b7c20: Bootstrap complete.
I20240502 21:27:46.198009 26633 ts_tablet_manager.cc:1367] T cca45c89b9fe40e7800d95c1598a9e52 P ee27c3f9e0ec46caa742ac386d6b7c20: Time spent bootstrapping tablet: real 0.040s user 0.034s sys 0.005s
I20240502 21:27:46.203372 26633 raft_consensus.cc:348] T cca45c89b9fe40e7800d95c1598a9e52 P ee27c3f9e0ec46caa742ac386d6b7c20 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } attrs { replace: true } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: NON_VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: true } }
I20240502 21:27:46.203727 26633 raft_consensus.cc:729] T cca45c89b9fe40e7800d95c1598a9e52 P ee27c3f9e0ec46caa742ac386d6b7c20 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ee27c3f9e0ec46caa742ac386d6b7c20, State: Initialized, Role: LEARNER
I20240502 21:27:46.204149 26633 consensus_queue.cc:260] T cca45c89b9fe40e7800d95c1598a9e52 P ee27c3f9e0ec46caa742ac386d6b7c20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } attrs { replace: true } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: NON_VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: true } }
I20240502 21:27:46.206431 26633 ts_tablet_manager.cc:1398] T cca45c89b9fe40e7800d95c1598a9e52 P ee27c3f9e0ec46caa742ac386d6b7c20: Time spent starting tablet: real 0.008s user 0.005s sys 0.004s
I20240502 21:27:46.207507 24205 tablet_copy_service.cc:342] P 009d3e7e8a8b4e2497de3a73d01ced49: Request end of tablet copy session ee27c3f9e0ec46caa742ac386d6b7c20-cca45c89b9fe40e7800d95c1598a9e52 received from {username='slave'} at 127.15.255.133:44245
I20240502 21:27:46.207705 24205 tablet_copy_service.cc:433] P 009d3e7e8a8b4e2497de3a73d01ced49: ending tablet copy session ee27c3f9e0ec46caa742ac386d6b7c20-cca45c89b9fe40e7800d95c1598a9e52 on tablet cca45c89b9fe40e7800d95c1598a9e52 with peer ee27c3f9e0ec46caa742ac386d6b7c20
I20240502 21:27:46.307973 25879 raft_consensus.cc:1203] T cca45c89b9fe40e7800d95c1598a9e52 P ee27c3f9e0ec46caa742ac386d6b7c20 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
W20240502 21:27:46.308514 26928 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:27:46.308840 26928 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:27:46.308996 26928 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:27:46.317385 26928 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 21:27:46.317489 26928 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:27:46.317639 26928 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.255.137
I20240502 21:27:46.328075 26928 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.255.148:46015
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--follower_unavailable_considered_failed_sec=30
--fs_data_dirs=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-8/data
--fs_wal_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-8/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.255.137:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-8/data/info.pb
--webserver_interface=127.15.255.137
--webserver_port=0
--tablet_copy_download_file_inject_latency_ms=1500
--tserver_master_addrs=127.15.255.190:33301
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.255.137
--log_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-8/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:27:46.329665 26928 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:27:46.331703 26928 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:27:46.344890 27003 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:27:46.346217 27004 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:27:46.346781 27006 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:27:46.347752 26928 server_base.cc:1034] running on GCE node
I20240502 21:27:46.348448 26928 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:27:46.349828 26928 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:27:46.351660 26928 hybrid_clock.cc:648] HybridClock initialized: now 1714685266351584 us; error 48 us; skew 500 ppm
I20240502 21:27:46.352030 26928 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:27:46.354118 26664 tablet_copy_client.cc:523] T 0b3c514213bb40ca98955d6409a3b372 P fb6004b8640a4906aa0cfe63e383d40e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:27:46.355311 26928 webserver.cc:460] Webserver started at http://127.15.255.137:43723/ using document root <none> and password file <none>
I20240502 21:27:46.356211 26928 fs_manager.cc:359] Metadata directory not provided
I20240502 21:27:46.356343 26928 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:27:46.356707 26928 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:27:46.358542 26664 tablet_bootstrap.cc:492] T 0b3c514213bb40ca98955d6409a3b372 P fb6004b8640a4906aa0cfe63e383d40e: Bootstrap starting.
I20240502 21:27:46.360571 26928 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-8/data/instance:
uuid: "4f58800ba2ed45eabd81c2f2e691aacd"
format_stamp: "Formatted at 2024-05-02 21:27:46 on dist-test-slave-jdfx"
I20240502 21:27:46.361557 26928 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-8/wal/instance:
uuid: "4f58800ba2ed45eabd81c2f2e691aacd"
format_stamp: "Formatted at 2024-05-02 21:27:46 on dist-test-slave-jdfx"
I20240502 21:27:46.369107 26664 log.cc:824] T 0b3c514213bb40ca98955d6409a3b372 P fb6004b8640a4906aa0cfe63e383d40e: Log is configured to *not* fsync() on all Append() calls
I20240502 21:27:46.369647 26928 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.005s sys 0.000s
I20240502 21:27:46.373750 27023 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:27:46.375294 26928 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20240502 21:27:46.375444 26928 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-8/data,/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-8/wal
uuid: "4f58800ba2ed45eabd81c2f2e691aacd"
format_stamp: "Formatted at 2024-05-02 21:27:46 on dist-test-slave-jdfx"
I20240502 21:27:46.375658 26928 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-8/wal
metadata directory: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-8/wal
1 data directories: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-8/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:27:46.380069 26666 tablet_copy_client.cc:523] T 4eebf4508766421eb6c547a0fbbc28b1 P e117afbf8819424e8a98476453dfde2a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:27:46.384327 26666 tablet_bootstrap.cc:492] T 4eebf4508766421eb6c547a0fbbc28b1 P e117afbf8819424e8a98476453dfde2a: Bootstrap starting.
I20240502 21:27:46.397845 26666 log.cc:824] T 4eebf4508766421eb6c547a0fbbc28b1 P e117afbf8819424e8a98476453dfde2a: Log is configured to *not* fsync() on all Append() calls
I20240502 21:27:46.407308 26689 tablet_copy_client.cc:523] T 89f86d58fe414b579845214534d338d8 P fb6004b8640a4906aa0cfe63e383d40e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:27:46.411442 26689 tablet_bootstrap.cc:492] T 89f86d58fe414b579845214534d338d8 P fb6004b8640a4906aa0cfe63e383d40e: Bootstrap starting.
I20240502 21:27:46.415078 26664 tablet_bootstrap.cc:492] T 0b3c514213bb40ca98955d6409a3b372 P fb6004b8640a4906aa0cfe63e383d40e: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:27:46.415607 26664 tablet_bootstrap.cc:492] T 0b3c514213bb40ca98955d6409a3b372 P fb6004b8640a4906aa0cfe63e383d40e: Bootstrap complete.
I20240502 21:27:46.416100 26664 ts_tablet_manager.cc:1367] T 0b3c514213bb40ca98955d6409a3b372 P fb6004b8640a4906aa0cfe63e383d40e: Time spent bootstrapping tablet: real 0.058s user 0.054s sys 0.000s
I20240502 21:27:46.422981 26664 raft_consensus.cc:348] T 0b3c514213bb40ca98955d6409a3b372 P fb6004b8640a4906aa0cfe63e383d40e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } attrs { replace: true } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: NON_VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: true } }
I20240502 21:27:46.423444 26664 raft_consensus.cc:729] T 0b3c514213bb40ca98955d6409a3b372 P fb6004b8640a4906aa0cfe63e383d40e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: fb6004b8640a4906aa0cfe63e383d40e, State: Initialized, Role: LEARNER
I20240502 21:27:46.424052 26664 consensus_queue.cc:260] T 0b3c514213bb40ca98955d6409a3b372 P fb6004b8640a4906aa0cfe63e383d40e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } attrs { replace: true } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: NON_VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: true } }
I20240502 21:27:46.432381 26664 ts_tablet_manager.cc:1398] T 0b3c514213bb40ca98955d6409a3b372 P fb6004b8640a4906aa0cfe63e383d40e: Time spent starting tablet: real 0.016s user 0.011s sys 0.008s
I20240502 21:27:46.433635 24205 tablet_copy_service.cc:342] P 009d3e7e8a8b4e2497de3a73d01ced49: Request end of tablet copy session fb6004b8640a4906aa0cfe63e383d40e-0b3c514213bb40ca98955d6409a3b372 received from {username='slave'} at 127.15.255.134:41395
I20240502 21:27:46.433765 24205 tablet_copy_service.cc:433] P 009d3e7e8a8b4e2497de3a73d01ced49: ending tablet copy session fb6004b8640a4906aa0cfe63e383d40e-0b3c514213bb40ca98955d6409a3b372 on tablet 0b3c514213bb40ca98955d6409a3b372 with peer fb6004b8640a4906aa0cfe63e383d40e
I20240502 21:27:46.445200 26687 tablet_copy_client.cc:523] T adf80b9d71314010a7a4cf052fd8a511 P a3e500d7077a48acb5edb33f7e178c81: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:27:46.452502 26687 tablet_bootstrap.cc:492] T adf80b9d71314010a7a4cf052fd8a511 P a3e500d7077a48acb5edb33f7e178c81: Bootstrap starting.
I20240502 21:27:46.466982 26687 log.cc:824] T adf80b9d71314010a7a4cf052fd8a511 P a3e500d7077a48acb5edb33f7e178c81: Log is configured to *not* fsync() on all Append() calls
I20240502 21:27:46.480412 26928 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:27:46.481249 26704 tablet_copy_client.cc:523] T 549666d19e724a96801db183a07809d8 P a3e500d7077a48acb5edb33f7e178c81: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:27:46.481429 26928 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:27:46.481717 26928 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:27:46.488783 26704 tablet_bootstrap.cc:492] T 549666d19e724a96801db183a07809d8 P a3e500d7077a48acb5edb33f7e178c81: Bootstrap starting.
I20240502 21:27:46.502928 26928 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:27:46.510399 26689 tablet_bootstrap.cc:492] T 89f86d58fe414b579845214534d338d8 P fb6004b8640a4906aa0cfe63e383d40e: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:27:46.510895 26689 tablet_bootstrap.cc:492] T 89f86d58fe414b579845214534d338d8 P fb6004b8640a4906aa0cfe63e383d40e: Bootstrap complete.
I20240502 21:27:46.511188 26689 ts_tablet_manager.cc:1367] T 89f86d58fe414b579845214534d338d8 P fb6004b8640a4906aa0cfe63e383d40e: Time spent bootstrapping tablet: real 0.100s user 0.042s sys 0.005s
I20240502 21:27:46.511353 26928 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:27:46.511458 26928 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:27:46.511554 26928 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:27:46.511626 26928 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 21:27:46.512254 26689 raft_consensus.cc:348] T 89f86d58fe414b579845214534d338d8 P fb6004b8640a4906aa0cfe63e383d40e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } attrs { replace: true } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: NON_VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: true } }
I20240502 21:27:46.512573 26689 raft_consensus.cc:729] T 89f86d58fe414b579845214534d338d8 P fb6004b8640a4906aa0cfe63e383d40e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: fb6004b8640a4906aa0cfe63e383d40e, State: Initialized, Role: LEARNER
I20240502 21:27:46.512800 26689 consensus_queue.cc:260] T 89f86d58fe414b579845214534d338d8 P fb6004b8640a4906aa0cfe63e383d40e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } attrs { replace: true } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: NON_VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: true } }
I20240502 21:27:46.513401 26689 ts_tablet_manager.cc:1398] T 89f86d58fe414b579845214534d338d8 P fb6004b8640a4906aa0cfe63e383d40e: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240502 21:27:46.519342 23935 tablet_copy_service.cc:342] P 6810a866decb4f5089f7d35c870e4204: Request end of tablet copy session fb6004b8640a4906aa0cfe63e383d40e-89f86d58fe414b579845214534d338d8 received from {username='slave'} at 127.15.255.134:44215
I20240502 21:27:46.519512 23935 tablet_copy_service.cc:433] P 6810a866decb4f5089f7d35c870e4204: ending tablet copy session fb6004b8640a4906aa0cfe63e383d40e-89f86d58fe414b579845214534d338d8 on tablet 89f86d58fe414b579845214534d338d8 with peer fb6004b8640a4906aa0cfe63e383d40e
I20240502 21:27:46.520105 26687 tablet_bootstrap.cc:492] T adf80b9d71314010a7a4cf052fd8a511 P a3e500d7077a48acb5edb33f7e178c81: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:27:46.520928 26687 tablet_bootstrap.cc:492] T adf80b9d71314010a7a4cf052fd8a511 P a3e500d7077a48acb5edb33f7e178c81: Bootstrap complete.
I20240502 21:27:46.521411 26687 ts_tablet_manager.cc:1367] T adf80b9d71314010a7a4cf052fd8a511 P a3e500d7077a48acb5edb33f7e178c81: Time spent bootstrapping tablet: real 0.069s user 0.055s sys 0.008s
I20240502 21:27:46.522604 26666 tablet_bootstrap.cc:492] T 4eebf4508766421eb6c547a0fbbc28b1 P e117afbf8819424e8a98476453dfde2a: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:27:46.523170 26666 tablet_bootstrap.cc:492] T 4eebf4508766421eb6c547a0fbbc28b1 P e117afbf8819424e8a98476453dfde2a: Bootstrap complete.
I20240502 21:27:46.524109 26666 ts_tablet_manager.cc:1367] T 4eebf4508766421eb6c547a0fbbc28b1 P e117afbf8819424e8a98476453dfde2a: Time spent bootstrapping tablet: real 0.140s user 0.063s sys 0.010s
I20240502 21:27:46.527348 26687 raft_consensus.cc:348] T adf80b9d71314010a7a4cf052fd8a511 P a3e500d7077a48acb5edb33f7e178c81 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } attrs { replace: true } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: true } }
I20240502 21:27:46.527829 26687 raft_consensus.cc:729] T adf80b9d71314010a7a4cf052fd8a511 P a3e500d7077a48acb5edb33f7e178c81 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a3e500d7077a48acb5edb33f7e178c81, State: Initialized, Role: LEARNER
I20240502 21:27:46.528374 26687 consensus_queue.cc:260] T adf80b9d71314010a7a4cf052fd8a511 P a3e500d7077a48acb5edb33f7e178c81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } attrs { replace: true } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: true } }
I20240502 21:27:46.530391 26666 raft_consensus.cc:348] T 4eebf4508766421eb6c547a0fbbc28b1 P e117afbf8819424e8a98476453dfde2a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } attrs { replace: true } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: NON_VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: true } }
I20240502 21:27:46.530870 26666 raft_consensus.cc:729] T 4eebf4508766421eb6c547a0fbbc28b1 P e117afbf8819424e8a98476453dfde2a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e117afbf8819424e8a98476453dfde2a, State: Initialized, Role: LEARNER
I20240502 21:27:46.531399 26666 consensus_queue.cc:260] T 4eebf4508766421eb6c547a0fbbc28b1 P e117afbf8819424e8a98476453dfde2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } attrs { replace: true } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: NON_VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: true } }
I20240502 21:27:46.536350 26687 ts_tablet_manager.cc:1398] T adf80b9d71314010a7a4cf052fd8a511 P a3e500d7077a48acb5edb33f7e178c81: Time spent starting tablet: real 0.015s user 0.007s sys 0.007s
I20240502 21:27:46.537225 23935 tablet_copy_service.cc:342] P 6810a866decb4f5089f7d35c870e4204: Request end of tablet copy session a3e500d7077a48acb5edb33f7e178c81-adf80b9d71314010a7a4cf052fd8a511 received from {username='slave'} at 127.15.255.136:46499
I20240502 21:27:46.537351 23935 tablet_copy_service.cc:433] P 6810a866decb4f5089f7d35c870e4204: ending tablet copy session a3e500d7077a48acb5edb33f7e178c81-adf80b9d71314010a7a4cf052fd8a511 on tablet adf80b9d71314010a7a4cf052fd8a511 with peer a3e500d7077a48acb5edb33f7e178c81
I20240502 21:27:46.539430 26666 ts_tablet_manager.cc:1398] T 4eebf4508766421eb6c547a0fbbc28b1 P e117afbf8819424e8a98476453dfde2a: Time spent starting tablet: real 0.015s user 0.011s sys 0.000s
I20240502 21:27:46.540599 23935 tablet_copy_service.cc:342] P 6810a866decb4f5089f7d35c870e4204: Request end of tablet copy session e117afbf8819424e8a98476453dfde2a-4eebf4508766421eb6c547a0fbbc28b1 received from {username='slave'} at 127.15.255.135:39893
I20240502 21:27:46.540721 23935 tablet_copy_service.cc:433] P 6810a866decb4f5089f7d35c870e4204: ending tablet copy session e117afbf8819424e8a98476453dfde2a-4eebf4508766421eb6c547a0fbbc28b1 on tablet 4eebf4508766421eb6c547a0fbbc28b1 with peer e117afbf8819424e8a98476453dfde2a
I20240502 21:27:46.571712 26713 tablet_copy_client.cc:523] T 6bd759d700a24e9f8ad11f243cf7ebfa P ee27c3f9e0ec46caa742ac386d6b7c20: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:27:46.575261 26713 tablet_bootstrap.cc:492] T 6bd759d700a24e9f8ad11f243cf7ebfa P ee27c3f9e0ec46caa742ac386d6b7c20: Bootstrap starting.
I20240502 21:27:46.596009 26704 tablet_bootstrap.cc:492] T 549666d19e724a96801db183a07809d8 P a3e500d7077a48acb5edb33f7e178c81: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:27:46.596479 26704 tablet_bootstrap.cc:492] T 549666d19e724a96801db183a07809d8 P a3e500d7077a48acb5edb33f7e178c81: Bootstrap complete.
I20240502 21:27:46.596791 26704 ts_tablet_manager.cc:1367] T 549666d19e724a96801db183a07809d8 P a3e500d7077a48acb5edb33f7e178c81: Time spent bootstrapping tablet: real 0.108s user 0.046s sys 0.006s
I20240502 21:27:46.597877 26704 raft_consensus.cc:348] T 549666d19e724a96801db183a07809d8 P a3e500d7077a48acb5edb33f7e178c81 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } attrs { replace: true } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: true } }
I20240502 21:27:46.598155 26704 raft_consensus.cc:729] T 549666d19e724a96801db183a07809d8 P a3e500d7077a48acb5edb33f7e178c81 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a3e500d7077a48acb5edb33f7e178c81, State: Initialized, Role: LEARNER
I20240502 21:27:46.598369 26704 consensus_queue.cc:260] T 549666d19e724a96801db183a07809d8 P a3e500d7077a48acb5edb33f7e178c81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } attrs { replace: true } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: true } }
I20240502 21:27:46.598937 26704 ts_tablet_manager.cc:1398] T 549666d19e724a96801db183a07809d8 P a3e500d7077a48acb5edb33f7e178c81: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 21:27:46.599807 23935 tablet_copy_service.cc:342] P 6810a866decb4f5089f7d35c870e4204: Request end of tablet copy session a3e500d7077a48acb5edb33f7e178c81-549666d19e724a96801db183a07809d8 received from {username='slave'} at 127.15.255.136:46499
I20240502 21:27:46.599949 23935 tablet_copy_service.cc:433] P 6810a866decb4f5089f7d35c870e4204: ending tablet copy session a3e500d7077a48acb5edb33f7e178c81-549666d19e724a96801db183a07809d8 on tablet 549666d19e724a96801db183a07809d8 with peer a3e500d7077a48acb5edb33f7e178c81
I20240502 21:27:46.643508 26713 tablet_bootstrap.cc:492] T 6bd759d700a24e9f8ad11f243cf7ebfa P ee27c3f9e0ec46caa742ac386d6b7c20: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:27:46.644115 26713 tablet_bootstrap.cc:492] T 6bd759d700a24e9f8ad11f243cf7ebfa P ee27c3f9e0ec46caa742ac386d6b7c20: Bootstrap complete.
I20240502 21:27:46.644498 26713 ts_tablet_manager.cc:1367] T 6bd759d700a24e9f8ad11f243cf7ebfa P ee27c3f9e0ec46caa742ac386d6b7c20: Time spent bootstrapping tablet: real 0.069s user 0.040s sys 0.013s
I20240502 21:27:46.645572 26713 raft_consensus.cc:348] T 6bd759d700a24e9f8ad11f243cf7ebfa P ee27c3f9e0ec46caa742ac386d6b7c20 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } attrs { replace: true } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: NON_VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: true } }
I20240502 21:27:46.645871 26713 raft_consensus.cc:729] T 6bd759d700a24e9f8ad11f243cf7ebfa P ee27c3f9e0ec46caa742ac386d6b7c20 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ee27c3f9e0ec46caa742ac386d6b7c20, State: Initialized, Role: LEARNER
I20240502 21:27:46.646075 26713 consensus_queue.cc:260] T 6bd759d700a24e9f8ad11f243cf7ebfa P ee27c3f9e0ec46caa742ac386d6b7c20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } attrs { replace: true } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: NON_VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: true } }
I20240502 21:27:46.646654 26713 ts_tablet_manager.cc:1398] T 6bd759d700a24e9f8ad11f243cf7ebfa P ee27c3f9e0ec46caa742ac386d6b7c20: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 21:27:46.647563 24205 tablet_copy_service.cc:342] P 009d3e7e8a8b4e2497de3a73d01ced49: Request end of tablet copy session ee27c3f9e0ec46caa742ac386d6b7c20-6bd759d700a24e9f8ad11f243cf7ebfa received from {username='slave'} at 127.15.255.133:44245
I20240502 21:27:46.647681 24205 tablet_copy_service.cc:433] P 009d3e7e8a8b4e2497de3a73d01ced49: ending tablet copy session ee27c3f9e0ec46caa742ac386d6b7c20-6bd759d700a24e9f8ad11f243cf7ebfa on tablet 6bd759d700a24e9f8ad11f243cf7ebfa with peer ee27c3f9e0ec46caa742ac386d6b7c20
I20240502 21:27:46.673553 26147 raft_consensus.cc:1203] T 4eebf4508766421eb6c547a0fbbc28b1 P e117afbf8819424e8a98476453dfde2a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 21:27:46.708890 26928 rpc_server.cc:307] RPC server started. Bound to: 127.15.255.137:40161
I20240502 21:27:46.708890 27179 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.255.137:40161 every 8 connection(s)
I20240502 21:27:46.710297 26928 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-8/data/info.pb
I20240502 21:27:46.711387 16382 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvNciQZ/build/asan/bin/kudu as pid 26928
I20240502 21:27:46.711608 16382 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-8/wal/instance
I20240502 21:27:46.724179 26745 tablet_copy_client.cc:523] T 11cb9a2fbda946fb9f303f2c465bccb6 P e117afbf8819424e8a98476453dfde2a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:27:46.727916 26745 tablet_bootstrap.cc:492] T 11cb9a2fbda946fb9f303f2c465bccb6 P e117afbf8819424e8a98476453dfde2a: Bootstrap starting.
I20240502 21:27:46.729543 26752 tablet_copy_client.cc:523] T 8f14db150fda449e962be2466f1d2a0e P ee27c3f9e0ec46caa742ac386d6b7c20: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:27:46.732846 26752 tablet_bootstrap.cc:492] T 8f14db150fda449e962be2466f1d2a0e P ee27c3f9e0ec46caa742ac386d6b7c20: Bootstrap starting.
I20240502 21:27:46.740247 27182 heartbeater.cc:347] Connected to a master server at 127.15.255.190:33301
I20240502 21:27:46.740630 27182 heartbeater.cc:464] Registering TS with master...
I20240502 21:27:46.741711 27182 heartbeater.cc:511] Master 127.15.255.190:33301 requested a full tablet report, sending...
I20240502 21:27:46.743274 23766 ts_manager.cc:194] Registered new tserver with Master: 4f58800ba2ed45eabd81c2f2e691aacd (127.15.255.137:40161)
I20240502 21:27:46.744472 23766 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.255.137:55643
I20240502 21:27:46.747753 26761 tablet_copy_client.cc:523] T 906cfe7530754271bb1ee57c470109bd P fb6004b8640a4906aa0cfe63e383d40e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:27:46.751178 26761 tablet_bootstrap.cc:492] T 906cfe7530754271bb1ee57c470109bd P fb6004b8640a4906aa0cfe63e383d40e: Bootstrap starting.
I20240502 21:27:46.753001 26013 raft_consensus.cc:1203] T 0b3c514213bb40ca98955d6409a3b372 P fb6004b8640a4906aa0cfe63e383d40e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 21:27:46.781994 26745 tablet_bootstrap.cc:492] T 11cb9a2fbda946fb9f303f2c465bccb6 P e117afbf8819424e8a98476453dfde2a: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:27:46.782547 26745 tablet_bootstrap.cc:492] T 11cb9a2fbda946fb9f303f2c465bccb6 P e117afbf8819424e8a98476453dfde2a: Bootstrap complete.
I20240502 21:27:46.782862 26745 ts_tablet_manager.cc:1367] T 11cb9a2fbda946fb9f303f2c465bccb6 P e117afbf8819424e8a98476453dfde2a: Time spent bootstrapping tablet: real 0.055s user 0.041s sys 0.009s
I20240502 21:27:46.783859 26745 raft_consensus.cc:348] T 11cb9a2fbda946fb9f303f2c465bccb6 P e117afbf8819424e8a98476453dfde2a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } attrs { replace: true } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: NON_VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: true } }
I20240502 21:27:46.784193 26745 raft_consensus.cc:729] T 11cb9a2fbda946fb9f303f2c465bccb6 P e117afbf8819424e8a98476453dfde2a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e117afbf8819424e8a98476453dfde2a, State: Initialized, Role: LEARNER
I20240502 21:27:46.784500 26745 consensus_queue.cc:260] T 11cb9a2fbda946fb9f303f2c465bccb6 P e117afbf8819424e8a98476453dfde2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } attrs { replace: true } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: NON_VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: true } }
I20240502 21:27:46.785216 26745 ts_tablet_manager.cc:1398] T 11cb9a2fbda946fb9f303f2c465bccb6 P e117afbf8819424e8a98476453dfde2a: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 21:27:46.785655 25879 raft_consensus.cc:1203] T 6bd759d700a24e9f8ad11f243cf7ebfa P ee27c3f9e0ec46caa742ac386d6b7c20 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 21:27:46.786396 24338 tablet_copy_service.cc:342] P 038c5d9d1d7a49c981c4c34fbc00d9e6: Request end of tablet copy session e117afbf8819424e8a98476453dfde2a-11cb9a2fbda946fb9f303f2c465bccb6 received from {username='slave'} at 127.15.255.135:44869
I20240502 21:27:46.786567 24338 tablet_copy_service.cc:433] P 038c5d9d1d7a49c981c4c34fbc00d9e6: ending tablet copy session e117afbf8819424e8a98476453dfde2a-11cb9a2fbda946fb9f303f2c465bccb6 on tablet 11cb9a2fbda946fb9f303f2c465bccb6 with peer e117afbf8819424e8a98476453dfde2a
I20240502 21:27:46.809512 26752 tablet_bootstrap.cc:492] T 8f14db150fda449e962be2466f1d2a0e P ee27c3f9e0ec46caa742ac386d6b7c20: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:27:46.809952 26752 tablet_bootstrap.cc:492] T 8f14db150fda449e962be2466f1d2a0e P ee27c3f9e0ec46caa742ac386d6b7c20: Bootstrap complete.
I20240502 21:27:46.810201 26752 ts_tablet_manager.cc:1367] T 8f14db150fda449e962be2466f1d2a0e P ee27c3f9e0ec46caa742ac386d6b7c20: Time spent bootstrapping tablet: real 0.077s user 0.047s sys 0.000s
I20240502 21:27:46.810855 26752 raft_consensus.cc:348] T 8f14db150fda449e962be2466f1d2a0e P ee27c3f9e0ec46caa742ac386d6b7c20 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } attrs { replace: true } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: NON_VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: true } }
I20240502 21:27:46.811045 26752 raft_consensus.cc:729] T 8f14db150fda449e962be2466f1d2a0e P ee27c3f9e0ec46caa742ac386d6b7c20 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ee27c3f9e0ec46caa742ac386d6b7c20, State: Initialized, Role: LEARNER
I20240502 21:27:46.811249 26752 consensus_queue.cc:260] T 8f14db150fda449e962be2466f1d2a0e P ee27c3f9e0ec46caa742ac386d6b7c20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } attrs { replace: true } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: NON_VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: true } }
I20240502 21:27:46.811882 26752 ts_tablet_manager.cc:1398] T 8f14db150fda449e962be2466f1d2a0e P ee27c3f9e0ec46caa742ac386d6b7c20: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240502 21:27:46.812542 26761 tablet_bootstrap.cc:492] T 906cfe7530754271bb1ee57c470109bd P fb6004b8640a4906aa0cfe63e383d40e: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:27:46.812723 23935 tablet_copy_service.cc:342] P 6810a866decb4f5089f7d35c870e4204: Request end of tablet copy session ee27c3f9e0ec46caa742ac386d6b7c20-8f14db150fda449e962be2466f1d2a0e received from {username='slave'} at 127.15.255.133:36415
I20240502 21:27:46.812847 23935 tablet_copy_service.cc:433] P 6810a866decb4f5089f7d35c870e4204: ending tablet copy session ee27c3f9e0ec46caa742ac386d6b7c20-8f14db150fda449e962be2466f1d2a0e on tablet 8f14db150fda449e962be2466f1d2a0e with peer ee27c3f9e0ec46caa742ac386d6b7c20
I20240502 21:27:46.812945 26761 tablet_bootstrap.cc:492] T 906cfe7530754271bb1ee57c470109bd P fb6004b8640a4906aa0cfe63e383d40e: Bootstrap complete.
I20240502 21:27:46.813248 26761 ts_tablet_manager.cc:1367] T 906cfe7530754271bb1ee57c470109bd P fb6004b8640a4906aa0cfe63e383d40e: Time spent bootstrapping tablet: real 0.062s user 0.047s sys 0.001s
I20240502 21:27:46.816129 26475 raft_consensus.cc:1050] T cca45c89b9fe40e7800d95c1598a9e52 P 009d3e7e8a8b4e2497de3a73d01ced49: attempting to promote NON_VOTER ee27c3f9e0ec46caa742ac386d6b7c20 to VOTER
I20240502 21:27:46.816136 26761 raft_consensus.cc:348] T 906cfe7530754271bb1ee57c470109bd P fb6004b8640a4906aa0cfe63e383d40e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } attrs { replace: true } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: NON_VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: true } }
I20240502 21:27:46.816382 26761 raft_consensus.cc:729] T 906cfe7530754271bb1ee57c470109bd P fb6004b8640a4906aa0cfe63e383d40e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: fb6004b8640a4906aa0cfe63e383d40e, State: Initialized, Role: LEARNER
I20240502 21:27:46.816659 26761 consensus_queue.cc:260] T 906cfe7530754271bb1ee57c470109bd P fb6004b8640a4906aa0cfe63e383d40e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } attrs { replace: true } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: NON_VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: true } }
I20240502 21:27:46.817174 26761 ts_tablet_manager.cc:1398] T 906cfe7530754271bb1ee57c470109bd P fb6004b8640a4906aa0cfe63e383d40e: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 21:27:46.817261 26475 consensus_queue.cc:237] T cca45c89b9fe40e7800d95c1598a9e52 P 009d3e7e8a8b4e2497de3a73d01ced49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } attrs { replace: true } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: false } }
I20240502 21:27:46.818320 23935 tablet_copy_service.cc:342] P 6810a866decb4f5089f7d35c870e4204: Request end of tablet copy session fb6004b8640a4906aa0cfe63e383d40e-906cfe7530754271bb1ee57c470109bd received from {username='slave'} at 127.15.255.134:44215
I20240502 21:27:46.818432 23935 tablet_copy_service.cc:433] P 6810a866decb4f5089f7d35c870e4204: ending tablet copy session fb6004b8640a4906aa0cfe63e383d40e-906cfe7530754271bb1ee57c470109bd on tablet 906cfe7530754271bb1ee57c470109bd with peer fb6004b8640a4906aa0cfe63e383d40e
I20240502 21:27:46.825667 24048 raft_consensus.cc:1261] T cca45c89b9fe40e7800d95c1598a9e52 P 6c194a6db6c145acafc0f3068b8ce1fd [term 1 FOLLOWER]: Refusing update from remote peer 009d3e7e8a8b4e2497de3a73d01ced49: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:27:46.826210 25879 raft_consensus.cc:1261] T cca45c89b9fe40e7800d95c1598a9e52 P ee27c3f9e0ec46caa742ac386d6b7c20 [term 1 LEARNER]: Refusing update from remote peer 009d3e7e8a8b4e2497de3a73d01ced49: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:27:46.826897 26554 consensus_queue.cc:1035] T cca45c89b9fe40e7800d95c1598a9e52 P 009d3e7e8a8b4e2497de3a73d01ced49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:27:46.827440 26554 consensus_queue.cc:1035] T cca45c89b9fe40e7800d95c1598a9e52 P 009d3e7e8a8b4e2497de3a73d01ced49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:27:46.831246 24316 raft_consensus.cc:1261] T cca45c89b9fe40e7800d95c1598a9e52 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 1 FOLLOWER]: Refusing update from remote peer 009d3e7e8a8b4e2497de3a73d01ced49: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:27:46.831823 25728 consensus_queue.cc:1035] T cca45c89b9fe40e7800d95c1598a9e52 P 009d3e7e8a8b4e2497de3a73d01ced49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:27:46.834091 25728 raft_consensus.cc:2932] T cca45c89b9fe40e7800d95c1598a9e52 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ee27c3f9e0ec46caa742ac386d6b7c20 (127.15.255.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } attrs { replace: true } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: false } } }
I20240502 21:27:46.835175 24048 raft_consensus.cc:2932] T cca45c89b9fe40e7800d95c1598a9e52 P 6c194a6db6c145acafc0f3068b8ce1fd [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ee27c3f9e0ec46caa742ac386d6b7c20 (127.15.255.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } attrs { replace: true } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: false } } }
I20240502 21:27:46.838205 24315 raft_consensus.cc:2932] T cca45c89b9fe40e7800d95c1598a9e52 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ee27c3f9e0ec46caa742ac386d6b7c20 (127.15.255.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } attrs { replace: true } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: false } } }
I20240502 21:27:46.838438 25879 raft_consensus.cc:2932] T cca45c89b9fe40e7800d95c1598a9e52 P ee27c3f9e0ec46caa742ac386d6b7c20 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ee27c3f9e0ec46caa742ac386d6b7c20 (127.15.255.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } attrs { replace: true } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: false } } }
I20240502 21:27:46.844671 23772 catalog_manager.cc:5484] T cca45c89b9fe40e7800d95c1598a9e52 P ee27c3f9e0ec46caa742ac386d6b7c20 reported cstate change: config changed from index 4 to 5, ee27c3f9e0ec46caa742ac386d6b7c20 (127.15.255.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } attrs { replace: true } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: false } } }
I20240502 21:27:46.856585 26790 tablet_copy_client.cc:523] T a2a0537a60f849daae2061c2b0c4c89f P a3e500d7077a48acb5edb33f7e178c81: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:27:46.859580 26790 tablet_bootstrap.cc:492] T a2a0537a60f849daae2061c2b0c4c89f P a3e500d7077a48acb5edb33f7e178c81: Bootstrap starting.
I20240502 21:27:46.862140 24184 consensus_queue.cc:237] T cca45c89b9fe40e7800d95c1598a9e52 P 009d3e7e8a8b4e2497de3a73d01ced49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: false } }
I20240502 21:27:46.864089 25879 raft_consensus.cc:1261] T cca45c89b9fe40e7800d95c1598a9e52 P ee27c3f9e0ec46caa742ac386d6b7c20 [term 1 FOLLOWER]: Refusing update from remote peer 009d3e7e8a8b4e2497de3a73d01ced49: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:27:46.864248 24048 raft_consensus.cc:1261] T cca45c89b9fe40e7800d95c1598a9e52 P 6c194a6db6c145acafc0f3068b8ce1fd [term 1 FOLLOWER]: Refusing update from remote peer 009d3e7e8a8b4e2497de3a73d01ced49: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:27:46.864661 25728 consensus_queue.cc:1035] T cca45c89b9fe40e7800d95c1598a9e52 P 009d3e7e8a8b4e2497de3a73d01ced49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:27:46.865130 25728 consensus_queue.cc:1035] T cca45c89b9fe40e7800d95c1598a9e52 P 009d3e7e8a8b4e2497de3a73d01ced49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:27:46.866899 25728 raft_consensus.cc:2932] T cca45c89b9fe40e7800d95c1598a9e52 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 038c5d9d1d7a49c981c4c34fbc00d9e6 (127.15.255.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: false } } }
I20240502 21:27:46.867512 25879 raft_consensus.cc:2932] T cca45c89b9fe40e7800d95c1598a9e52 P ee27c3f9e0ec46caa742ac386d6b7c20 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 038c5d9d1d7a49c981c4c34fbc00d9e6 (127.15.255.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: false } } }
I20240502 21:27:46.870695 23756 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet cca45c89b9fe40e7800d95c1598a9e52 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:27:46.872315 23770 catalog_manager.cc:5484] T cca45c89b9fe40e7800d95c1598a9e52 P 009d3e7e8a8b4e2497de3a73d01ced49 reported cstate change: config changed from index 5 to 6, VOTER 038c5d9d1d7a49c981c4c34fbc00d9e6 (127.15.255.132) evicted. New cstate: current_term: 1 leader_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:27:46.877405 26280 raft_consensus.cc:1203] T 549666d19e724a96801db183a07809d8 P a3e500d7077a48acb5edb33f7e178c81 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 21:27:46.878873 24297 tablet_service.cc:1514] Processing DeleteTablet for tablet cca45c89b9fe40e7800d95c1598a9e52 with delete_type TABLET_DATA_TOMBSTONED (TS 038c5d9d1d7a49c981c4c34fbc00d9e6 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:58268
I20240502 21:27:46.879355 24048 raft_consensus.cc:2932] T cca45c89b9fe40e7800d95c1598a9e52 P 6c194a6db6c145acafc0f3068b8ce1fd [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 038c5d9d1d7a49c981c4c34fbc00d9e6 (127.15.255.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: false } } }
I20240502 21:27:46.882755 27224 tablet_replica.cc:330] T cca45c89b9fe40e7800d95c1598a9e52 P 038c5d9d1d7a49c981c4c34fbc00d9e6: stopping tablet replica
I20240502 21:27:46.883222 27224 raft_consensus.cc:2229] T cca45c89b9fe40e7800d95c1598a9e52 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:46.883518 27224 raft_consensus.cc:2258] T cca45c89b9fe40e7800d95c1598a9e52 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:46.885798 27224 ts_tablet_manager.cc:1875] T cca45c89b9fe40e7800d95c1598a9e52 P 038c5d9d1d7a49c981c4c34fbc00d9e6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:27:46.889847 26280 raft_consensus.cc:1203] T adf80b9d71314010a7a4cf052fd8a511 P a3e500d7077a48acb5edb33f7e178c81 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 21:27:46.894002 27224 ts_tablet_manager.cc:1888] T cca45c89b9fe40e7800d95c1598a9e52 P 038c5d9d1d7a49c981c4c34fbc00d9e6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 21:27:46.894138 27224 log.cc:1196] T cca45c89b9fe40e7800d95c1598a9e52 P 038c5d9d1d7a49c981c4c34fbc00d9e6: Deleting WAL directory at /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-3/wal/wals/cca45c89b9fe40e7800d95c1598a9e52
I20240502 21:27:46.894946 23755 catalog_manager.cc:4830] TS 038c5d9d1d7a49c981c4c34fbc00d9e6 (127.15.255.132:46813): tablet cca45c89b9fe40e7800d95c1598a9e52 (table rebalance_test_table_2 [id=85ca987b38d84a7db5119dfc88b4feaf]) successfully deleted
I20240502 21:27:46.918079 26790 tablet_bootstrap.cc:492] T a2a0537a60f849daae2061c2b0c4c89f P a3e500d7077a48acb5edb33f7e178c81: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:27:46.918426 26790 tablet_bootstrap.cc:492] T a2a0537a60f849daae2061c2b0c4c89f P a3e500d7077a48acb5edb33f7e178c81: Bootstrap complete.
I20240502 21:27:46.918644 26790 ts_tablet_manager.cc:1367] T a2a0537a60f849daae2061c2b0c4c89f P a3e500d7077a48acb5edb33f7e178c81: Time spent bootstrapping tablet: real 0.059s user 0.050s sys 0.007s
I20240502 21:27:46.919242 26790 raft_consensus.cc:348] T a2a0537a60f849daae2061c2b0c4c89f P a3e500d7077a48acb5edb33f7e178c81 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } attrs { replace: true } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: true } }
I20240502 21:27:46.919432 26790 raft_consensus.cc:729] T a2a0537a60f849daae2061c2b0c4c89f P a3e500d7077a48acb5edb33f7e178c81 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a3e500d7077a48acb5edb33f7e178c81, State: Initialized, Role: LEARNER
I20240502 21:27:46.919581 26790 consensus_queue.cc:260] T a2a0537a60f849daae2061c2b0c4c89f P a3e500d7077a48acb5edb33f7e178c81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } attrs { replace: true } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: true } }
I20240502 21:27:46.920017 26790 ts_tablet_manager.cc:1398] T a2a0537a60f849daae2061c2b0c4c89f P a3e500d7077a48acb5edb33f7e178c81: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 21:27:46.920766 24205 tablet_copy_service.cc:342] P 009d3e7e8a8b4e2497de3a73d01ced49: Request end of tablet copy session a3e500d7077a48acb5edb33f7e178c81-a2a0537a60f849daae2061c2b0c4c89f received from {username='slave'} at 127.15.255.136:51951
I20240502 21:27:46.920894 24205 tablet_copy_service.cc:433] P 009d3e7e8a8b4e2497de3a73d01ced49: ending tablet copy session a3e500d7077a48acb5edb33f7e178c81-a2a0537a60f849daae2061c2b0c4c89f on tablet a2a0537a60f849daae2061c2b0c4c89f with peer a3e500d7077a48acb5edb33f7e178c81
I20240502 21:27:46.952708 26013 raft_consensus.cc:1203] T 89f86d58fe414b579845214534d338d8 P fb6004b8640a4906aa0cfe63e383d40e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 21:27:46.981954 26013 raft_consensus.cc:1203] T 906cfe7530754271bb1ee57c470109bd P fb6004b8640a4906aa0cfe63e383d40e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 21:27:47.099084 26833 tablet_copy_client.cc:523] T 3cb3e62ca0e94e659efb3b7b0c2c9c91 P e117afbf8819424e8a98476453dfde2a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:27:47.101768 26147 raft_consensus.cc:1203] T 11cb9a2fbda946fb9f303f2c465bccb6 P e117afbf8819424e8a98476453dfde2a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 21:27:47.102519 26833 tablet_bootstrap.cc:492] T 3cb3e62ca0e94e659efb3b7b0c2c9c91 P e117afbf8819424e8a98476453dfde2a: Bootstrap starting.
I20240502 21:27:47.109902 26814 tablet_copy_client.cc:523] T fe044374ad7b4999afc72ab812466749 P ee27c3f9e0ec46caa742ac386d6b7c20: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:27:47.113672 26814 tablet_bootstrap.cc:492] T fe044374ad7b4999afc72ab812466749 P ee27c3f9e0ec46caa742ac386d6b7c20: Bootstrap starting.
I20240502 21:27:47.152487 26833 tablet_bootstrap.cc:492] T 3cb3e62ca0e94e659efb3b7b0c2c9c91 P e117afbf8819424e8a98476453dfde2a: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:27:47.153081 26833 tablet_bootstrap.cc:492] T 3cb3e62ca0e94e659efb3b7b0c2c9c91 P e117afbf8819424e8a98476453dfde2a: Bootstrap complete.
I20240502 21:27:47.153457 26833 ts_tablet_manager.cc:1367] T 3cb3e62ca0e94e659efb3b7b0c2c9c91 P e117afbf8819424e8a98476453dfde2a: Time spent bootstrapping tablet: real 0.051s user 0.044s sys 0.005s
I20240502 21:27:47.154352 26833 raft_consensus.cc:348] T 3cb3e62ca0e94e659efb3b7b0c2c9c91 P e117afbf8819424e8a98476453dfde2a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } attrs { replace: true } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: NON_VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: true } }
I20240502 21:27:47.154666 26833 raft_consensus.cc:729] T 3cb3e62ca0e94e659efb3b7b0c2c9c91 P e117afbf8819424e8a98476453dfde2a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e117afbf8819424e8a98476453dfde2a, State: Initialized, Role: LEARNER
I20240502 21:27:47.154896 26833 consensus_queue.cc:260] T 3cb3e62ca0e94e659efb3b7b0c2c9c91 P e117afbf8819424e8a98476453dfde2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } attrs { replace: true } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: NON_VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: true } }
I20240502 21:27:47.155959 26833 ts_tablet_manager.cc:1398] T 3cb3e62ca0e94e659efb3b7b0c2c9c91 P e117afbf8819424e8a98476453dfde2a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:27:47.157617 24070 tablet_copy_service.cc:342] P 6c194a6db6c145acafc0f3068b8ce1fd: Request end of tablet copy session e117afbf8819424e8a98476453dfde2a-3cb3e62ca0e94e659efb3b7b0c2c9c91 received from {username='slave'} at 127.15.255.135:53383
I20240502 21:27:47.157802 24070 tablet_copy_service.cc:433] P 6c194a6db6c145acafc0f3068b8ce1fd: ending tablet copy session e117afbf8819424e8a98476453dfde2a-3cb3e62ca0e94e659efb3b7b0c2c9c91 on tablet 3cb3e62ca0e94e659efb3b7b0c2c9c91 with peer e117afbf8819424e8a98476453dfde2a
I20240502 21:27:47.165226 26814 tablet_bootstrap.cc:492] T fe044374ad7b4999afc72ab812466749 P ee27c3f9e0ec46caa742ac386d6b7c20: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:27:47.165655 26814 tablet_bootstrap.cc:492] T fe044374ad7b4999afc72ab812466749 P ee27c3f9e0ec46caa742ac386d6b7c20: Bootstrap complete.
I20240502 21:27:47.165938 26814 ts_tablet_manager.cc:1367] T fe044374ad7b4999afc72ab812466749 P ee27c3f9e0ec46caa742ac386d6b7c20: Time spent bootstrapping tablet: real 0.052s user 0.047s sys 0.003s
I20240502 21:27:47.166707 26814 raft_consensus.cc:348] T fe044374ad7b4999afc72ab812466749 P ee27c3f9e0ec46caa742ac386d6b7c20 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } attrs { replace: true } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: NON_VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: true } }
I20240502 21:27:47.166954 26814 raft_consensus.cc:729] T fe044374ad7b4999afc72ab812466749 P ee27c3f9e0ec46caa742ac386d6b7c20 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ee27c3f9e0ec46caa742ac386d6b7c20, State: Initialized, Role: LEARNER
I20240502 21:27:47.167192 26814 consensus_queue.cc:260] T fe044374ad7b4999afc72ab812466749 P ee27c3f9e0ec46caa742ac386d6b7c20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } attrs { replace: true } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: NON_VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: true } }
I20240502 21:27:47.167806 26814 ts_tablet_manager.cc:1398] T fe044374ad7b4999afc72ab812466749 P ee27c3f9e0ec46caa742ac386d6b7c20: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 21:27:47.168612 24338 tablet_copy_service.cc:342] P 038c5d9d1d7a49c981c4c34fbc00d9e6: Request end of tablet copy session ee27c3f9e0ec46caa742ac386d6b7c20-fe044374ad7b4999afc72ab812466749 received from {username='slave'} at 127.15.255.133:45805
I20240502 21:27:47.168761 24338 tablet_copy_service.cc:433] P 038c5d9d1d7a49c981c4c34fbc00d9e6: ending tablet copy session ee27c3f9e0ec46caa742ac386d6b7c20-fe044374ad7b4999afc72ab812466749 on tablet fe044374ad7b4999afc72ab812466749 with peer ee27c3f9e0ec46caa742ac386d6b7c20
I20240502 21:27:47.176465 25879 raft_consensus.cc:1203] T 8f14db150fda449e962be2466f1d2a0e P ee27c3f9e0ec46caa742ac386d6b7c20 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 21:27:47.198374 26835 tablet_copy_client.cc:523] T a274546743354e79bffa26ae7000e895 P e117afbf8819424e8a98476453dfde2a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:27:47.202142 26835 tablet_bootstrap.cc:492] T a274546743354e79bffa26ae7000e895 P e117afbf8819424e8a98476453dfde2a: Bootstrap starting.
I20240502 21:27:47.235416 26586 raft_consensus.cc:1050] T 4eebf4508766421eb6c547a0fbbc28b1 P 6810a866decb4f5089f7d35c870e4204: attempting to promote NON_VOTER e117afbf8819424e8a98476453dfde2a to VOTER
I20240502 21:27:47.236536 26586 consensus_queue.cc:237] T 4eebf4508766421eb6c547a0fbbc28b1 P 6810a866decb4f5089f7d35c870e4204 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } attrs { replace: true } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } }
I20240502 21:27:47.238986 26147 raft_consensus.cc:1261] T 4eebf4508766421eb6c547a0fbbc28b1 P e117afbf8819424e8a98476453dfde2a [term 1 LEARNER]: Refusing update from remote peer 6810a866decb4f5089f7d35c870e4204: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:27:47.239449 24315 raft_consensus.cc:1261] T 4eebf4508766421eb6c547a0fbbc28b1 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 1 FOLLOWER]: Refusing update from remote peer 6810a866decb4f5089f7d35c870e4204: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:27:47.239984 24184 raft_consensus.cc:1261] T 4eebf4508766421eb6c547a0fbbc28b1 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 1 FOLLOWER]: Refusing update from remote peer 6810a866decb4f5089f7d35c870e4204: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:27:47.239956 26588 consensus_queue.cc:1035] T 4eebf4508766421eb6c547a0fbbc28b1 P 6810a866decb4f5089f7d35c870e4204 [LEADER]: Connected to new peer: Peer: permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:27:47.241909 26475 raft_consensus.cc:1050] T 0b3c514213bb40ca98955d6409a3b372 P 009d3e7e8a8b4e2497de3a73d01ced49: attempting to promote NON_VOTER fb6004b8640a4906aa0cfe63e383d40e to VOTER
I20240502 21:27:47.242727 26475 consensus_queue.cc:237] T 0b3c514213bb40ca98955d6409a3b372 P 009d3e7e8a8b4e2497de3a73d01ced49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } attrs { replace: true } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: false } }
I20240502 21:27:47.244261 27234 consensus_queue.cc:1035] T 4eebf4508766421eb6c547a0fbbc28b1 P 6810a866decb4f5089f7d35c870e4204 [LEADER]: Connected to new peer: Peer: permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:27:47.244884 27234 consensus_queue.cc:1035] T 4eebf4508766421eb6c547a0fbbc28b1 P 6810a866decb4f5089f7d35c870e4204 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:27:47.249789 26013 raft_consensus.cc:1261] T 0b3c514213bb40ca98955d6409a3b372 P fb6004b8640a4906aa0cfe63e383d40e [term 1 LEARNER]: Refusing update from remote peer 009d3e7e8a8b4e2497de3a73d01ced49: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:27:47.249848 24316 raft_consensus.cc:1261] T 0b3c514213bb40ca98955d6409a3b372 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 1 FOLLOWER]: Refusing update from remote peer 009d3e7e8a8b4e2497de3a73d01ced49: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:27:47.250360 23916 raft_consensus.cc:1261] T 0b3c514213bb40ca98955d6409a3b372 P 6810a866decb4f5089f7d35c870e4204 [term 1 FOLLOWER]: Refusing update from remote peer 009d3e7e8a8b4e2497de3a73d01ced49: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:27:47.250409 26475 consensus_queue.cc:1035] T 0b3c514213bb40ca98955d6409a3b372 P 009d3e7e8a8b4e2497de3a73d01ced49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:27:47.250730 26554 consensus_queue.cc:1035] T 0b3c514213bb40ca98955d6409a3b372 P 009d3e7e8a8b4e2497de3a73d01ced49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:27:47.251526 27215 consensus_queue.cc:1035] T 0b3c514213bb40ca98955d6409a3b372 P 009d3e7e8a8b4e2497de3a73d01ced49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:27:47.254127 26835 tablet_bootstrap.cc:492] T a274546743354e79bffa26ae7000e895 P e117afbf8819424e8a98476453dfde2a: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:27:47.254572 26835 tablet_bootstrap.cc:492] T a274546743354e79bffa26ae7000e895 P e117afbf8819424e8a98476453dfde2a: Bootstrap complete.
I20240502 21:27:47.254890 26835 ts_tablet_manager.cc:1367] T a274546743354e79bffa26ae7000e895 P e117afbf8819424e8a98476453dfde2a: Time spent bootstrapping tablet: real 0.053s user 0.048s sys 0.000s
I20240502 21:27:47.255950 26835 raft_consensus.cc:348] T a274546743354e79bffa26ae7000e895 P e117afbf8819424e8a98476453dfde2a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } attrs { replace: true } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: NON_VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: true } }
I20240502 21:27:47.256187 26835 raft_consensus.cc:729] T a274546743354e79bffa26ae7000e895 P e117afbf8819424e8a98476453dfde2a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e117afbf8819424e8a98476453dfde2a, State: Initialized, Role: LEARNER
I20240502 21:27:47.256379 26835 consensus_queue.cc:260] T a274546743354e79bffa26ae7000e895 P e117afbf8819424e8a98476453dfde2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } attrs { replace: true } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: NON_VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: true } }
I20240502 21:27:47.258405 26835 ts_tablet_manager.cc:1398] T a274546743354e79bffa26ae7000e895 P e117afbf8819424e8a98476453dfde2a: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 21:27:47.258528 26588 raft_consensus.cc:2932] T 4eebf4508766421eb6c547a0fbbc28b1 P 6810a866decb4f5089f7d35c870e4204 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e117afbf8819424e8a98476453dfde2a (127.15.255.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } attrs { replace: true } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } } }
I20240502 21:27:47.259320 24070 tablet_copy_service.cc:342] P 6c194a6db6c145acafc0f3068b8ce1fd: Request end of tablet copy session e117afbf8819424e8a98476453dfde2a-a274546743354e79bffa26ae7000e895 received from {username='slave'} at 127.15.255.135:53383
I20240502 21:27:47.259430 24070 tablet_copy_service.cc:433] P 6c194a6db6c145acafc0f3068b8ce1fd: ending tablet copy session e117afbf8819424e8a98476453dfde2a-a274546743354e79bffa26ae7000e895 on tablet a274546743354e79bffa26ae7000e895 with peer e117afbf8819424e8a98476453dfde2a
I20240502 21:27:47.262929 26147 raft_consensus.cc:2932] T 4eebf4508766421eb6c547a0fbbc28b1 P e117afbf8819424e8a98476453dfde2a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e117afbf8819424e8a98476453dfde2a (127.15.255.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } attrs { replace: true } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } } }
I20240502 21:27:47.267151 23768 catalog_manager.cc:5484] T 4eebf4508766421eb6c547a0fbbc28b1 P e117afbf8819424e8a98476453dfde2a reported cstate change: config changed from index 4 to 5, e117afbf8819424e8a98476453dfde2a (127.15.255.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "6810a866decb4f5089f7d35c870e4204" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } attrs { replace: true } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } } }
I20240502 21:27:47.268551 26475 raft_consensus.cc:2932] T 0b3c514213bb40ca98955d6409a3b372 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, fb6004b8640a4906aa0cfe63e383d40e (127.15.255.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } attrs { replace: true } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: false } } }
I20240502 21:27:47.269382 24184 raft_consensus.cc:2932] T 4eebf4508766421eb6c547a0fbbc28b1 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e117afbf8819424e8a98476453dfde2a (127.15.255.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } attrs { replace: true } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } } }
I20240502 21:27:47.271203 24316 raft_consensus.cc:2932] T 0b3c514213bb40ca98955d6409a3b372 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, fb6004b8640a4906aa0cfe63e383d40e (127.15.255.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } attrs { replace: true } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: false } } }
I20240502 21:27:47.273182 26013 raft_consensus.cc:2932] T 0b3c514213bb40ca98955d6409a3b372 P fb6004b8640a4906aa0cfe63e383d40e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, fb6004b8640a4906aa0cfe63e383d40e (127.15.255.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } attrs { replace: true } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: false } } }
I20240502 21:27:47.275782 26280 raft_consensus.cc:1203] T a2a0537a60f849daae2061c2b0c4c89f P a3e500d7077a48acb5edb33f7e178c81 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 21:27:47.276988 26852 tablet_copy_client.cc:523] T d5697d8b06094211b261f144de9d46f8 P a3e500d7077a48acb5edb33f7e178c81: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:27:47.277004 26849 tablet_copy_client.cc:523] T fcc9cbeae6f343af9f84c73349f3dadb P fb6004b8640a4906aa0cfe63e383d40e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:27:47.280304 26852 tablet_bootstrap.cc:492] T d5697d8b06094211b261f144de9d46f8 P a3e500d7077a48acb5edb33f7e178c81: Bootstrap starting.
I20240502 21:27:47.280900 23768 catalog_manager.cc:5484] T 0b3c514213bb40ca98955d6409a3b372 P fb6004b8640a4906aa0cfe63e383d40e reported cstate change: config changed from index 4 to 5, fb6004b8640a4906aa0cfe63e383d40e (127.15.255.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } attrs { replace: true } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: false } } }
I20240502 21:27:47.282789 27215 raft_consensus.cc:1050] T 6bd759d700a24e9f8ad11f243cf7ebfa P 009d3e7e8a8b4e2497de3a73d01ced49: attempting to promote NON_VOTER ee27c3f9e0ec46caa742ac386d6b7c20 to VOTER
I20240502 21:27:47.283569 27215 consensus_queue.cc:237] T 6bd759d700a24e9f8ad11f243cf7ebfa P 009d3e7e8a8b4e2497de3a73d01ced49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } attrs { replace: true } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: false } }
I20240502 21:27:47.287575 24317 raft_consensus.cc:2932] T 4eebf4508766421eb6c547a0fbbc28b1 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e117afbf8819424e8a98476453dfde2a (127.15.255.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } attrs { replace: true } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } } }
I20240502 21:27:47.292471 26849 tablet_bootstrap.cc:492] T fcc9cbeae6f343af9f84c73349f3dadb P fb6004b8640a4906aa0cfe63e383d40e: Bootstrap starting.
I20240502 21:27:47.294513 23916 consensus_queue.cc:237] T 4eebf4508766421eb6c547a0fbbc28b1 P 6810a866decb4f5089f7d35c870e4204 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } }
I20240502 21:27:47.297338 26147 raft_consensus.cc:1261] T 4eebf4508766421eb6c547a0fbbc28b1 P e117afbf8819424e8a98476453dfde2a [term 1 FOLLOWER]: Refusing update from remote peer 6810a866decb4f5089f7d35c870e4204: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:27:47.297907 26586 consensus_queue.cc:1035] T 4eebf4508766421eb6c547a0fbbc28b1 P 6810a866decb4f5089f7d35c870e4204 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:27:47.298102 24317 raft_consensus.cc:1261] T 4eebf4508766421eb6c547a0fbbc28b1 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 1 FOLLOWER]: Refusing update from remote peer 6810a866decb4f5089f7d35c870e4204: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:27:47.298750 27234 consensus_queue.cc:1035] T 4eebf4508766421eb6c547a0fbbc28b1 P 6810a866decb4f5089f7d35c870e4204 [LEADER]: Connected to new peer: Peer: permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:27:47.300163 26471 raft_consensus.cc:1050] T 549666d19e724a96801db183a07809d8 P 6810a866decb4f5089f7d35c870e4204: attempting to promote NON_VOTER a3e500d7077a48acb5edb33f7e178c81 to VOTER
I20240502 21:27:47.300978 26471 consensus_queue.cc:237] T 549666d19e724a96801db183a07809d8 P 6810a866decb4f5089f7d35c870e4204 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } attrs { replace: true } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } }
I20240502 21:27:47.301582 26586 raft_consensus.cc:2932] T 4eebf4508766421eb6c547a0fbbc28b1 P 6810a866decb4f5089f7d35c870e4204 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 009d3e7e8a8b4e2497de3a73d01ced49 (127.15.255.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } } }
I20240502 21:27:47.305070 26147 raft_consensus.cc:2932] T 4eebf4508766421eb6c547a0fbbc28b1 P e117afbf8819424e8a98476453dfde2a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 009d3e7e8a8b4e2497de3a73d01ced49 (127.15.255.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } } }
I20240502 21:27:47.307982 23916 raft_consensus.cc:1261] T 6bd759d700a24e9f8ad11f243cf7ebfa P 6810a866decb4f5089f7d35c870e4204 [term 1 FOLLOWER]: Refusing update from remote peer 009d3e7e8a8b4e2497de3a73d01ced49: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:27:47.308280 25879 raft_consensus.cc:1261] T 6bd759d700a24e9f8ad11f243cf7ebfa P ee27c3f9e0ec46caa742ac386d6b7c20 [term 1 LEARNER]: Refusing update from remote peer 009d3e7e8a8b4e2497de3a73d01ced49: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:27:47.309286 25728 consensus_queue.cc:1035] T 6bd759d700a24e9f8ad11f243cf7ebfa P 009d3e7e8a8b4e2497de3a73d01ced49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:27:47.309845 25728 consensus_queue.cc:1035] T 6bd759d700a24e9f8ad11f243cf7ebfa P 009d3e7e8a8b4e2497de3a73d01ced49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:27:47.311185 23757 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 4eebf4508766421eb6c547a0fbbc28b1 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:27:47.314631 23766 catalog_manager.cc:5484] T 4eebf4508766421eb6c547a0fbbc28b1 P 6810a866decb4f5089f7d35c870e4204 reported cstate change: config changed from index 5 to 6, VOTER 009d3e7e8a8b4e2497de3a73d01ced49 (127.15.255.131) evicted. New cstate: current_term: 1 leader_uuid: "6810a866decb4f5089f7d35c870e4204" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:27:47.315480 24317 raft_consensus.cc:2932] T 4eebf4508766421eb6c547a0fbbc28b1 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 009d3e7e8a8b4e2497de3a73d01ced49 (127.15.255.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } } }
I20240502 21:27:47.319855 24162 tablet_service.cc:1514] Processing DeleteTablet for tablet 4eebf4508766421eb6c547a0fbbc28b1 with delete_type TABLET_DATA_TOMBSTONED (TS 009d3e7e8a8b4e2497de3a73d01ced49 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:33102
I20240502 21:27:47.322391 24316 raft_consensus.cc:1261] T 6bd759d700a24e9f8ad11f243cf7ebfa P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 1 FOLLOWER]: Refusing update from remote peer 009d3e7e8a8b4e2497de3a73d01ced49: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:27:47.322906 26554 consensus_queue.cc:1035] T 6bd759d700a24e9f8ad11f243cf7ebfa P 009d3e7e8a8b4e2497de3a73d01ced49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:27:47.328289 23915 raft_consensus.cc:2932] T 0b3c514213bb40ca98955d6409a3b372 P 6810a866decb4f5089f7d35c870e4204 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, fb6004b8640a4906aa0cfe63e383d40e (127.15.255.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } attrs { replace: true } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: false } } }
I20240502 21:27:47.336445 26876 tablet_copy_client.cc:523] T 16bb47f03a2f46fdb7344e018f07fdcb P ee27c3f9e0ec46caa742ac386d6b7c20: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:27:47.337189 24184 consensus_queue.cc:237] T 0b3c514213bb40ca98955d6409a3b372 P 009d3e7e8a8b4e2497de3a73d01ced49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: false } }
I20240502 21:27:47.339805 26876 tablet_bootstrap.cc:492] T 16bb47f03a2f46fdb7344e018f07fdcb P ee27c3f9e0ec46caa742ac386d6b7c20: Bootstrap starting.
I20240502 21:27:47.341939 24316 raft_consensus.cc:1261] T 0b3c514213bb40ca98955d6409a3b372 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 1 FOLLOWER]: Refusing update from remote peer 009d3e7e8a8b4e2497de3a73d01ced49: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:27:47.342504 25728 consensus_queue.cc:1035] T 0b3c514213bb40ca98955d6409a3b372 P 009d3e7e8a8b4e2497de3a73d01ced49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:27:47.343655 26013 raft_consensus.cc:1261] T 0b3c514213bb40ca98955d6409a3b372 P fb6004b8640a4906aa0cfe63e383d40e [term 1 FOLLOWER]: Refusing update from remote peer 009d3e7e8a8b4e2497de3a73d01ced49: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:27:47.344552 26554 consensus_queue.cc:1035] T 0b3c514213bb40ca98955d6409a3b372 P 009d3e7e8a8b4e2497de3a73d01ced49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:27:47.348340 26012 raft_consensus.cc:2932] T 0b3c514213bb40ca98955d6409a3b372 P fb6004b8640a4906aa0cfe63e383d40e [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 6810a866decb4f5089f7d35c870e4204 (127.15.255.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: false } } }
I20240502 21:27:47.350229 27296 tablet_replica.cc:330] T 4eebf4508766421eb6c547a0fbbc28b1 P 009d3e7e8a8b4e2497de3a73d01ced49: stopping tablet replica
I20240502 21:27:47.350741 27296 raft_consensus.cc:2229] T 4eebf4508766421eb6c547a0fbbc28b1 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:47.351058 27296 raft_consensus.cc:2258] T 4eebf4508766421eb6c547a0fbbc28b1 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:47.353421 27296 ts_tablet_manager.cc:1875] T 4eebf4508766421eb6c547a0fbbc28b1 P 009d3e7e8a8b4e2497de3a73d01ced49: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:27:47.353678 23766 catalog_manager.cc:5484] T 0b3c514213bb40ca98955d6409a3b372 P fb6004b8640a4906aa0cfe63e383d40e reported cstate change: config changed from index 5 to 6, VOTER 6810a866decb4f5089f7d35c870e4204 (127.15.255.129) evicted. New cstate: current_term: 1 leader_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: false } } }
I20240502 21:27:47.347716 25728 raft_consensus.cc:2932] T 0b3c514213bb40ca98955d6409a3b372 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 6810a866decb4f5089f7d35c870e4204 (127.15.255.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: false } } }
I20240502 21:27:47.359820 23894 tablet_service.cc:1514] Processing DeleteTablet for tablet 0b3c514213bb40ca98955d6409a3b372 with delete_type TABLET_DATA_TOMBSTONED (TS 6810a866decb4f5089f7d35c870e4204 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:33512
I20240502 21:27:47.365217 26879 tablet_copy_client.cc:523] T f84bbe304597450c978ae28e0c4e7a3e P a3e500d7077a48acb5edb33f7e178c81: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:27:47.368067 23756 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 0b3c514213bb40ca98955d6409a3b372 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:27:47.372280 26879 tablet_bootstrap.cc:492] T f84bbe304597450c978ae28e0c4e7a3e P a3e500d7077a48acb5edb33f7e178c81: Bootstrap starting.
I20240502 21:27:47.372844 27296 ts_tablet_manager.cc:1888] T 4eebf4508766421eb6c547a0fbbc28b1 P 009d3e7e8a8b4e2497de3a73d01ced49: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 21:27:47.373003 27296 log.cc:1196] T 4eebf4508766421eb6c547a0fbbc28b1 P 009d3e7e8a8b4e2497de3a73d01ced49: Deleting WAL directory at /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-2/wal/wals/4eebf4508766421eb6c547a0fbbc28b1
I20240502 21:27:47.373869 23756 catalog_manager.cc:4830] TS 009d3e7e8a8b4e2497de3a73d01ced49 (127.15.255.131:37895): tablet 4eebf4508766421eb6c547a0fbbc28b1 (table rebalance_test_table_1 [id=e962de13fbc44d1986b1e1ee191d22e2]) successfully deleted
I20240502 21:27:47.376744 24316 raft_consensus.cc:2932] T 0b3c514213bb40ca98955d6409a3b372 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 6810a866decb4f5089f7d35c870e4204 (127.15.255.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: false } } }
I20240502 21:27:47.398497 27301 tablet_replica.cc:330] T 0b3c514213bb40ca98955d6409a3b372 P 6810a866decb4f5089f7d35c870e4204: stopping tablet replica
I20240502 21:27:47.399111 27301 raft_consensus.cc:2229] T 0b3c514213bb40ca98955d6409a3b372 P 6810a866decb4f5089f7d35c870e4204 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:47.399690 27301 raft_consensus.cc:2258] T 0b3c514213bb40ca98955d6409a3b372 P 6810a866decb4f5089f7d35c870e4204 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:47.401171 25728 raft_consensus.cc:2932] T 6bd759d700a24e9f8ad11f243cf7ebfa P 009d3e7e8a8b4e2497de3a73d01ced49 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ee27c3f9e0ec46caa742ac386d6b7c20 (127.15.255.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } attrs { replace: true } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: false } } }
I20240502 21:27:47.402160 25879 raft_consensus.cc:2932] T 6bd759d700a24e9f8ad11f243cf7ebfa P ee27c3f9e0ec46caa742ac386d6b7c20 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ee27c3f9e0ec46caa742ac386d6b7c20 (127.15.255.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } attrs { replace: true } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: false } } }
I20240502 21:27:47.406116 24316 raft_consensus.cc:2932] T 6bd759d700a24e9f8ad11f243cf7ebfa P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ee27c3f9e0ec46caa742ac386d6b7c20 (127.15.255.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } attrs { replace: true } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: false } } }
I20240502 21:27:47.408485 27301 ts_tablet_manager.cc:1875] T 0b3c514213bb40ca98955d6409a3b372 P 6810a866decb4f5089f7d35c870e4204: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:27:47.411729 26280 raft_consensus.cc:1261] T 549666d19e724a96801db183a07809d8 P a3e500d7077a48acb5edb33f7e178c81 [term 1 LEARNER]: Refusing update from remote peer 6810a866decb4f5089f7d35c870e4204: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:27:47.411737 23768 catalog_manager.cc:5484] T 6bd759d700a24e9f8ad11f243cf7ebfa P 009d3e7e8a8b4e2497de3a73d01ced49 reported cstate change: config changed from index 4 to 5, ee27c3f9e0ec46caa742ac386d6b7c20 (127.15.255.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:27:47.412725 26586 consensus_queue.cc:1035] T 549666d19e724a96801db183a07809d8 P 6810a866decb4f5089f7d35c870e4204 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:27:47.415381 26876 tablet_bootstrap.cc:492] T 16bb47f03a2f46fdb7344e018f07fdcb P ee27c3f9e0ec46caa742ac386d6b7c20: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:27:47.415939 26876 tablet_bootstrap.cc:492] T 16bb47f03a2f46fdb7344e018f07fdcb P ee27c3f9e0ec46caa742ac386d6b7c20: Bootstrap complete.
I20240502 21:27:47.416352 26876 ts_tablet_manager.cc:1367] T 16bb47f03a2f46fdb7344e018f07fdcb P ee27c3f9e0ec46caa742ac386d6b7c20: Time spent bootstrapping tablet: real 0.077s user 0.036s sys 0.010s
I20240502 21:27:47.421639 26876 raft_consensus.cc:348] T 16bb47f03a2f46fdb7344e018f07fdcb P ee27c3f9e0ec46caa742ac386d6b7c20 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } attrs { replace: true } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: NON_VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: true } }
I20240502 21:27:47.422055 26876 raft_consensus.cc:729] T 16bb47f03a2f46fdb7344e018f07fdcb P ee27c3f9e0ec46caa742ac386d6b7c20 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ee27c3f9e0ec46caa742ac386d6b7c20, State: Initialized, Role: LEARNER
I20240502 21:27:47.422364 26876 consensus_queue.cc:260] T 16bb47f03a2f46fdb7344e018f07fdcb P ee27c3f9e0ec46caa742ac386d6b7c20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } attrs { replace: true } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: NON_VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: true } }
I20240502 21:27:47.422940 24048 raft_consensus.cc:1261] T 549666d19e724a96801db183a07809d8 P 6c194a6db6c145acafc0f3068b8ce1fd [term 1 FOLLOWER]: Refusing update from remote peer 6810a866decb4f5089f7d35c870e4204: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:27:47.423545 26586 consensus_queue.cc:1035] T 549666d19e724a96801db183a07809d8 P 6810a866decb4f5089f7d35c870e4204 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:27:47.423825 26876 ts_tablet_manager.cc:1398] T 16bb47f03a2f46fdb7344e018f07fdcb P ee27c3f9e0ec46caa742ac386d6b7c20: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240502 21:27:47.425202 24205 tablet_copy_service.cc:342] P 009d3e7e8a8b4e2497de3a73d01ced49: Request end of tablet copy session ee27c3f9e0ec46caa742ac386d6b7c20-16bb47f03a2f46fdb7344e018f07fdcb received from {username='slave'} at 127.15.255.133:44245
I20240502 21:27:47.425336 24205 tablet_copy_service.cc:433] P 009d3e7e8a8b4e2497de3a73d01ced49: ending tablet copy session ee27c3f9e0ec46caa742ac386d6b7c20-16bb47f03a2f46fdb7344e018f07fdcb on tablet 16bb47f03a2f46fdb7344e018f07fdcb with peer ee27c3f9e0ec46caa742ac386d6b7c20
I20240502 21:27:47.431109 24184 consensus_queue.cc:237] T 6bd759d700a24e9f8ad11f243cf7ebfa P 009d3e7e8a8b4e2497de3a73d01ced49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: false } }
I20240502 21:27:47.433045 25879 raft_consensus.cc:1261] T 6bd759d700a24e9f8ad11f243cf7ebfa P ee27c3f9e0ec46caa742ac386d6b7c20 [term 1 FOLLOWER]: Refusing update from remote peer 009d3e7e8a8b4e2497de3a73d01ced49: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:27:47.433214 23915 raft_consensus.cc:1261] T 6bd759d700a24e9f8ad11f243cf7ebfa P 6810a866decb4f5089f7d35c870e4204 [term 1 FOLLOWER]: Refusing update from remote peer 009d3e7e8a8b4e2497de3a73d01ced49: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:27:47.433761 27215 consensus_queue.cc:1035] T 6bd759d700a24e9f8ad11f243cf7ebfa P 009d3e7e8a8b4e2497de3a73d01ced49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:27:47.434337 27292 consensus_queue.cc:1035] T 6bd759d700a24e9f8ad11f243cf7ebfa P 009d3e7e8a8b4e2497de3a73d01ced49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:27:47.434906 23915 raft_consensus.cc:2932] T 6bd759d700a24e9f8ad11f243cf7ebfa P 6810a866decb4f5089f7d35c870e4204 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ee27c3f9e0ec46caa742ac386d6b7c20 (127.15.255.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } attrs { replace: true } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: false } } }
I20240502 21:27:47.437606 27292 raft_consensus.cc:2932] T 6bd759d700a24e9f8ad11f243cf7ebfa P 009d3e7e8a8b4e2497de3a73d01ced49 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 038c5d9d1d7a49c981c4c34fbc00d9e6 (127.15.255.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: false } } }
I20240502 21:27:47.438258 25878 raft_consensus.cc:2932] T 6bd759d700a24e9f8ad11f243cf7ebfa P ee27c3f9e0ec46caa742ac386d6b7c20 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 038c5d9d1d7a49c981c4c34fbc00d9e6 (127.15.255.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: false } } }
I20240502 21:27:47.441614 23756 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 6bd759d700a24e9f8ad11f243cf7ebfa with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:27:47.441942 27234 raft_consensus.cc:1050] T 89f86d58fe414b579845214534d338d8 P 6810a866decb4f5089f7d35c870e4204: attempting to promote NON_VOTER fb6004b8640a4906aa0cfe63e383d40e to VOTER
I20240502 21:27:47.442770 27234 consensus_queue.cc:237] T 89f86d58fe414b579845214534d338d8 P 6810a866decb4f5089f7d35c870e4204 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } attrs { replace: true } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: false } }
I20240502 21:27:47.444399 23768 catalog_manager.cc:5484] T 6bd759d700a24e9f8ad11f243cf7ebfa P 009d3e7e8a8b4e2497de3a73d01ced49 reported cstate change: config changed from index 5 to 6, VOTER 038c5d9d1d7a49c981c4c34fbc00d9e6 (127.15.255.132) evicted. New cstate: current_term: 1 leader_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:27:47.449611 23914 raft_consensus.cc:2932] T 6bd759d700a24e9f8ad11f243cf7ebfa P 6810a866decb4f5089f7d35c870e4204 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 038c5d9d1d7a49c981c4c34fbc00d9e6 (127.15.255.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: false } } }
I20240502 21:27:47.454737 25879 raft_consensus.cc:1203] T fe044374ad7b4999afc72ab812466749 P ee27c3f9e0ec46caa742ac386d6b7c20 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 21:27:47.460069 24184 raft_consensus.cc:1261] T 549666d19e724a96801db183a07809d8 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 1 FOLLOWER]: Refusing update from remote peer 6810a866decb4f5089f7d35c870e4204: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:27:47.461069 26588 consensus_queue.cc:1035] T 549666d19e724a96801db183a07809d8 P 6810a866decb4f5089f7d35c870e4204 [LEADER]: Connected to new peer: Peer: permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:27:47.473817 24297 tablet_service.cc:1514] Processing DeleteTablet for tablet 6bd759d700a24e9f8ad11f243cf7ebfa with delete_type TABLET_DATA_TOMBSTONED (TS 038c5d9d1d7a49c981c4c34fbc00d9e6 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:58268
I20240502 21:27:47.474797 26588 raft_consensus.cc:2932] T 549666d19e724a96801db183a07809d8 P 6810a866decb4f5089f7d35c870e4204 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a3e500d7077a48acb5edb33f7e178c81 (127.15.255.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } attrs { replace: true } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } } }
I20240502 21:27:47.477979 26013 raft_consensus.cc:1261] T 89f86d58fe414b579845214534d338d8 P fb6004b8640a4906aa0cfe63e383d40e [term 1 LEARNER]: Refusing update from remote peer 6810a866decb4f5089f7d35c870e4204: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:27:47.479015 27301 ts_tablet_manager.cc:1888] T 0b3c514213bb40ca98955d6409a3b372 P 6810a866decb4f5089f7d35c870e4204: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 21:27:47.479172 27301 log.cc:1196] T 0b3c514213bb40ca98955d6409a3b372 P 6810a866decb4f5089f7d35c870e4204: Deleting WAL directory at /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-0/wal/wals/0b3c514213bb40ca98955d6409a3b372
I20240502 21:27:47.479909 26471 consensus_queue.cc:1035] T 89f86d58fe414b579845214534d338d8 P 6810a866decb4f5089f7d35c870e4204 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:27:47.481315 24185 raft_consensus.cc:1261] T 89f86d58fe414b579845214534d338d8 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 1 FOLLOWER]: Refusing update from remote peer 6810a866decb4f5089f7d35c870e4204: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:27:47.483211 26588 consensus_queue.cc:1035] T 89f86d58fe414b579845214534d338d8 P 6810a866decb4f5089f7d35c870e4204 [LEADER]: Connected to new peer: Peer: permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:27:47.485508 26852 tablet_bootstrap.cc:492] T d5697d8b06094211b261f144de9d46f8 P a3e500d7077a48acb5edb33f7e178c81: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:27:47.485268 23766 catalog_manager.cc:5484] T 549666d19e724a96801db183a07809d8 P 6810a866decb4f5089f7d35c870e4204 reported cstate change: config changed from index 4 to 5, a3e500d7077a48acb5edb33f7e178c81 (127.15.255.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "6810a866decb4f5089f7d35c870e4204" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:27:47.485911 26852 tablet_bootstrap.cc:492] T d5697d8b06094211b261f144de9d46f8 P a3e500d7077a48acb5edb33f7e178c81: Bootstrap complete.
I20240502 21:27:47.486193 26852 ts_tablet_manager.cc:1367] T d5697d8b06094211b261f144de9d46f8 P a3e500d7077a48acb5edb33f7e178c81: Time spent bootstrapping tablet: real 0.206s user 0.072s sys 0.066s
I20240502 21:27:47.487841 23757 catalog_manager.cc:4830] TS 6810a866decb4f5089f7d35c870e4204 (127.15.255.129:40435): tablet 0b3c514213bb40ca98955d6409a3b372 (table rebalance_test_table_1 [id=e962de13fbc44d1986b1e1ee191d22e2]) successfully deleted
I20240502 21:27:47.489912 27312 tablet_replica.cc:330] T 6bd759d700a24e9f8ad11f243cf7ebfa P 038c5d9d1d7a49c981c4c34fbc00d9e6: stopping tablet replica
I20240502 21:27:47.490517 26852 raft_consensus.cc:348] T d5697d8b06094211b261f144de9d46f8 P a3e500d7077a48acb5edb33f7e178c81 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } attrs { replace: true } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: true } }
I20240502 21:27:47.490909 27312 raft_consensus.cc:2229] T 6bd759d700a24e9f8ad11f243cf7ebfa P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:47.490797 26852 raft_consensus.cc:729] T d5697d8b06094211b261f144de9d46f8 P a3e500d7077a48acb5edb33f7e178c81 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a3e500d7077a48acb5edb33f7e178c81, State: Initialized, Role: LEARNER
I20240502 21:27:47.491258 26852 consensus_queue.cc:260] T d5697d8b06094211b261f144de9d46f8 P a3e500d7077a48acb5edb33f7e178c81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } attrs { replace: true } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: true } }
I20240502 21:27:47.491432 27312 raft_consensus.cc:2258] T 6bd759d700a24e9f8ad11f243cf7ebfa P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:47.492162 24183 raft_consensus.cc:2932] T 549666d19e724a96801db183a07809d8 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a3e500d7077a48acb5edb33f7e178c81 (127.15.255.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } attrs { replace: true } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } } }
I20240502 21:27:47.492755 26852 ts_tablet_manager.cc:1398] T d5697d8b06094211b261f144de9d46f8 P a3e500d7077a48acb5edb33f7e178c81: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20240502 21:27:47.492839 27312 ts_tablet_manager.cc:1875] T 6bd759d700a24e9f8ad11f243cf7ebfa P 038c5d9d1d7a49c981c4c34fbc00d9e6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:27:47.495713 24338 tablet_copy_service.cc:342] P 038c5d9d1d7a49c981c4c34fbc00d9e6: Request end of tablet copy session a3e500d7077a48acb5edb33f7e178c81-d5697d8b06094211b261f144de9d46f8 received from {username='slave'} at 127.15.255.136:38895
I20240502 21:27:47.495855 24338 tablet_copy_service.cc:433] P 038c5d9d1d7a49c981c4c34fbc00d9e6: ending tablet copy session a3e500d7077a48acb5edb33f7e178c81-d5697d8b06094211b261f144de9d46f8 on tablet d5697d8b06094211b261f144de9d46f8 with peer a3e500d7077a48acb5edb33f7e178c81
I20240502 21:27:47.499490 26586 raft_consensus.cc:1050] T 906cfe7530754271bb1ee57c470109bd P 6810a866decb4f5089f7d35c870e4204: attempting to promote NON_VOTER fb6004b8640a4906aa0cfe63e383d40e to VOTER
I20240502 21:27:47.500357 26586 consensus_queue.cc:237] T 906cfe7530754271bb1ee57c470109bd P 6810a866decb4f5089f7d35c870e4204 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } attrs { replace: true } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: false } }
I20240502 21:27:47.503448 24317 raft_consensus.cc:1261] T 906cfe7530754271bb1ee57c470109bd P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 1 FOLLOWER]: Refusing update from remote peer 6810a866decb4f5089f7d35c870e4204: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:27:47.503890 26012 raft_consensus.cc:1261] T 906cfe7530754271bb1ee57c470109bd P fb6004b8640a4906aa0cfe63e383d40e [term 1 LEARNER]: Refusing update from remote peer 6810a866decb4f5089f7d35c870e4204: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:27:47.504343 26588 consensus_queue.cc:1035] T 906cfe7530754271bb1ee57c470109bd P 6810a866decb4f5089f7d35c870e4204 [LEADER]: Connected to new peer: Peer: permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:27:47.505159 26588 consensus_queue.cc:1035] T 906cfe7530754271bb1ee57c470109bd P 6810a866decb4f5089f7d35c870e4204 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:27:47.505594 27312 ts_tablet_manager.cc:1888] T 6bd759d700a24e9f8ad11f243cf7ebfa P 038c5d9d1d7a49c981c4c34fbc00d9e6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 21:27:47.505707 27312 log.cc:1196] T 6bd759d700a24e9f8ad11f243cf7ebfa P 038c5d9d1d7a49c981c4c34fbc00d9e6: Deleting WAL directory at /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-3/wal/wals/6bd759d700a24e9f8ad11f243cf7ebfa
I20240502 21:27:47.507736 23755 catalog_manager.cc:4830] TS 038c5d9d1d7a49c981c4c34fbc00d9e6 (127.15.255.132:46813): tablet 6bd759d700a24e9f8ad11f243cf7ebfa (table rebalance_test_table_7 [id=754dbb3bce6a4875a0b642faf5315709]) successfully deleted
I20240502 21:27:47.508843 26588 raft_consensus.cc:1050] T adf80b9d71314010a7a4cf052fd8a511 P 6810a866decb4f5089f7d35c870e4204: attempting to promote NON_VOTER a3e500d7077a48acb5edb33f7e178c81 to VOTER
I20240502 21:27:47.509652 26588 consensus_queue.cc:237] T adf80b9d71314010a7a4cf052fd8a511 P 6810a866decb4f5089f7d35c870e4204 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } attrs { replace: true } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } }
W20240502 21:27:47.516312 23981 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.132:46813, user_credentials={real_user=slave}} blocked reactor thread for 57047.2us
I20240502 21:27:47.519482 26147 raft_consensus.cc:1203] T 3cb3e62ca0e94e659efb3b7b0c2c9c91 P e117afbf8819424e8a98476453dfde2a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 21:27:47.521832 23916 consensus_queue.cc:237] T 549666d19e724a96801db183a07809d8 P 6810a866decb4f5089f7d35c870e4204 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } }
I20240502 21:27:47.527922 26944 tablet_copy_client.cc:523] T 83b95f3ae7564b81b31a2f5673ee22bb P e117afbf8819424e8a98476453dfde2a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:27:47.531400 26944 tablet_bootstrap.cc:492] T 83b95f3ae7564b81b31a2f5673ee22bb P e117afbf8819424e8a98476453dfde2a: Bootstrap starting.
I20240502 21:27:47.538029 24316 raft_consensus.cc:1261] T 89f86d58fe414b579845214534d338d8 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 1 FOLLOWER]: Refusing update from remote peer 6810a866decb4f5089f7d35c870e4204: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:27:47.538226 24315 raft_consensus.cc:1261] T adf80b9d71314010a7a4cf052fd8a511 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 1 FOLLOWER]: Refusing update from remote peer 6810a866decb4f5089f7d35c870e4204: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:27:47.538872 26588 consensus_queue.cc:1035] T 89f86d58fe414b579845214534d338d8 P 6810a866decb4f5089f7d35c870e4204 [LEADER]: Connected to new peer: Peer: permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:27:47.539192 27327 consensus_queue.cc:1035] T adf80b9d71314010a7a4cf052fd8a511 P 6810a866decb4f5089f7d35c870e4204 [LEADER]: Connected to new peer: Peer: permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:27:47.541945 26279 raft_consensus.cc:1261] T 549666d19e724a96801db183a07809d8 P a3e500d7077a48acb5edb33f7e178c81 [term 1 FOLLOWER]: Refusing update from remote peer 6810a866decb4f5089f7d35c870e4204: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:27:47.543100 26279 raft_consensus.cc:1261] T adf80b9d71314010a7a4cf052fd8a511 P a3e500d7077a48acb5edb33f7e178c81 [term 1 LEARNER]: Refusing update from remote peer 6810a866decb4f5089f7d35c870e4204: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:27:47.544494 26588 raft_consensus.cc:2932] T 906cfe7530754271bb1ee57c470109bd P 6810a866decb4f5089f7d35c870e4204 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, fb6004b8640a4906aa0cfe63e383d40e (127.15.255.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } attrs { replace: true } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: false } } }
I20240502 21:27:47.549811 27326 consensus_queue.cc:1035] T 549666d19e724a96801db183a07809d8 P 6810a866decb4f5089f7d35c870e4204 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:27:47.549747 26012 raft_consensus.cc:2932] T 906cfe7530754271bb1ee57c470109bd P fb6004b8640a4906aa0cfe63e383d40e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, fb6004b8640a4906aa0cfe63e383d40e (127.15.255.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } attrs { replace: true } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: false } } }
I20240502 21:27:47.550724 27326 consensus_queue.cc:1035] T adf80b9d71314010a7a4cf052fd8a511 P 6810a866decb4f5089f7d35c870e4204 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:27:47.551858 26279 raft_consensus.cc:2932] T 549666d19e724a96801db183a07809d8 P a3e500d7077a48acb5edb33f7e178c81 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a3e500d7077a48acb5edb33f7e178c81 (127.15.255.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } attrs { replace: true } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } } }
I20240502 21:27:47.560036 23766 catalog_manager.cc:5484] T 906cfe7530754271bb1ee57c470109bd P fb6004b8640a4906aa0cfe63e383d40e reported cstate change: config changed from index 4 to 5, fb6004b8640a4906aa0cfe63e383d40e (127.15.255.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "6810a866decb4f5089f7d35c870e4204" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } attrs { replace: true } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: false } } }
I20240502 21:27:47.562804 26147 raft_consensus.cc:1203] T a274546743354e79bffa26ae7000e895 P e117afbf8819424e8a98476453dfde2a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 21:27:47.567395 24048 raft_consensus.cc:2932] T 549666d19e724a96801db183a07809d8 P 6c194a6db6c145acafc0f3068b8ce1fd [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a3e500d7077a48acb5edb33f7e178c81 (127.15.255.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } attrs { replace: true } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } } }
I20240502 21:27:47.568081 26278 raft_consensus.cc:2932] T adf80b9d71314010a7a4cf052fd8a511 P a3e500d7077a48acb5edb33f7e178c81 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a3e500d7077a48acb5edb33f7e178c81 (127.15.255.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } attrs { replace: true } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } } }
I20240502 21:27:47.567679 27328 raft_consensus.cc:2932] T adf80b9d71314010a7a4cf052fd8a511 P 6810a866decb4f5089f7d35c870e4204 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a3e500d7077a48acb5edb33f7e178c81 (127.15.255.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } attrs { replace: true } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } } }
I20240502 21:27:47.573283 23768 catalog_manager.cc:5484] T adf80b9d71314010a7a4cf052fd8a511 P a3e500d7077a48acb5edb33f7e178c81 reported cstate change: config changed from index 4 to 5, a3e500d7077a48acb5edb33f7e178c81 (127.15.255.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "6810a866decb4f5089f7d35c870e4204" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } attrs { replace: true } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } } }
I20240502 21:27:47.578478 27328 raft_consensus.cc:2932] T 549666d19e724a96801db183a07809d8 P 6810a866decb4f5089f7d35c870e4204 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 009d3e7e8a8b4e2497de3a73d01ced49 (127.15.255.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } } }
I20240502 21:27:47.580083 24317 raft_consensus.cc:2932] T adf80b9d71314010a7a4cf052fd8a511 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a3e500d7077a48acb5edb33f7e178c81 (127.15.255.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } attrs { replace: true } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } } }
I20240502 21:27:47.580613 26279 raft_consensus.cc:2932] T 549666d19e724a96801db183a07809d8 P a3e500d7077a48acb5edb33f7e178c81 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 009d3e7e8a8b4e2497de3a73d01ced49 (127.15.255.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } } }
I20240502 21:27:47.587122 26471 raft_consensus.cc:1050] T 8f14db150fda449e962be2466f1d2a0e P 6810a866decb4f5089f7d35c870e4204: attempting to promote NON_VOTER ee27c3f9e0ec46caa742ac386d6b7c20 to VOTER
I20240502 21:27:47.589066 26588 raft_consensus.cc:2932] T 89f86d58fe414b579845214534d338d8 P 6810a866decb4f5089f7d35c870e4204 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, fb6004b8640a4906aa0cfe63e383d40e (127.15.255.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } attrs { replace: true } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: false } } }
I20240502 21:27:47.589936 24049 raft_consensus.cc:1261] T 906cfe7530754271bb1ee57c470109bd P 6c194a6db6c145acafc0f3068b8ce1fd [term 1 FOLLOWER]: Refusing update from remote peer 6810a866decb4f5089f7d35c870e4204: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:27:47.591128 24047 raft_consensus.cc:1261] T adf80b9d71314010a7a4cf052fd8a511 P 6c194a6db6c145acafc0f3068b8ce1fd [term 1 FOLLOWER]: Refusing update from remote peer 6810a866decb4f5089f7d35c870e4204: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:27:47.591867 26471 consensus_queue.cc:237] T 8f14db150fda449e962be2466f1d2a0e P 6810a866decb4f5089f7d35c870e4204 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } attrs { replace: true } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: false } }
I20240502 21:27:47.592226 27326 consensus_queue.cc:1035] T 906cfe7530754271bb1ee57c470109bd P 6810a866decb4f5089f7d35c870e4204 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:27:47.593040 27326 consensus_queue.cc:1035] T adf80b9d71314010a7a4cf052fd8a511 P 6810a866decb4f5089f7d35c870e4204 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:27:47.593888 23768 catalog_manager.cc:5484] T 549666d19e724a96801db183a07809d8 P a3e500d7077a48acb5edb33f7e178c81 reported cstate change: config changed from index 5 to 6, VOTER 009d3e7e8a8b4e2497de3a73d01ced49 (127.15.255.131) evicted. New cstate: current_term: 1 leader_uuid: "6810a866decb4f5089f7d35c870e4204" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } } }
I20240502 21:27:47.593916 24317 raft_consensus.cc:2932] T 906cfe7530754271bb1ee57c470109bd P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, fb6004b8640a4906aa0cfe63e383d40e (127.15.255.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } attrs { replace: true } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: false } } }
I20240502 21:27:47.595424 24315 raft_consensus.cc:2932] T 89f86d58fe414b579845214534d338d8 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, fb6004b8640a4906aa0cfe63e383d40e (127.15.255.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } attrs { replace: true } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: false } } }
I20240502 21:27:47.597182 23914 consensus_queue.cc:237] T 906cfe7530754271bb1ee57c470109bd P 6810a866decb4f5089f7d35c870e4204 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: false } }
I20240502 21:27:47.598750 23757 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 549666d19e724a96801db183a07809d8 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:27:47.599712 24162 tablet_service.cc:1514] Processing DeleteTablet for tablet 549666d19e724a96801db183a07809d8 with delete_type TABLET_DATA_TOMBSTONED (TS 009d3e7e8a8b4e2497de3a73d01ced49 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:33102
I20240502 21:27:47.600087 27296 tablet_replica.cc:330] T 549666d19e724a96801db183a07809d8 P 009d3e7e8a8b4e2497de3a73d01ced49: stopping tablet replica
I20240502 21:27:47.600299 27296 raft_consensus.cc:2229] T 549666d19e724a96801db183a07809d8 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:47.600574 27296 raft_consensus.cc:2258] T 549666d19e724a96801db183a07809d8 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:47.599555 26012 raft_consensus.cc:1261] T 906cfe7530754271bb1ee57c470109bd P fb6004b8640a4906aa0cfe63e383d40e [term 1 FOLLOWER]: Refusing update from remote peer 6810a866decb4f5089f7d35c870e4204: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:27:47.602631 26420 raft_consensus.cc:1050] T 11cb9a2fbda946fb9f303f2c465bccb6 P 038c5d9d1d7a49c981c4c34fbc00d9e6: attempting to promote NON_VOTER e117afbf8819424e8a98476453dfde2a to VOTER
I20240502 21:27:47.603082 27326 consensus_queue.cc:1035] T 906cfe7530754271bb1ee57c470109bd P 6810a866decb4f5089f7d35c870e4204 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:27:47.603209 27296 ts_tablet_manager.cc:1875] T 549666d19e724a96801db183a07809d8 P 009d3e7e8a8b4e2497de3a73d01ced49: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:27:47.604036 26420 consensus_queue.cc:237] T 11cb9a2fbda946fb9f303f2c465bccb6 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } attrs { replace: true } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } }
I20240502 21:27:47.605796 24317 raft_consensus.cc:1261] T 906cfe7530754271bb1ee57c470109bd P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 1 FOLLOWER]: Refusing update from remote peer 6810a866decb4f5089f7d35c870e4204: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:27:47.606904 27326 consensus_queue.cc:1035] T 906cfe7530754271bb1ee57c470109bd P 6810a866decb4f5089f7d35c870e4204 [LEADER]: Connected to new peer: Peer: permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:27:47.610321 24317 raft_consensus.cc:2932] T 906cfe7530754271bb1ee57c470109bd P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 6c194a6db6c145acafc0f3068b8ce1fd (127.15.255.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: false } } }
I20240502 21:27:47.610337 27326 raft_consensus.cc:2932] T 906cfe7530754271bb1ee57c470109bd P 6810a866decb4f5089f7d35c870e4204 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 6c194a6db6c145acafc0f3068b8ce1fd (127.15.255.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: false } } }
I20240502 21:27:47.611353 27296 ts_tablet_manager.cc:1888] T 549666d19e724a96801db183a07809d8 P 009d3e7e8a8b4e2497de3a73d01ced49: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 21:27:47.611871 26012 raft_consensus.cc:2932] T 89f86d58fe414b579845214534d338d8 P fb6004b8640a4906aa0cfe63e383d40e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, fb6004b8640a4906aa0cfe63e383d40e (127.15.255.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } attrs { replace: true } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: false } } }
I20240502 21:27:47.616226 27296 log.cc:1196] T 549666d19e724a96801db183a07809d8 P 009d3e7e8a8b4e2497de3a73d01ced49: Deleting WAL directory at /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-2/wal/wals/549666d19e724a96801db183a07809d8
I20240502 21:27:47.617187 23756 catalog_manager.cc:4830] TS 009d3e7e8a8b4e2497de3a73d01ced49 (127.15.255.131:37895): tablet 549666d19e724a96801db183a07809d8 (table rebalance_test_table_3 [id=933ce90acb9f4b91ab6bd41e87a8fe99]) successfully deleted
I20240502 21:27:47.617468 23768 catalog_manager.cc:5484] T 89f86d58fe414b579845214534d338d8 P fb6004b8640a4906aa0cfe63e383d40e reported cstate change: config changed from index 4 to 5, fb6004b8640a4906aa0cfe63e383d40e (127.15.255.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "6810a866decb4f5089f7d35c870e4204" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } attrs { replace: true } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: false } } }
I20240502 21:27:47.619639 26012 raft_consensus.cc:2932] T 906cfe7530754271bb1ee57c470109bd P fb6004b8640a4906aa0cfe63e383d40e [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 6c194a6db6c145acafc0f3068b8ce1fd (127.15.255.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: false } } }
I20240502 21:27:47.622303 24046 raft_consensus.cc:2932] T 906cfe7530754271bb1ee57c470109bd P 6c194a6db6c145acafc0f3068b8ce1fd [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, fb6004b8640a4906aa0cfe63e383d40e (127.15.255.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } attrs { replace: true } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: false } } }
I20240502 21:27:47.624131 26147 raft_consensus.cc:1261] T 11cb9a2fbda946fb9f303f2c465bccb6 P e117afbf8819424e8a98476453dfde2a [term 1 LEARNER]: Refusing update from remote peer 038c5d9d1d7a49c981c4c34fbc00d9e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:27:47.624331 24047 raft_consensus.cc:1261] T 11cb9a2fbda946fb9f303f2c465bccb6 P 6c194a6db6c145acafc0f3068b8ce1fd [term 1 FOLLOWER]: Refusing update from remote peer 038c5d9d1d7a49c981c4c34fbc00d9e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:27:47.624976 26420 consensus_queue.cc:1035] T 11cb9a2fbda946fb9f303f2c465bccb6 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:27:47.626655 25879 raft_consensus.cc:1261] T 8f14db150fda449e962be2466f1d2a0e P ee27c3f9e0ec46caa742ac386d6b7c20 [term 1 LEARNER]: Refusing update from remote peer 6810a866decb4f5089f7d35c870e4204: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:27:47.626801 26560 consensus_queue.cc:1035] T 11cb9a2fbda946fb9f303f2c465bccb6 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:27:47.626998 24184 raft_consensus.cc:1261] T 8f14db150fda449e962be2466f1d2a0e P 009d3e7e8a8b4e2497de3a73d01ced49 [term 1 FOLLOWER]: Refusing update from remote peer 6810a866decb4f5089f7d35c870e4204: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:27:47.627632 26586 consensus_queue.cc:1035] T 8f14db150fda449e962be2466f1d2a0e P 6810a866decb4f5089f7d35c870e4204 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:27:47.628024 24315 raft_consensus.cc:1261] T 8f14db150fda449e962be2466f1d2a0e P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 1 FOLLOWER]: Refusing update from remote peer 6810a866decb4f5089f7d35c870e4204: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:27:47.628244 27327 consensus_queue.cc:1035] T 8f14db150fda449e962be2466f1d2a0e P 6810a866decb4f5089f7d35c870e4204 [LEADER]: Connected to new peer: Peer: permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:27:47.629261 24185 raft_consensus.cc:1261] T 11cb9a2fbda946fb9f303f2c465bccb6 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 1 FOLLOWER]: Refusing update from remote peer 038c5d9d1d7a49c981c4c34fbc00d9e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:27:47.629770 26560 consensus_queue.cc:1035] T 11cb9a2fbda946fb9f303f2c465bccb6 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:27:47.630344 23757 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 906cfe7530754271bb1ee57c470109bd with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:27:47.634107 23770 catalog_manager.cc:5484] T 906cfe7530754271bb1ee57c470109bd P 6810a866decb4f5089f7d35c870e4204 reported cstate change: config changed from index 5 to 6, VOTER 6c194a6db6c145acafc0f3068b8ce1fd (127.15.255.130) evicted. New cstate: current_term: 1 leader_uuid: "6810a866decb4f5089f7d35c870e4204" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:27:47.638641 24027 tablet_service.cc:1514] Processing DeleteTablet for tablet 906cfe7530754271bb1ee57c470109bd with delete_type TABLET_DATA_TOMBSTONED (TS 6c194a6db6c145acafc0f3068b8ce1fd not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:47386
I20240502 21:27:47.639664 27344 tablet_replica.cc:330] T 906cfe7530754271bb1ee57c470109bd P 6c194a6db6c145acafc0f3068b8ce1fd: stopping tablet replica
I20240502 21:27:47.640197 27344 raft_consensus.cc:2229] T 906cfe7530754271bb1ee57c470109bd P 6c194a6db6c145acafc0f3068b8ce1fd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:47.640658 27344 raft_consensus.cc:2258] T 906cfe7530754271bb1ee57c470109bd P 6c194a6db6c145acafc0f3068b8ce1fd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:47.639523 23916 consensus_queue.cc:237] T adf80b9d71314010a7a4cf052fd8a511 P 6810a866decb4f5089f7d35c870e4204 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } }
I20240502 21:27:47.642380 27344 ts_tablet_manager.cc:1875] T 906cfe7530754271bb1ee57c470109bd P 6c194a6db6c145acafc0f3068b8ce1fd: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:27:47.643873 26471 consensus_queue.cc:1035] T 8f14db150fda449e962be2466f1d2a0e P 6810a866decb4f5089f7d35c870e4204 [LEADER]: Connected to new peer: Peer: permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:27:47.644272 26279 raft_consensus.cc:1261] T adf80b9d71314010a7a4cf052fd8a511 P a3e500d7077a48acb5edb33f7e178c81 [term 1 FOLLOWER]: Refusing update from remote peer 6810a866decb4f5089f7d35c870e4204: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:27:47.644884 26471 consensus_queue.cc:1035] T adf80b9d71314010a7a4cf052fd8a511 P 6810a866decb4f5089f7d35c870e4204 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:27:47.646467 24049 raft_consensus.cc:1261] T 549666d19e724a96801db183a07809d8 P 6c194a6db6c145acafc0f3068b8ce1fd [term 1 FOLLOWER]: Refusing update from remote peer 6810a866decb4f5089f7d35c870e4204: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:27:47.647560 27326 consensus_queue.cc:1035] T 549666d19e724a96801db183a07809d8 P 6810a866decb4f5089f7d35c870e4204 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:27:47.647977 24183 raft_consensus.cc:2932] T 89f86d58fe414b579845214534d338d8 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, fb6004b8640a4906aa0cfe63e383d40e (127.15.255.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } attrs { replace: true } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: false } } }
I20240502 21:27:47.650313 24317 raft_consensus.cc:1261] T adf80b9d71314010a7a4cf052fd8a511 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 1 FOLLOWER]: Refusing update from remote peer 6810a866decb4f5089f7d35c870e4204: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:27:47.650830 27326 consensus_queue.cc:1035] T adf80b9d71314010a7a4cf052fd8a511 P 6810a866decb4f5089f7d35c870e4204 [LEADER]: Connected to new peer: Peer: permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:27:47.654012 27319 raft_consensus.cc:2932] T adf80b9d71314010a7a4cf052fd8a511 P 6810a866decb4f5089f7d35c870e4204 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 6c194a6db6c145acafc0f3068b8ce1fd (127.15.255.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } } }
I20240502 21:27:47.654143 26279 raft_consensus.cc:2932] T adf80b9d71314010a7a4cf052fd8a511 P a3e500d7077a48acb5edb33f7e178c81 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 6c194a6db6c145acafc0f3068b8ce1fd (127.15.255.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } } }
I20240502 21:27:47.658721 23757 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet adf80b9d71314010a7a4cf052fd8a511 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:27:47.659790 24317 raft_consensus.cc:2932] T adf80b9d71314010a7a4cf052fd8a511 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 6c194a6db6c145acafc0f3068b8ce1fd (127.15.255.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } } }
I20240502 21:27:47.660909 23770 catalog_manager.cc:5484] T adf80b9d71314010a7a4cf052fd8a511 P 6810a866decb4f5089f7d35c870e4204 reported cstate change: config changed from index 5 to 6, VOTER 6c194a6db6c145acafc0f3068b8ce1fd (127.15.255.130) evicted. New cstate: current_term: 1 leader_uuid: "6810a866decb4f5089f7d35c870e4204" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:27:47.665428 24027 tablet_service.cc:1514] Processing DeleteTablet for tablet adf80b9d71314010a7a4cf052fd8a511 with delete_type TABLET_DATA_TOMBSTONED (TS 6c194a6db6c145acafc0f3068b8ce1fd not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:47386
I20240502 21:27:47.669196 27344 ts_tablet_manager.cc:1888] T 906cfe7530754271bb1ee57c470109bd P 6c194a6db6c145acafc0f3068b8ce1fd: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 21:27:47.669407 27344 log.cc:1196] T 906cfe7530754271bb1ee57c470109bd P 6c194a6db6c145acafc0f3068b8ce1fd: Deleting WAL directory at /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-1/wal/wals/906cfe7530754271bb1ee57c470109bd
I20240502 21:27:47.668190 24046 raft_consensus.cc:2932] T 549666d19e724a96801db183a07809d8 P 6c194a6db6c145acafc0f3068b8ce1fd [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 009d3e7e8a8b4e2497de3a73d01ced49 (127.15.255.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } } }
I20240502 21:27:47.672935 23757 catalog_manager.cc:4830] TS 6c194a6db6c145acafc0f3068b8ce1fd (127.15.255.130:41235): tablet 906cfe7530754271bb1ee57c470109bd (table rebalance_test_table_2 [id=85ca987b38d84a7db5119dfc88b4feaf]) successfully deleted
I20240502 21:27:47.683808 24045 raft_consensus.cc:2932] T adf80b9d71314010a7a4cf052fd8a511 P 6c194a6db6c145acafc0f3068b8ce1fd [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a3e500d7077a48acb5edb33f7e178c81 (127.15.255.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } attrs { replace: true } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } } }
I20240502 21:27:47.686771 25878 raft_consensus.cc:2932] T 8f14db150fda449e962be2466f1d2a0e P ee27c3f9e0ec46caa742ac386d6b7c20 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ee27c3f9e0ec46caa742ac386d6b7c20 (127.15.255.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } attrs { replace: true } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: false } } }
I20240502 21:27:47.687852 27344 tablet_replica.cc:330] T adf80b9d71314010a7a4cf052fd8a511 P 6c194a6db6c145acafc0f3068b8ce1fd: stopping tablet replica
I20240502 21:27:47.688050 27344 raft_consensus.cc:2229] T adf80b9d71314010a7a4cf052fd8a511 P 6c194a6db6c145acafc0f3068b8ce1fd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:47.688249 27344 raft_consensus.cc:2258] T adf80b9d71314010a7a4cf052fd8a511 P 6c194a6db6c145acafc0f3068b8ce1fd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:47.691061 27313 raft_consensus.cc:2932] T 8f14db150fda449e962be2466f1d2a0e P 6810a866decb4f5089f7d35c870e4204 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ee27c3f9e0ec46caa742ac386d6b7c20 (127.15.255.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } attrs { replace: true } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: false } } }
I20240502 21:27:47.698055 23770 catalog_manager.cc:5484] T 8f14db150fda449e962be2466f1d2a0e P 6810a866decb4f5089f7d35c870e4204 reported cstate change: config changed from index 4 to 5, ee27c3f9e0ec46caa742ac386d6b7c20 (127.15.255.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "6810a866decb4f5089f7d35c870e4204" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:27:47.698194 24317 raft_consensus.cc:2932] T 8f14db150fda449e962be2466f1d2a0e P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ee27c3f9e0ec46caa742ac386d6b7c20 (127.15.255.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } attrs { replace: true } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: false } } }
I20240502 21:27:47.703951 27344 ts_tablet_manager.cc:1875] T adf80b9d71314010a7a4cf052fd8a511 P 6c194a6db6c145acafc0f3068b8ce1fd: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:27:47.705093 26560 raft_consensus.cc:2932] T 11cb9a2fbda946fb9f303f2c465bccb6 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e117afbf8819424e8a98476453dfde2a (127.15.255.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } attrs { replace: true } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } } }
I20240502 21:27:47.704668 24184 raft_consensus.cc:2932] T 8f14db150fda449e962be2466f1d2a0e P 009d3e7e8a8b4e2497de3a73d01ced49 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ee27c3f9e0ec46caa742ac386d6b7c20 (127.15.255.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } attrs { replace: true } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: false } } }
I20240502 21:27:47.710245 27344 ts_tablet_manager.cc:1888] T adf80b9d71314010a7a4cf052fd8a511 P 6c194a6db6c145acafc0f3068b8ce1fd: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 21:27:47.710433 27344 log.cc:1196] T adf80b9d71314010a7a4cf052fd8a511 P 6c194a6db6c145acafc0f3068b8ce1fd: Deleting WAL directory at /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-1/wal/wals/adf80b9d71314010a7a4cf052fd8a511
I20240502 21:27:47.711212 23757 catalog_manager.cc:4830] TS 6c194a6db6c145acafc0f3068b8ce1fd (127.15.255.130:41235): tablet adf80b9d71314010a7a4cf052fd8a511 (table rebalance_test_table_0 [id=1e673d0be4694336bbf65d9c114c614f]) successfully deleted
I20240502 21:27:47.713742 24047 raft_consensus.cc:2932] T 11cb9a2fbda946fb9f303f2c465bccb6 P 6c194a6db6c145acafc0f3068b8ce1fd [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e117afbf8819424e8a98476453dfde2a (127.15.255.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } attrs { replace: true } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } } }
I20240502 21:27:47.717200 23916 consensus_queue.cc:237] T 8f14db150fda449e962be2466f1d2a0e P 6810a866decb4f5089f7d35c870e4204 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: false } }
I20240502 21:27:47.718063 26147 raft_consensus.cc:2932] T 11cb9a2fbda946fb9f303f2c465bccb6 P e117afbf8819424e8a98476453dfde2a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e117afbf8819424e8a98476453dfde2a (127.15.255.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } attrs { replace: true } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } } }
I20240502 21:27:47.721467 25878 raft_consensus.cc:1261] T 8f14db150fda449e962be2466f1d2a0e P ee27c3f9e0ec46caa742ac386d6b7c20 [term 1 FOLLOWER]: Refusing update from remote peer 6810a866decb4f5089f7d35c870e4204: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:27:47.722824 24183 raft_consensus.cc:1261] T 8f14db150fda449e962be2466f1d2a0e P 009d3e7e8a8b4e2497de3a73d01ced49 [term 1 FOLLOWER]: Refusing update from remote peer 6810a866decb4f5089f7d35c870e4204: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:27:47.722584 23770 catalog_manager.cc:5484] T 11cb9a2fbda946fb9f303f2c465bccb6 P 038c5d9d1d7a49c981c4c34fbc00d9e6 reported cstate change: config changed from index 4 to 5, e117afbf8819424e8a98476453dfde2a (127.15.255.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:27:47.723611 26879 tablet_bootstrap.cc:492] T f84bbe304597450c978ae28e0c4e7a3e P a3e500d7077a48acb5edb33f7e178c81: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:27:47.724017 27328 consensus_queue.cc:1035] T 8f14db150fda449e962be2466f1d2a0e P 6810a866decb4f5089f7d35c870e4204 [LEADER]: Connected to new peer: Peer: permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:27:47.724619 27327 consensus_queue.cc:1035] T 8f14db150fda449e962be2466f1d2a0e P 6810a866decb4f5089f7d35c870e4204 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:27:47.725111 25879 raft_consensus.cc:1203] T 16bb47f03a2f46fdb7344e018f07fdcb P ee27c3f9e0ec46caa742ac386d6b7c20 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 21:27:47.725950 26879 tablet_bootstrap.cc:492] T f84bbe304597450c978ae28e0c4e7a3e P a3e500d7077a48acb5edb33f7e178c81: Bootstrap complete.
I20240502 21:27:47.726255 26879 ts_tablet_manager.cc:1367] T f84bbe304597450c978ae28e0c4e7a3e P a3e500d7077a48acb5edb33f7e178c81: Time spent bootstrapping tablet: real 0.354s user 0.045s sys 0.004s
I20240502 21:27:47.727078 26879 raft_consensus.cc:348] T f84bbe304597450c978ae28e0c4e7a3e P a3e500d7077a48acb5edb33f7e178c81 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } attrs { replace: true } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: true } }
I20240502 21:27:47.727329 26879 raft_consensus.cc:729] T f84bbe304597450c978ae28e0c4e7a3e P a3e500d7077a48acb5edb33f7e178c81 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a3e500d7077a48acb5edb33f7e178c81, State: Initialized, Role: LEARNER
I20240502 21:27:47.727541 26879 consensus_queue.cc:260] T f84bbe304597450c978ae28e0c4e7a3e P a3e500d7077a48acb5edb33f7e178c81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } attrs { replace: true } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: true } }
I20240502 21:27:47.729347 26879 ts_tablet_manager.cc:1398] T f84bbe304597450c978ae28e0c4e7a3e P a3e500d7077a48acb5edb33f7e178c81: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 21:27:47.730062 24205 tablet_copy_service.cc:342] P 009d3e7e8a8b4e2497de3a73d01ced49: Request end of tablet copy session a3e500d7077a48acb5edb33f7e178c81-f84bbe304597450c978ae28e0c4e7a3e received from {username='slave'} at 127.15.255.136:51951
I20240502 21:27:47.730183 24205 tablet_copy_service.cc:433] P 009d3e7e8a8b4e2497de3a73d01ced49: ending tablet copy session a3e500d7077a48acb5edb33f7e178c81-f84bbe304597450c978ae28e0c4e7a3e on tablet f84bbe304597450c978ae28e0c4e7a3e with peer a3e500d7077a48acb5edb33f7e178c81
I20240502 21:27:47.730046 27234 raft_consensus.cc:2932] T 8f14db150fda449e962be2466f1d2a0e P 6810a866decb4f5089f7d35c870e4204 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 038c5d9d1d7a49c981c4c34fbc00d9e6 (127.15.255.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: false } } }
I20240502 21:27:47.730684 24183 raft_consensus.cc:2932] T 8f14db150fda449e962be2466f1d2a0e P 009d3e7e8a8b4e2497de3a73d01ced49 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 038c5d9d1d7a49c981c4c34fbc00d9e6 (127.15.255.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: false } } }
I20240502 21:27:47.733162 24184 raft_consensus.cc:2932] T 11cb9a2fbda946fb9f303f2c465bccb6 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e117afbf8819424e8a98476453dfde2a (127.15.255.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } attrs { replace: true } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } } }
I20240502 21:27:47.736877 25878 raft_consensus.cc:2932] T 8f14db150fda449e962be2466f1d2a0e P ee27c3f9e0ec46caa742ac386d6b7c20 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 038c5d9d1d7a49c981c4c34fbc00d9e6 (127.15.255.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: false } } }
I20240502 21:27:47.737531 24317 consensus_queue.cc:237] T 11cb9a2fbda946fb9f303f2c465bccb6 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } }
I20240502 21:27:47.740820 26147 raft_consensus.cc:1261] T 11cb9a2fbda946fb9f303f2c465bccb6 P e117afbf8819424e8a98476453dfde2a [term 1 FOLLOWER]: Refusing update from remote peer 038c5d9d1d7a49c981c4c34fbc00d9e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:27:47.741477 26560 consensus_queue.cc:1035] T 11cb9a2fbda946fb9f303f2c465bccb6 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:27:47.741945 23768 catalog_manager.cc:5484] T 8f14db150fda449e962be2466f1d2a0e P ee27c3f9e0ec46caa742ac386d6b7c20 reported cstate change: config changed from index 5 to 6, VOTER 038c5d9d1d7a49c981c4c34fbc00d9e6 (127.15.255.132) evicted. New cstate: current_term: 1 leader_uuid: "6810a866decb4f5089f7d35c870e4204" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: false } } }
I20240502 21:27:47.744613 23757 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 8f14db150fda449e962be2466f1d2a0e with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:27:47.746035 26420 raft_consensus.cc:2932] T 11cb9a2fbda946fb9f303f2c465bccb6 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 6c194a6db6c145acafc0f3068b8ce1fd (127.15.255.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } } }
I20240502 21:27:47.746863 24185 raft_consensus.cc:1261] T 11cb9a2fbda946fb9f303f2c465bccb6 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 1 FOLLOWER]: Refusing update from remote peer 038c5d9d1d7a49c981c4c34fbc00d9e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:27:47.747110 24297 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f14db150fda449e962be2466f1d2a0e with delete_type TABLET_DATA_TOMBSTONED (TS 038c5d9d1d7a49c981c4c34fbc00d9e6 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:58268
I20240502 21:27:47.747507 27312 tablet_replica.cc:330] T 8f14db150fda449e962be2466f1d2a0e P 038c5d9d1d7a49c981c4c34fbc00d9e6: stopping tablet replica
I20240502 21:27:47.747749 27312 raft_consensus.cc:2229] T 8f14db150fda449e962be2466f1d2a0e P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:47.747747 27349 consensus_queue.cc:1035] T 11cb9a2fbda946fb9f303f2c465bccb6 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:27:47.748101 27312 raft_consensus.cc:2258] T 8f14db150fda449e962be2466f1d2a0e P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:47.748322 27182 heartbeater.cc:503] Master 127.15.255.190:33301 was elected leader, sending a full tablet report...
I20240502 21:27:47.749135 27312 ts_tablet_manager.cc:1875] T 8f14db150fda449e962be2466f1d2a0e P 038c5d9d1d7a49c981c4c34fbc00d9e6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:27:47.750118 24184 raft_consensus.cc:2932] T 11cb9a2fbda946fb9f303f2c465bccb6 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 6c194a6db6c145acafc0f3068b8ce1fd (127.15.255.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } } }
I20240502 21:27:47.752611 23755 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 11cb9a2fbda946fb9f303f2c465bccb6 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:27:47.752540 26147 raft_consensus.cc:2932] T 11cb9a2fbda946fb9f303f2c465bccb6 P e117afbf8819424e8a98476453dfde2a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 6c194a6db6c145acafc0f3068b8ce1fd (127.15.255.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } } }
I20240502 21:27:47.754139 23772 catalog_manager.cc:5484] T 11cb9a2fbda946fb9f303f2c465bccb6 P 038c5d9d1d7a49c981c4c34fbc00d9e6 reported cstate change: config changed from index 5 to 6, VOTER 6c194a6db6c145acafc0f3068b8ce1fd (127.15.255.130) evicted. New cstate: current_term: 1 leader_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:47.757916 27312 ts_tablet_manager.cc:1888] T 8f14db150fda449e962be2466f1d2a0e P 038c5d9d1d7a49c981c4c34fbc00d9e6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 21:27:47.758117 27312 log.cc:1196] T 8f14db150fda449e962be2466f1d2a0e P 038c5d9d1d7a49c981c4c34fbc00d9e6: Deleting WAL directory at /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-3/wal/wals/8f14db150fda449e962be2466f1d2a0e
I20240502 21:27:47.758867 24027 tablet_service.cc:1514] Processing DeleteTablet for tablet 11cb9a2fbda946fb9f303f2c465bccb6 with delete_type TABLET_DATA_TOMBSTONED (TS 6c194a6db6c145acafc0f3068b8ce1fd not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:47386
I20240502 21:27:47.759301 27344 tablet_replica.cc:330] T 11cb9a2fbda946fb9f303f2c465bccb6 P 6c194a6db6c145acafc0f3068b8ce1fd: stopping tablet replica
I20240502 21:27:47.759536 27344 raft_consensus.cc:2229] T 11cb9a2fbda946fb9f303f2c465bccb6 P 6c194a6db6c145acafc0f3068b8ce1fd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:47.759876 27344 raft_consensus.cc:2258] T 11cb9a2fbda946fb9f303f2c465bccb6 P 6c194a6db6c145acafc0f3068b8ce1fd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:47.760026 23755 catalog_manager.cc:4830] TS 038c5d9d1d7a49c981c4c34fbc00d9e6 (127.15.255.132:46813): tablet 8f14db150fda449e962be2466f1d2a0e (table rebalance_test_table_4 [id=9d2b330e886e4e83a437506bf99d77a3]) successfully deleted
I20240502 21:27:47.760843 27344 ts_tablet_manager.cc:1875] T 11cb9a2fbda946fb9f303f2c465bccb6 P 6c194a6db6c145acafc0f3068b8ce1fd: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:27:47.767900 27344 ts_tablet_manager.cc:1888] T 11cb9a2fbda946fb9f303f2c465bccb6 P 6c194a6db6c145acafc0f3068b8ce1fd: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 21:27:47.768086 27344 log.cc:1196] T 11cb9a2fbda946fb9f303f2c465bccb6 P 6c194a6db6c145acafc0f3068b8ce1fd: Deleting WAL directory at /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-1/wal/wals/11cb9a2fbda946fb9f303f2c465bccb6
I20240502 21:27:47.769125 23757 catalog_manager.cc:4830] TS 6c194a6db6c145acafc0f3068b8ce1fd (127.15.255.130:41235): tablet 11cb9a2fbda946fb9f303f2c465bccb6 (table rebalance_test_table_5 [id=e10e5576c0ea45a09c0eab75c2023b78]) successfully deleted
I20240502 21:27:47.780011 26279 raft_consensus.cc:1203] T d5697d8b06094211b261f144de9d46f8 P a3e500d7077a48acb5edb33f7e178c81 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 21:27:47.788213 26944 tablet_bootstrap.cc:492] T 83b95f3ae7564b81b31a2f5673ee22bb P e117afbf8819424e8a98476453dfde2a: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:27:47.788743 26944 tablet_bootstrap.cc:492] T 83b95f3ae7564b81b31a2f5673ee22bb P e117afbf8819424e8a98476453dfde2a: Bootstrap complete.
I20240502 21:27:47.789039 26944 ts_tablet_manager.cc:1367] T 83b95f3ae7564b81b31a2f5673ee22bb P e117afbf8819424e8a98476453dfde2a: Time spent bootstrapping tablet: real 0.258s user 0.052s sys 0.009s
I20240502 21:27:47.789955 26944 raft_consensus.cc:348] T 83b95f3ae7564b81b31a2f5673ee22bb P e117afbf8819424e8a98476453dfde2a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } attrs { replace: true } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: NON_VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: true } }
I20240502 21:27:47.790267 26944 raft_consensus.cc:729] T 83b95f3ae7564b81b31a2f5673ee22bb P e117afbf8819424e8a98476453dfde2a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e117afbf8819424e8a98476453dfde2a, State: Initialized, Role: LEARNER
I20240502 21:27:47.790508 26944 consensus_queue.cc:260] T 83b95f3ae7564b81b31a2f5673ee22bb P e117afbf8819424e8a98476453dfde2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } attrs { replace: true } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: NON_VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: true } }
I20240502 21:27:47.791251 26944 ts_tablet_manager.cc:1398] T 83b95f3ae7564b81b31a2f5673ee22bb P e117afbf8819424e8a98476453dfde2a: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20240502 21:27:47.792189 23935 tablet_copy_service.cc:342] P 6810a866decb4f5089f7d35c870e4204: Request end of tablet copy session e117afbf8819424e8a98476453dfde2a-83b95f3ae7564b81b31a2f5673ee22bb received from {username='slave'} at 127.15.255.135:39893
I20240502 21:27:47.793282 26849 tablet_bootstrap.cc:492] T fcc9cbeae6f343af9f84c73349f3dadb P fb6004b8640a4906aa0cfe63e383d40e: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:27:47.793701 26849 tablet_bootstrap.cc:492] T fcc9cbeae6f343af9f84c73349f3dadb P fb6004b8640a4906aa0cfe63e383d40e: Bootstrap complete.
I20240502 21:27:47.793988 26849 ts_tablet_manager.cc:1367] T fcc9cbeae6f343af9f84c73349f3dadb P fb6004b8640a4906aa0cfe63e383d40e: Time spent bootstrapping tablet: real 0.514s user 0.047s sys 0.004s
I20240502 21:27:47.794019 23935 tablet_copy_service.cc:433] P 6810a866decb4f5089f7d35c870e4204: ending tablet copy session e117afbf8819424e8a98476453dfde2a-83b95f3ae7564b81b31a2f5673ee22bb on tablet 83b95f3ae7564b81b31a2f5673ee22bb with peer e117afbf8819424e8a98476453dfde2a
I20240502 21:27:47.794843 26849 raft_consensus.cc:348] T fcc9cbeae6f343af9f84c73349f3dadb P fb6004b8640a4906aa0cfe63e383d40e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } attrs { replace: true } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: NON_VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: true } }
I20240502 21:27:47.795153 26849 raft_consensus.cc:729] T fcc9cbeae6f343af9f84c73349f3dadb P fb6004b8640a4906aa0cfe63e383d40e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: fb6004b8640a4906aa0cfe63e383d40e, State: Initialized, Role: LEARNER
I20240502 21:27:47.795380 26849 consensus_queue.cc:260] T fcc9cbeae6f343af9f84c73349f3dadb P fb6004b8640a4906aa0cfe63e383d40e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } attrs { replace: true } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: NON_VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: true } }
I20240502 21:27:47.796708 26849 ts_tablet_manager.cc:1398] T fcc9cbeae6f343af9f84c73349f3dadb P fb6004b8640a4906aa0cfe63e383d40e: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240502 21:27:47.797839 27292 raft_consensus.cc:1050] T a2a0537a60f849daae2061c2b0c4c89f P 009d3e7e8a8b4e2497de3a73d01ced49: attempting to promote NON_VOTER a3e500d7077a48acb5edb33f7e178c81 to VOTER
I20240502 21:27:47.799162 27292 consensus_queue.cc:237] T a2a0537a60f849daae2061c2b0c4c89f P 009d3e7e8a8b4e2497de3a73d01ced49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } attrs { replace: true } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } }
I20240502 21:27:47.801915 23916 raft_consensus.cc:1261] T a2a0537a60f849daae2061c2b0c4c89f P 6810a866decb4f5089f7d35c870e4204 [term 1 FOLLOWER]: Refusing update from remote peer 009d3e7e8a8b4e2497de3a73d01ced49: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:27:47.802006 24045 raft_consensus.cc:1261] T a2a0537a60f849daae2061c2b0c4c89f P 6c194a6db6c145acafc0f3068b8ce1fd [term 1 FOLLOWER]: Refusing update from remote peer 009d3e7e8a8b4e2497de3a73d01ced49: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:27:47.802402 26280 raft_consensus.cc:1261] T a2a0537a60f849daae2061c2b0c4c89f P a3e500d7077a48acb5edb33f7e178c81 [term 1 LEARNER]: Refusing update from remote peer 009d3e7e8a8b4e2497de3a73d01ced49: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:27:47.802475 26279 raft_consensus.cc:1203] T f84bbe304597450c978ae28e0c4e7a3e P a3e500d7077a48acb5edb33f7e178c81 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 21:27:47.802879 24338 tablet_copy_service.cc:342] P 038c5d9d1d7a49c981c4c34fbc00d9e6: Request end of tablet copy session fb6004b8640a4906aa0cfe63e383d40e-fcc9cbeae6f343af9f84c73349f3dadb received from {username='slave'} at 127.15.255.134:51733
I20240502 21:27:47.803014 24338 tablet_copy_service.cc:433] P 038c5d9d1d7a49c981c4c34fbc00d9e6: ending tablet copy session fb6004b8640a4906aa0cfe63e383d40e-fcc9cbeae6f343af9f84c73349f3dadb on tablet fcc9cbeae6f343af9f84c73349f3dadb with peer fb6004b8640a4906aa0cfe63e383d40e
I20240502 21:27:47.803570 26554 consensus_queue.cc:1035] T a2a0537a60f849daae2061c2b0c4c89f P 009d3e7e8a8b4e2497de3a73d01ced49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:27:47.803869 27292 consensus_queue.cc:1035] T a2a0537a60f849daae2061c2b0c4c89f P 009d3e7e8a8b4e2497de3a73d01ced49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:27:47.804385 27292 consensus_queue.cc:1035] T a2a0537a60f849daae2061c2b0c4c89f P 009d3e7e8a8b4e2497de3a73d01ced49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:27:47.818485 27215 raft_consensus.cc:2932] T a2a0537a60f849daae2061c2b0c4c89f P 009d3e7e8a8b4e2497de3a73d01ced49 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a3e500d7077a48acb5edb33f7e178c81 (127.15.255.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } attrs { replace: true } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } } }
I20240502 21:27:47.819339 24045 raft_consensus.cc:2932] T a2a0537a60f849daae2061c2b0c4c89f P 6c194a6db6c145acafc0f3068b8ce1fd [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a3e500d7077a48acb5edb33f7e178c81 (127.15.255.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } attrs { replace: true } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } } }
I20240502 21:27:47.823038 26279 raft_consensus.cc:2932] T a2a0537a60f849daae2061c2b0c4c89f P a3e500d7077a48acb5edb33f7e178c81 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a3e500d7077a48acb5edb33f7e178c81 (127.15.255.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } attrs { replace: true } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } } }
I20240502 21:27:47.824079 23914 raft_consensus.cc:2932] T a2a0537a60f849daae2061c2b0c4c89f P 6810a866decb4f5089f7d35c870e4204 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a3e500d7077a48acb5edb33f7e178c81 (127.15.255.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } attrs { replace: true } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } } }
I20240502 21:27:47.827939 23766 catalog_manager.cc:5484] T a2a0537a60f849daae2061c2b0c4c89f P 009d3e7e8a8b4e2497de3a73d01ced49 reported cstate change: config changed from index 4 to 5, a3e500d7077a48acb5edb33f7e178c81 (127.15.255.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:27:47.837095 24184 consensus_queue.cc:237] T a2a0537a60f849daae2061c2b0c4c89f P 009d3e7e8a8b4e2497de3a73d01ced49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } }
I20240502 21:27:47.838969 26279 raft_consensus.cc:1261] T a2a0537a60f849daae2061c2b0c4c89f P a3e500d7077a48acb5edb33f7e178c81 [term 1 FOLLOWER]: Refusing update from remote peer 009d3e7e8a8b4e2497de3a73d01ced49: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:27:47.839434 26475 consensus_queue.cc:1035] T a2a0537a60f849daae2061c2b0c4c89f P 009d3e7e8a8b4e2497de3a73d01ced49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:27:47.841478 24045 raft_consensus.cc:1261] T a2a0537a60f849daae2061c2b0c4c89f P 6c194a6db6c145acafc0f3068b8ce1fd [term 1 FOLLOWER]: Refusing update from remote peer 009d3e7e8a8b4e2497de3a73d01ced49: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:27:47.841636 27215 raft_consensus.cc:2932] T a2a0537a60f849daae2061c2b0c4c89f P 009d3e7e8a8b4e2497de3a73d01ced49 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 6810a866decb4f5089f7d35c870e4204 (127.15.255.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } } }
I20240502 21:27:47.842216 27292 consensus_queue.cc:1035] T a2a0537a60f849daae2061c2b0c4c89f P 009d3e7e8a8b4e2497de3a73d01ced49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:27:47.842211 26280 raft_consensus.cc:2932] T a2a0537a60f849daae2061c2b0c4c89f P a3e500d7077a48acb5edb33f7e178c81 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 6810a866decb4f5089f7d35c870e4204 (127.15.255.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } } }
I20240502 21:27:47.843950 24045 raft_consensus.cc:2932] T a2a0537a60f849daae2061c2b0c4c89f P 6c194a6db6c145acafc0f3068b8ce1fd [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 6810a866decb4f5089f7d35c870e4204 (127.15.255.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } } }
I20240502 21:27:47.846179 23756 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet a2a0537a60f849daae2061c2b0c4c89f with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:27:47.847323 23770 catalog_manager.cc:5484] T a2a0537a60f849daae2061c2b0c4c89f P a3e500d7077a48acb5edb33f7e178c81 reported cstate change: config changed from index 5 to 6, VOTER 6810a866decb4f5089f7d35c870e4204 (127.15.255.129) evicted. New cstate: current_term: 1 leader_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } } }
I20240502 21:27:47.851426 23894 tablet_service.cc:1514] Processing DeleteTablet for tablet a2a0537a60f849daae2061c2b0c4c89f with delete_type TABLET_DATA_TOMBSTONED (TS 6810a866decb4f5089f7d35c870e4204 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:33512
I20240502 21:27:47.851795 27301 tablet_replica.cc:330] T a2a0537a60f849daae2061c2b0c4c89f P 6810a866decb4f5089f7d35c870e4204: stopping tablet replica
I20240502 21:27:47.851959 27301 raft_consensus.cc:2229] T a2a0537a60f849daae2061c2b0c4c89f P 6810a866decb4f5089f7d35c870e4204 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:47.852185 27301 raft_consensus.cc:2258] T a2a0537a60f849daae2061c2b0c4c89f P 6810a866decb4f5089f7d35c870e4204 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:47.853215 27301 ts_tablet_manager.cc:1875] T a2a0537a60f849daae2061c2b0c4c89f P 6810a866decb4f5089f7d35c870e4204: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:27:47.858244 26147 raft_consensus.cc:1203] T 83b95f3ae7564b81b31a2f5673ee22bb P e117afbf8819424e8a98476453dfde2a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 21:27:47.859720 27301 ts_tablet_manager.cc:1888] T a2a0537a60f849daae2061c2b0c4c89f P 6810a866decb4f5089f7d35c870e4204: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 21:27:47.859925 27301 log.cc:1196] T a2a0537a60f849daae2061c2b0c4c89f P 6810a866decb4f5089f7d35c870e4204: Deleting WAL directory at /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-0/wal/wals/a2a0537a60f849daae2061c2b0c4c89f
I20240502 21:27:47.860585 23757 catalog_manager.cc:4830] TS 6810a866decb4f5089f7d35c870e4204 (127.15.255.129:40435): tablet a2a0537a60f849daae2061c2b0c4c89f (table rebalance_test_table_6 [id=9aaf56e5a31f42a79b45624cc2c43779]) successfully deleted
I20240502 21:27:47.917783 24045 consensus_queue.cc:237] T 8af15fc41246471db31f7824985b5903 P 6c194a6db6c145acafc0f3068b8ce1fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } attrs { replace: true } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: NON_VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: true } }
I20240502 21:27:47.919863 24317 raft_consensus.cc:1261] T 8af15fc41246471db31f7824985b5903 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 1 FOLLOWER]: Refusing update from remote peer 6c194a6db6c145acafc0f3068b8ce1fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:27:47.920325 24184 raft_consensus.cc:1261] T 8af15fc41246471db31f7824985b5903 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 1 FOLLOWER]: Refusing update from remote peer 6c194a6db6c145acafc0f3068b8ce1fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:27:47.920527 27324 consensus_queue.cc:1035] T 8af15fc41246471db31f7824985b5903 P 6c194a6db6c145acafc0f3068b8ce1fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:27:47.921324 27324 consensus_queue.cc:1035] T 8af15fc41246471db31f7824985b5903 P 6c194a6db6c145acafc0f3068b8ce1fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:27:47.923691 27317 raft_consensus.cc:2932] T 8af15fc41246471db31f7824985b5903 P 6c194a6db6c145acafc0f3068b8ce1fd [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER fb6004b8640a4906aa0cfe63e383d40e (127.15.255.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } attrs { replace: true } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: NON_VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: true } } }
I20240502 21:27:47.928022 24317 raft_consensus.cc:2932] T 8af15fc41246471db31f7824985b5903 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER fb6004b8640a4906aa0cfe63e383d40e (127.15.255.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } attrs { replace: true } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: NON_VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: true } } }
W20240502 21:27:47.929522 23980 consensus_peers.cc:485] T 8af15fc41246471db31f7824985b5903 P 6c194a6db6c145acafc0f3068b8ce1fd -> Peer fb6004b8640a4906aa0cfe63e383d40e (127.15.255.134:32825): Couldn't send request to peer fb6004b8640a4906aa0cfe63e383d40e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8af15fc41246471db31f7824985b5903. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:47.933884 23772 catalog_manager.cc:5484] T 8af15fc41246471db31f7824985b5903 P 038c5d9d1d7a49c981c4c34fbc00d9e6 reported cstate change: config changed from index -1 to 4, NON_VOTER fb6004b8640a4906aa0cfe63e383d40e (127.15.255.134) added. New cstate: current_term: 1 leader_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } attrs { replace: true } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: NON_VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: true } } }
I20240502 21:27:47.938124 24184 raft_consensus.cc:2932] T 8af15fc41246471db31f7824985b5903 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER fb6004b8640a4906aa0cfe63e383d40e (127.15.255.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } attrs { replace: true } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: NON_VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: true } } }
I20240502 21:27:47.945178 24317 consensus_queue.cc:237] T fef55863beb6435dbb25bfb6d71e3cd7 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } attrs { replace: true } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: true } }
I20240502 21:27:47.947804 24184 raft_consensus.cc:1261] T fef55863beb6435dbb25bfb6d71e3cd7 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 1 FOLLOWER]: Refusing update from remote peer 038c5d9d1d7a49c981c4c34fbc00d9e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:27:47.948046 23914 raft_consensus.cc:1261] T fef55863beb6435dbb25bfb6d71e3cd7 P 6810a866decb4f5089f7d35c870e4204 [term 1 FOLLOWER]: Refusing update from remote peer 038c5d9d1d7a49c981c4c34fbc00d9e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:27:47.948402 26560 consensus_queue.cc:1035] T fef55863beb6435dbb25bfb6d71e3cd7 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 21:27:47.948777 24252 consensus_peers.cc:485] T fef55863beb6435dbb25bfb6d71e3cd7 P 038c5d9d1d7a49c981c4c34fbc00d9e6 -> Peer a3e500d7077a48acb5edb33f7e178c81 (127.15.255.136:43595): Couldn't send request to peer a3e500d7077a48acb5edb33f7e178c81. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fef55863beb6435dbb25bfb6d71e3cd7. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:47.948944 26560 consensus_queue.cc:1035] T fef55863beb6435dbb25bfb6d71e3cd7 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:27:47.951166 27349 raft_consensus.cc:2932] T fef55863beb6435dbb25bfb6d71e3cd7 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a3e500d7077a48acb5edb33f7e178c81 (127.15.255.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } attrs { replace: true } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: true } } }
I20240502 21:27:47.952168 23914 raft_consensus.cc:2932] T fef55863beb6435dbb25bfb6d71e3cd7 P 6810a866decb4f5089f7d35c870e4204 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a3e500d7077a48acb5edb33f7e178c81 (127.15.255.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } attrs { replace: true } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: true } } }
I20240502 21:27:47.955938 24184 raft_consensus.cc:2932] T fef55863beb6435dbb25bfb6d71e3cd7 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER a3e500d7077a48acb5edb33f7e178c81 (127.15.255.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } attrs { replace: true } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: true } } }
I20240502 21:27:47.956758 23768 catalog_manager.cc:5484] T fef55863beb6435dbb25bfb6d71e3cd7 P 038c5d9d1d7a49c981c4c34fbc00d9e6 reported cstate change: config changed from index -1 to 4, NON_VOTER a3e500d7077a48acb5edb33f7e178c81 (127.15.255.136) added. New cstate: current_term: 1 leader_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:47.969732 24317 consensus_queue.cc:237] T 52f621ea3f174490948eae854c706a92 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } attrs { replace: true } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: NON_VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: true } }
I20240502 21:27:47.973537 23914 raft_consensus.cc:1261] T 52f621ea3f174490948eae854c706a92 P 6810a866decb4f5089f7d35c870e4204 [term 1 FOLLOWER]: Refusing update from remote peer 038c5d9d1d7a49c981c4c34fbc00d9e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 21:27:47.973635 24250 consensus_peers.cc:485] T 52f621ea3f174490948eae854c706a92 P 038c5d9d1d7a49c981c4c34fbc00d9e6 -> Peer ee27c3f9e0ec46caa742ac386d6b7c20 (127.15.255.133:40841): Couldn't send request to peer ee27c3f9e0ec46caa742ac386d6b7c20. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 52f621ea3f174490948eae854c706a92. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:47.973843 24045 raft_consensus.cc:1261] T 52f621ea3f174490948eae854c706a92 P 6c194a6db6c145acafc0f3068b8ce1fd [term 1 FOLLOWER]: Refusing update from remote peer 038c5d9d1d7a49c981c4c34fbc00d9e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:27:47.974510 27348 consensus_queue.cc:1035] T 52f621ea3f174490948eae854c706a92 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:27:47.975036 26560 consensus_queue.cc:1035] T 52f621ea3f174490948eae854c706a92 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:27:47.977974 26560 raft_consensus.cc:2932] T 52f621ea3f174490948eae854c706a92 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ee27c3f9e0ec46caa742ac386d6b7c20 (127.15.255.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } attrs { replace: true } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: NON_VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: true } } }
I20240502 21:27:47.982496 24045 raft_consensus.cc:2932] T 52f621ea3f174490948eae854c706a92 P 6c194a6db6c145acafc0f3068b8ce1fd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ee27c3f9e0ec46caa742ac386d6b7c20 (127.15.255.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } attrs { replace: true } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: NON_VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: true } } }
I20240502 21:27:47.982707 23914 raft_consensus.cc:2932] T 52f621ea3f174490948eae854c706a92 P 6810a866decb4f5089f7d35c870e4204 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ee27c3f9e0ec46caa742ac386d6b7c20 (127.15.255.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } attrs { replace: true } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: NON_VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: true } } }
I20240502 21:27:47.985738 23772 catalog_manager.cc:5484] T 52f621ea3f174490948eae854c706a92 P 038c5d9d1d7a49c981c4c34fbc00d9e6 reported cstate change: config changed from index -1 to 4, NON_VOTER ee27c3f9e0ec46caa742ac386d6b7c20 (127.15.255.133) added. New cstate: current_term: 1 leader_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: NON_VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:47.992673 27349 raft_consensus.cc:1050] T fe044374ad7b4999afc72ab812466749 P 038c5d9d1d7a49c981c4c34fbc00d9e6: attempting to promote NON_VOTER ee27c3f9e0ec46caa742ac386d6b7c20 to VOTER
I20240502 21:27:47.993453 27349 consensus_queue.cc:237] T fe044374ad7b4999afc72ab812466749 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } attrs { replace: true } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: false } }
I20240502 21:27:47.997990 24184 raft_consensus.cc:1261] T fe044374ad7b4999afc72ab812466749 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 1 FOLLOWER]: Refusing update from remote peer 038c5d9d1d7a49c981c4c34fbc00d9e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:27:47.998234 25878 raft_consensus.cc:1261] T fe044374ad7b4999afc72ab812466749 P ee27c3f9e0ec46caa742ac386d6b7c20 [term 1 LEARNER]: Refusing update from remote peer 038c5d9d1d7a49c981c4c34fbc00d9e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:27:47.998523 27348 consensus_queue.cc:1035] T fe044374ad7b4999afc72ab812466749 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:27:48.000128 27348 consensus_queue.cc:1035] T fe044374ad7b4999afc72ab812466749 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:27:48.001397 23914 raft_consensus.cc:1261] T fe044374ad7b4999afc72ab812466749 P 6810a866decb4f5089f7d35c870e4204 [term 1 FOLLOWER]: Refusing update from remote peer 038c5d9d1d7a49c981c4c34fbc00d9e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:27:48.002003 26560 consensus_queue.cc:1035] T fe044374ad7b4999afc72ab812466749 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:27:48.010177 27317 raft_consensus.cc:1050] T 3cb3e62ca0e94e659efb3b7b0c2c9c91 P 6c194a6db6c145acafc0f3068b8ce1fd: attempting to promote NON_VOTER e117afbf8819424e8a98476453dfde2a to VOTER
I20240502 21:27:48.011204 27317 consensus_queue.cc:237] T 3cb3e62ca0e94e659efb3b7b0c2c9c91 P 6c194a6db6c145acafc0f3068b8ce1fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } attrs { replace: true } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } }
I20240502 21:27:48.014487 26560 raft_consensus.cc:2932] T fe044374ad7b4999afc72ab812466749 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ee27c3f9e0ec46caa742ac386d6b7c20 (127.15.255.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } attrs { replace: true } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: false } } }
I20240502 21:27:48.015149 23916 raft_consensus.cc:1261] T 3cb3e62ca0e94e659efb3b7b0c2c9c91 P 6810a866decb4f5089f7d35c870e4204 [term 1 FOLLOWER]: Refusing update from remote peer 6c194a6db6c145acafc0f3068b8ce1fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:27:48.015537 26147 raft_consensus.cc:1261] T 3cb3e62ca0e94e659efb3b7b0c2c9c91 P e117afbf8819424e8a98476453dfde2a [term 1 LEARNER]: Refusing update from remote peer 6c194a6db6c145acafc0f3068b8ce1fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:27:48.015887 24185 raft_consensus.cc:1261] T 3cb3e62ca0e94e659efb3b7b0c2c9c91 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 1 FOLLOWER]: Refusing update from remote peer 6c194a6db6c145acafc0f3068b8ce1fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:27:48.016717 25878 raft_consensus.cc:2932] T fe044374ad7b4999afc72ab812466749 P ee27c3f9e0ec46caa742ac386d6b7c20 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ee27c3f9e0ec46caa742ac386d6b7c20 (127.15.255.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } attrs { replace: true } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: false } } }
I20240502 21:27:48.017182 27324 consensus_queue.cc:1035] T 3cb3e62ca0e94e659efb3b7b0c2c9c91 P 6c194a6db6c145acafc0f3068b8ce1fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:27:48.017733 27324 consensus_queue.cc:1035] T 3cb3e62ca0e94e659efb3b7b0c2c9c91 P 6c194a6db6c145acafc0f3068b8ce1fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:27:48.018188 27324 consensus_queue.cc:1035] T 3cb3e62ca0e94e659efb3b7b0c2c9c91 P 6c194a6db6c145acafc0f3068b8ce1fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:27:48.019209 23916 raft_consensus.cc:2932] T fe044374ad7b4999afc72ab812466749 P 6810a866decb4f5089f7d35c870e4204 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ee27c3f9e0ec46caa742ac386d6b7c20 (127.15.255.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } attrs { replace: true } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: false } } }
I20240502 21:27:48.023022 23770 catalog_manager.cc:5484] T fe044374ad7b4999afc72ab812466749 P ee27c3f9e0ec46caa742ac386d6b7c20 reported cstate change: config changed from index 4 to 5, ee27c3f9e0ec46caa742ac386d6b7c20 (127.15.255.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } attrs { replace: true } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: false } } }
I20240502 21:27:48.030289 24317 consensus_queue.cc:237] T fe044374ad7b4999afc72ab812466749 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: false } }
I20240502 21:27:48.030200 24184 raft_consensus.cc:2932] T fe044374ad7b4999afc72ab812466749 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ee27c3f9e0ec46caa742ac386d6b7c20 (127.15.255.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } attrs { replace: true } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: false } } }
I20240502 21:27:48.032675 25878 raft_consensus.cc:1261] T fe044374ad7b4999afc72ab812466749 P ee27c3f9e0ec46caa742ac386d6b7c20 [term 1 FOLLOWER]: Refusing update from remote peer 038c5d9d1d7a49c981c4c34fbc00d9e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:27:48.033277 26560 consensus_queue.cc:1035] T fe044374ad7b4999afc72ab812466749 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:27:48.033356 24045 consensus_queue.cc:237] T dc0e070513cc46ca82c1946423ad91c5 P 6c194a6db6c145acafc0f3068b8ce1fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } attrs { replace: true } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: NON_VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: true } }
I20240502 21:27:48.035022 23916 raft_consensus.cc:1261] T fe044374ad7b4999afc72ab812466749 P 6810a866decb4f5089f7d35c870e4204 [term 1 FOLLOWER]: Refusing update from remote peer 038c5d9d1d7a49c981c4c34fbc00d9e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:27:48.035632 26560 consensus_queue.cc:1035] T fe044374ad7b4999afc72ab812466749 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:27:48.036522 27348 raft_consensus.cc:2932] T fe044374ad7b4999afc72ab812466749 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 009d3e7e8a8b4e2497de3a73d01ced49 (127.15.255.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: false } } }
I20240502 21:27:48.037009 25879 raft_consensus.cc:2932] T fe044374ad7b4999afc72ab812466749 P ee27c3f9e0ec46caa742ac386d6b7c20 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 009d3e7e8a8b4e2497de3a73d01ced49 (127.15.255.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: false } } }
I20240502 21:27:48.042142 27310 raft_consensus.cc:1050] T a274546743354e79bffa26ae7000e895 P 6c194a6db6c145acafc0f3068b8ce1fd: attempting to promote NON_VOTER e117afbf8819424e8a98476453dfde2a to VOTER
I20240502 21:27:48.041930 26715 raft_consensus.cc:2932] T 3cb3e62ca0e94e659efb3b7b0c2c9c91 P 6c194a6db6c145acafc0f3068b8ce1fd [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e117afbf8819424e8a98476453dfde2a (127.15.255.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } attrs { replace: true } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } } }
I20240502 21:27:48.043007 27310 consensus_queue.cc:237] T a274546743354e79bffa26ae7000e895 P 6c194a6db6c145acafc0f3068b8ce1fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } attrs { replace: true } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } }
I20240502 21:27:48.043852 23755 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet fe044374ad7b4999afc72ab812466749 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:27:48.045993 23768 catalog_manager.cc:5484] T fe044374ad7b4999afc72ab812466749 P 038c5d9d1d7a49c981c4c34fbc00d9e6 reported cstate change: config changed from index 5 to 6, VOTER 009d3e7e8a8b4e2497de3a73d01ced49 (127.15.255.131) evicted. New cstate: current_term: 1 leader_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:27:48.046948 26012 raft_consensus.cc:1203] T fcc9cbeae6f343af9f84c73349f3dadb P fb6004b8640a4906aa0cfe63e383d40e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 21:27:48.050874 24183 raft_consensus.cc:2932] T 3cb3e62ca0e94e659efb3b7b0c2c9c91 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e117afbf8819424e8a98476453dfde2a (127.15.255.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } attrs { replace: true } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } } }
I20240502 21:27:48.054689 24162 tablet_service.cc:1514] Processing DeleteTablet for tablet fe044374ad7b4999afc72ab812466749 with delete_type TABLET_DATA_TOMBSTONED (Replica with old config index 5 (current committed config index is 6)) from {username='slave'} at 127.0.0.1:33102
I20240502 21:27:48.055054 27296 tablet_replica.cc:330] T fe044374ad7b4999afc72ab812466749 P 009d3e7e8a8b4e2497de3a73d01ced49: stopping tablet replica
I20240502 21:27:48.055275 27296 raft_consensus.cc:2229] T fe044374ad7b4999afc72ab812466749 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:48.057102 27296 raft_consensus.cc:2258] T fe044374ad7b4999afc72ab812466749 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:48.058811 27296 ts_tablet_manager.cc:1875] T fe044374ad7b4999afc72ab812466749 P 009d3e7e8a8b4e2497de3a73d01ced49: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:27:48.059015 24317 raft_consensus.cc:1261] T a274546743354e79bffa26ae7000e895 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 1 FOLLOWER]: Refusing update from remote peer 6c194a6db6c145acafc0f3068b8ce1fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:27:48.059851 27310 consensus_queue.cc:1035] T a274546743354e79bffa26ae7000e895 P 6c194a6db6c145acafc0f3068b8ce1fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:27:48.059897 23768 catalog_manager.cc:5484] T 3cb3e62ca0e94e659efb3b7b0c2c9c91 P 009d3e7e8a8b4e2497de3a73d01ced49 reported cstate change: config changed from index 4 to 5, e117afbf8819424e8a98476453dfde2a (127.15.255.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } attrs { replace: true } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } } }
I20240502 21:27:48.063889 27296 ts_tablet_manager.cc:1888] T fe044374ad7b4999afc72ab812466749 P 009d3e7e8a8b4e2497de3a73d01ced49: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 21:27:48.065649 23915 raft_consensus.cc:1261] T dc0e070513cc46ca82c1946423ad91c5 P 6810a866decb4f5089f7d35c870e4204 [term 1 FOLLOWER]: Refusing update from remote peer 6c194a6db6c145acafc0f3068b8ce1fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:27:48.065392 23914 raft_consensus.cc:2932] T 3cb3e62ca0e94e659efb3b7b0c2c9c91 P 6810a866decb4f5089f7d35c870e4204 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e117afbf8819424e8a98476453dfde2a (127.15.255.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } attrs { replace: true } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } } }
I20240502 21:27:48.066732 26145 raft_consensus.cc:1261] T a274546743354e79bffa26ae7000e895 P e117afbf8819424e8a98476453dfde2a [term 1 LEARNER]: Refusing update from remote peer 6c194a6db6c145acafc0f3068b8ce1fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:27:48.067242 27324 consensus_queue.cc:1035] T dc0e070513cc46ca82c1946423ad91c5 P 6c194a6db6c145acafc0f3068b8ce1fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:27:48.068274 26147 raft_consensus.cc:2932] T 3cb3e62ca0e94e659efb3b7b0c2c9c91 P e117afbf8819424e8a98476453dfde2a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e117afbf8819424e8a98476453dfde2a (127.15.255.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } attrs { replace: true } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } } }
W20240502 21:27:48.071957 23982 consensus_peers.cc:485] T dc0e070513cc46ca82c1946423ad91c5 P 6c194a6db6c145acafc0f3068b8ce1fd -> Peer e117afbf8819424e8a98476453dfde2a (127.15.255.135:32775): Couldn't send request to peer e117afbf8819424e8a98476453dfde2a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: dc0e070513cc46ca82c1946423ad91c5. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:48.072235 27324 consensus_queue.cc:1035] T a274546743354e79bffa26ae7000e895 P 6c194a6db6c145acafc0f3068b8ce1fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:27:48.069255 24183 raft_consensus.cc:1261] T dc0e070513cc46ca82c1946423ad91c5 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 1 FOLLOWER]: Refusing update from remote peer 6c194a6db6c145acafc0f3068b8ce1fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:27:48.075094 27296 log.cc:1196] T fe044374ad7b4999afc72ab812466749 P 009d3e7e8a8b4e2497de3a73d01ced49: Deleting WAL directory at /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-2/wal/wals/fe044374ad7b4999afc72ab812466749
I20240502 21:27:48.069532 24184 raft_consensus.cc:1261] T a274546743354e79bffa26ae7000e895 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 1 FOLLOWER]: Refusing update from remote peer 6c194a6db6c145acafc0f3068b8ce1fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:27:48.075546 27317 consensus_queue.cc:1035] T dc0e070513cc46ca82c1946423ad91c5 P 6c194a6db6c145acafc0f3068b8ce1fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:27:48.075664 27324 consensus_queue.cc:1035] T a274546743354e79bffa26ae7000e895 P 6c194a6db6c145acafc0f3068b8ce1fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:27:48.077744 23916 raft_consensus.cc:2932] T fe044374ad7b4999afc72ab812466749 P 6810a866decb4f5089f7d35c870e4204 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 009d3e7e8a8b4e2497de3a73d01ced49 (127.15.255.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: false } } }
I20240502 21:27:48.080900 23756 catalog_manager.cc:4830] TS 009d3e7e8a8b4e2497de3a73d01ced49 (127.15.255.131:37895): tablet fe044374ad7b4999afc72ab812466749 (table rebalance_test_table_6 [id=9aaf56e5a31f42a79b45624cc2c43779]) successfully deleted
I20240502 21:27:48.089581 27324 raft_consensus.cc:2932] T a274546743354e79bffa26ae7000e895 P 6c194a6db6c145acafc0f3068b8ce1fd [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e117afbf8819424e8a98476453dfde2a (127.15.255.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } attrs { replace: true } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } } }
I20240502 21:27:48.093958 27310 raft_consensus.cc:2932] T dc0e070513cc46ca82c1946423ad91c5 P 6c194a6db6c145acafc0f3068b8ce1fd [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER e117afbf8819424e8a98476453dfde2a (127.15.255.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } attrs { replace: true } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: NON_VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: true } } }
I20240502 21:27:48.093955 24317 raft_consensus.cc:2932] T a274546743354e79bffa26ae7000e895 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e117afbf8819424e8a98476453dfde2a (127.15.255.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } attrs { replace: true } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } } }
I20240502 21:27:48.097159 24183 raft_consensus.cc:2932] T dc0e070513cc46ca82c1946423ad91c5 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER e117afbf8819424e8a98476453dfde2a (127.15.255.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } attrs { replace: true } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: NON_VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: true } } }
I20240502 21:27:48.100486 23772 catalog_manager.cc:5484] T a274546743354e79bffa26ae7000e895 P 038c5d9d1d7a49c981c4c34fbc00d9e6 reported cstate change: config changed from index 4 to 5, e117afbf8819424e8a98476453dfde2a (127.15.255.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } attrs { replace: true } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } } }
I20240502 21:27:48.104179 26147 raft_consensus.cc:2932] T a274546743354e79bffa26ae7000e895 P e117afbf8819424e8a98476453dfde2a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e117afbf8819424e8a98476453dfde2a (127.15.255.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } attrs { replace: true } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } } }
I20240502 21:27:48.108309 24184 raft_consensus.cc:2932] T a274546743354e79bffa26ae7000e895 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e117afbf8819424e8a98476453dfde2a (127.15.255.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } attrs { replace: true } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } } }
I20240502 21:27:48.115001 23768 catalog_manager.cc:5484] T dc0e070513cc46ca82c1946423ad91c5 P 009d3e7e8a8b4e2497de3a73d01ced49 reported cstate change: config changed from index -1 to 4, NON_VOTER e117afbf8819424e8a98476453dfde2a (127.15.255.135) added. New cstate: current_term: 1 leader_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } attrs { replace: true } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: NON_VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: true } } }
I20240502 21:27:48.115784 23915 raft_consensus.cc:2932] T dc0e070513cc46ca82c1946423ad91c5 P 6810a866decb4f5089f7d35c870e4204 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER e117afbf8819424e8a98476453dfde2a (127.15.255.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } attrs { replace: true } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: NON_VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: true } } }
I20240502 21:27:48.127254 24045 consensus_queue.cc:237] T 3cb3e62ca0e94e659efb3b7b0c2c9c91 P 6c194a6db6c145acafc0f3068b8ce1fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } }
I20240502 21:27:48.129436 26147 raft_consensus.cc:1261] T 3cb3e62ca0e94e659efb3b7b0c2c9c91 P e117afbf8819424e8a98476453dfde2a [term 1 FOLLOWER]: Refusing update from remote peer 6c194a6db6c145acafc0f3068b8ce1fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:27:48.130342 24184 raft_consensus.cc:1261] T 3cb3e62ca0e94e659efb3b7b0c2c9c91 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 1 FOLLOWER]: Refusing update from remote peer 6c194a6db6c145acafc0f3068b8ce1fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:27:48.130831 27310 consensus_queue.cc:1035] T 3cb3e62ca0e94e659efb3b7b0c2c9c91 P 6c194a6db6c145acafc0f3068b8ce1fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:27:48.131203 27456 consensus_queue.cc:1035] T 3cb3e62ca0e94e659efb3b7b0c2c9c91 P 6c194a6db6c145acafc0f3068b8ce1fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:27:48.134320 26715 raft_consensus.cc:2932] T 3cb3e62ca0e94e659efb3b7b0c2c9c91 P 6c194a6db6c145acafc0f3068b8ce1fd [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 6810a866decb4f5089f7d35c870e4204 (127.15.255.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } } }
I20240502 21:27:48.134780 26147 raft_consensus.cc:2932] T 3cb3e62ca0e94e659efb3b7b0c2c9c91 P e117afbf8819424e8a98476453dfde2a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 6810a866decb4f5089f7d35c870e4204 (127.15.255.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } } }
I20240502 21:27:48.137440 24045 consensus_queue.cc:237] T a274546743354e79bffa26ae7000e895 P 6c194a6db6c145acafc0f3068b8ce1fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } }
I20240502 21:27:48.137914 24183 raft_consensus.cc:2932] T 3cb3e62ca0e94e659efb3b7b0c2c9c91 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 6810a866decb4f5089f7d35c870e4204 (127.15.255.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } } }
I20240502 21:27:48.139883 24184 raft_consensus.cc:1261] T a274546743354e79bffa26ae7000e895 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 1 FOLLOWER]: Refusing update from remote peer 6c194a6db6c145acafc0f3068b8ce1fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:27:48.139827 23770 catalog_manager.cc:5484] T 3cb3e62ca0e94e659efb3b7b0c2c9c91 P e117afbf8819424e8a98476453dfde2a reported cstate change: config changed from index 5 to 6, VOTER 6810a866decb4f5089f7d35c870e4204 (127.15.255.129) evicted. New cstate: current_term: 1 leader_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } } }
I20240502 21:27:48.140491 27458 consensus_queue.cc:1035] T a274546743354e79bffa26ae7000e895 P 6c194a6db6c145acafc0f3068b8ce1fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:27:48.142463 26147 raft_consensus.cc:1261] T a274546743354e79bffa26ae7000e895 P e117afbf8819424e8a98476453dfde2a [term 1 FOLLOWER]: Refusing update from remote peer 6c194a6db6c145acafc0f3068b8ce1fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:27:48.143311 23757 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 3cb3e62ca0e94e659efb3b7b0c2c9c91 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:27:48.143350 27457 raft_consensus.cc:2932] T a274546743354e79bffa26ae7000e895 P 6c194a6db6c145acafc0f3068b8ce1fd [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 038c5d9d1d7a49c981c4c34fbc00d9e6 (127.15.255.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } } }
I20240502 21:27:48.145244 27317 consensus_queue.cc:1035] T a274546743354e79bffa26ae7000e895 P 6c194a6db6c145acafc0f3068b8ce1fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:27:48.146059 23894 tablet_service.cc:1514] Processing DeleteTablet for tablet 3cb3e62ca0e94e659efb3b7b0c2c9c91 with delete_type TABLET_DATA_TOMBSTONED (TS 6810a866decb4f5089f7d35c870e4204 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:33512
I20240502 21:27:48.146399 27301 tablet_replica.cc:330] T 3cb3e62ca0e94e659efb3b7b0c2c9c91 P 6810a866decb4f5089f7d35c870e4204: stopping tablet replica
I20240502 21:27:48.146585 27301 raft_consensus.cc:2229] T 3cb3e62ca0e94e659efb3b7b0c2c9c91 P 6810a866decb4f5089f7d35c870e4204 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:48.146915 27301 raft_consensus.cc:2258] T 3cb3e62ca0e94e659efb3b7b0c2c9c91 P 6810a866decb4f5089f7d35c870e4204 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:48.147929 27301 ts_tablet_manager.cc:1875] T 3cb3e62ca0e94e659efb3b7b0c2c9c91 P 6810a866decb4f5089f7d35c870e4204: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:27:48.150002 23757 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet a274546743354e79bffa26ae7000e895 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:27:48.150620 26147 raft_consensus.cc:2932] T a274546743354e79bffa26ae7000e895 P e117afbf8819424e8a98476453dfde2a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 038c5d9d1d7a49c981c4c34fbc00d9e6 (127.15.255.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } } }
I20240502 21:27:48.154131 24183 raft_consensus.cc:2932] T a274546743354e79bffa26ae7000e895 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 038c5d9d1d7a49c981c4c34fbc00d9e6 (127.15.255.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } } }
I20240502 21:27:48.155128 23770 catalog_manager.cc:5484] T a274546743354e79bffa26ae7000e895 P 6c194a6db6c145acafc0f3068b8ce1fd reported cstate change: config changed from index 5 to 6, VOTER 038c5d9d1d7a49c981c4c34fbc00d9e6 (127.15.255.132) evicted. New cstate: current_term: 1 leader_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:48.157975 27301 ts_tablet_manager.cc:1888] T 3cb3e62ca0e94e659efb3b7b0c2c9c91 P 6810a866decb4f5089f7d35c870e4204: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 21:27:48.158114 27301 log.cc:1196] T 3cb3e62ca0e94e659efb3b7b0c2c9c91 P 6810a866decb4f5089f7d35c870e4204: Deleting WAL directory at /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-0/wal/wals/3cb3e62ca0e94e659efb3b7b0c2c9c91
I20240502 21:27:48.160347 23757 catalog_manager.cc:4830] TS 6810a866decb4f5089f7d35c870e4204 (127.15.255.129:40435): tablet 3cb3e62ca0e94e659efb3b7b0c2c9c91 (table rebalance_test_table_4 [id=9d2b330e886e4e83a437506bf99d77a3]) successfully deleted
I20240502 21:27:48.162617 24297 tablet_service.cc:1514] Processing DeleteTablet for tablet a274546743354e79bffa26ae7000e895 with delete_type TABLET_DATA_TOMBSTONED (TS 038c5d9d1d7a49c981c4c34fbc00d9e6 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:58268
I20240502 21:27:48.163014 27312 tablet_replica.cc:330] T a274546743354e79bffa26ae7000e895 P 038c5d9d1d7a49c981c4c34fbc00d9e6: stopping tablet replica
I20240502 21:27:48.163231 27312 raft_consensus.cc:2229] T a274546743354e79bffa26ae7000e895 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:48.163597 27312 raft_consensus.cc:2258] T a274546743354e79bffa26ae7000e895 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:48.164573 27312 ts_tablet_manager.cc:1875] T a274546743354e79bffa26ae7000e895 P 038c5d9d1d7a49c981c4c34fbc00d9e6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:27:48.169718 27312 ts_tablet_manager.cc:1888] T a274546743354e79bffa26ae7000e895 P 038c5d9d1d7a49c981c4c34fbc00d9e6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 21:27:48.169821 27312 log.cc:1196] T a274546743354e79bffa26ae7000e895 P 038c5d9d1d7a49c981c4c34fbc00d9e6: Deleting WAL directory at /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-3/wal/wals/a274546743354e79bffa26ae7000e895
I20240502 21:27:48.170435 23755 catalog_manager.cc:4830] TS 038c5d9d1d7a49c981c4c34fbc00d9e6 (127.15.255.132:46813): tablet a274546743354e79bffa26ae7000e895 (table rebalance_test_table_3 [id=933ce90acb9f4b91ab6bd41e87a8fe99]) successfully deleted
I20240502 21:27:48.177170 23915 consensus_queue.cc:237] T 30236241b670443489240f191d9b9d50 P 6810a866decb4f5089f7d35c870e4204 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } attrs { replace: true } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: NON_VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: true } }
I20240502 21:27:48.179260 24317 raft_consensus.cc:1261] T 30236241b670443489240f191d9b9d50 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 1 FOLLOWER]: Refusing update from remote peer 6810a866decb4f5089f7d35c870e4204: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 21:27:48.179580 23847 consensus_peers.cc:485] T 30236241b670443489240f191d9b9d50 P 6810a866decb4f5089f7d35c870e4204 -> Peer fb6004b8640a4906aa0cfe63e383d40e (127.15.255.134:32825): Couldn't send request to peer fb6004b8640a4906aa0cfe63e383d40e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 30236241b670443489240f191d9b9d50. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:48.179630 24045 raft_consensus.cc:1261] T 30236241b670443489240f191d9b9d50 P 6c194a6db6c145acafc0f3068b8ce1fd [term 1 FOLLOWER]: Refusing update from remote peer 6810a866decb4f5089f7d35c870e4204: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:27:48.180058 27328 consensus_queue.cc:1035] T 30236241b670443489240f191d9b9d50 P 6810a866decb4f5089f7d35c870e4204 [LEADER]: Connected to new peer: Peer: permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:27:48.180348 26586 consensus_queue.cc:1035] T 30236241b670443489240f191d9b9d50 P 6810a866decb4f5089f7d35c870e4204 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:27:48.182987 27234 raft_consensus.cc:2932] T 30236241b670443489240f191d9b9d50 P 6810a866decb4f5089f7d35c870e4204 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER fb6004b8640a4906aa0cfe63e383d40e (127.15.255.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } attrs { replace: true } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: NON_VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: true } } }
I20240502 21:27:48.184778 26475 raft_consensus.cc:1050] T f84bbe304597450c978ae28e0c4e7a3e P 009d3e7e8a8b4e2497de3a73d01ced49: attempting to promote NON_VOTER a3e500d7077a48acb5edb33f7e178c81 to VOTER
I20240502 21:27:48.185597 26475 consensus_queue.cc:237] T f84bbe304597450c978ae28e0c4e7a3e P 009d3e7e8a8b4e2497de3a73d01ced49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } attrs { replace: true } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } }
I20240502 21:27:48.186522 24045 raft_consensus.cc:2932] T 30236241b670443489240f191d9b9d50 P 6c194a6db6c145acafc0f3068b8ce1fd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER fb6004b8640a4906aa0cfe63e383d40e (127.15.255.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } attrs { replace: true } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: NON_VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: true } } }
I20240502 21:27:48.186834 24317 raft_consensus.cc:2932] T 30236241b670443489240f191d9b9d50 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER fb6004b8640a4906aa0cfe63e383d40e (127.15.255.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } attrs { replace: true } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: NON_VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: true } } }
I20240502 21:27:48.187994 23768 catalog_manager.cc:5484] T 30236241b670443489240f191d9b9d50 P 6810a866decb4f5089f7d35c870e4204 reported cstate change: config changed from index -1 to 4, NON_VOTER fb6004b8640a4906aa0cfe63e383d40e (127.15.255.134) added. New cstate: current_term: 1 leader_uuid: "6810a866decb4f5089f7d35c870e4204" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: NON_VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:48.190903 24047 raft_consensus.cc:1261] T f84bbe304597450c978ae28e0c4e7a3e P 6c194a6db6c145acafc0f3068b8ce1fd [term 1 FOLLOWER]: Refusing update from remote peer 009d3e7e8a8b4e2497de3a73d01ced49: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:27:48.191641 27292 consensus_queue.cc:1035] T f84bbe304597450c978ae28e0c4e7a3e P 009d3e7e8a8b4e2497de3a73d01ced49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:27:48.191967 24315 raft_consensus.cc:1261] T f84bbe304597450c978ae28e0c4e7a3e P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 1 FOLLOWER]: Refusing update from remote peer 009d3e7e8a8b4e2497de3a73d01ced49: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:27:48.192435 26280 raft_consensus.cc:1261] T f84bbe304597450c978ae28e0c4e7a3e P a3e500d7077a48acb5edb33f7e178c81 [term 1 LEARNER]: Refusing update from remote peer 009d3e7e8a8b4e2497de3a73d01ced49: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:27:48.193542 26554 consensus_queue.cc:1035] T f84bbe304597450c978ae28e0c4e7a3e P 009d3e7e8a8b4e2497de3a73d01ced49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:27:48.194021 26554 consensus_queue.cc:1035] T f84bbe304597450c978ae28e0c4e7a3e P 009d3e7e8a8b4e2497de3a73d01ced49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:27:48.196858 27215 raft_consensus.cc:2932] T f84bbe304597450c978ae28e0c4e7a3e P 009d3e7e8a8b4e2497de3a73d01ced49 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a3e500d7077a48acb5edb33f7e178c81 (127.15.255.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } attrs { replace: true } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } } }
I20240502 21:27:48.197288 26280 raft_consensus.cc:2932] T f84bbe304597450c978ae28e0c4e7a3e P a3e500d7077a48acb5edb33f7e178c81 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a3e500d7077a48acb5edb33f7e178c81 (127.15.255.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } attrs { replace: true } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } } }
I20240502 21:27:48.201493 23768 catalog_manager.cc:5484] T f84bbe304597450c978ae28e0c4e7a3e P a3e500d7077a48acb5edb33f7e178c81 reported cstate change: config changed from index 4 to 5, a3e500d7077a48acb5edb33f7e178c81 (127.15.255.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } attrs { replace: true } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } } }
I20240502 21:27:48.202791 24047 raft_consensus.cc:2932] T f84bbe304597450c978ae28e0c4e7a3e P 6c194a6db6c145acafc0f3068b8ce1fd [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a3e500d7077a48acb5edb33f7e178c81 (127.15.255.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } attrs { replace: true } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } } }
I20240502 21:27:48.217170 24317 raft_consensus.cc:2932] T f84bbe304597450c978ae28e0c4e7a3e P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a3e500d7077a48acb5edb33f7e178c81 (127.15.255.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } attrs { replace: true } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } } }
I20240502 21:27:48.272186 23915 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "89f86d58fe414b579845214534d338d8"
dest_uuid: "6810a866decb4f5089f7d35c870e4204"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:49366
I20240502 21:27:48.272454 23915 raft_consensus.cc:595] T 89f86d58fe414b579845214534d338d8 P 6810a866decb4f5089f7d35c870e4204 [term 1 LEADER]: Received request to transfer leadership
I20240502 21:27:48.337435 27215 raft_consensus.cc:1050] T 16bb47f03a2f46fdb7344e018f07fdcb P 009d3e7e8a8b4e2497de3a73d01ced49: attempting to promote NON_VOTER ee27c3f9e0ec46caa742ac386d6b7c20 to VOTER
I20240502 21:27:48.338340 27215 consensus_queue.cc:237] T 16bb47f03a2f46fdb7344e018f07fdcb P 009d3e7e8a8b4e2497de3a73d01ced49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } attrs { replace: true } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: false } }
I20240502 21:27:48.340817 23915 raft_consensus.cc:1261] T 16bb47f03a2f46fdb7344e018f07fdcb P 6810a866decb4f5089f7d35c870e4204 [term 1 FOLLOWER]: Refusing update from remote peer 009d3e7e8a8b4e2497de3a73d01ced49: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:27:48.340835 24046 raft_consensus.cc:1261] T 16bb47f03a2f46fdb7344e018f07fdcb P 6c194a6db6c145acafc0f3068b8ce1fd [term 1 FOLLOWER]: Refusing update from remote peer 009d3e7e8a8b4e2497de3a73d01ced49: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:27:48.341341 26475 consensus_queue.cc:1035] T 16bb47f03a2f46fdb7344e018f07fdcb P 009d3e7e8a8b4e2497de3a73d01ced49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:27:48.341641 26554 consensus_queue.cc:1035] T 16bb47f03a2f46fdb7344e018f07fdcb P 009d3e7e8a8b4e2497de3a73d01ced49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:27:48.342571 25879 raft_consensus.cc:1261] T 16bb47f03a2f46fdb7344e018f07fdcb P ee27c3f9e0ec46caa742ac386d6b7c20 [term 1 LEARNER]: Refusing update from remote peer 009d3e7e8a8b4e2497de3a73d01ced49: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:27:48.343134 26475 consensus_queue.cc:1035] T 16bb47f03a2f46fdb7344e018f07fdcb P 009d3e7e8a8b4e2497de3a73d01ced49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:27:48.356730 26475 raft_consensus.cc:2932] T 16bb47f03a2f46fdb7344e018f07fdcb P 009d3e7e8a8b4e2497de3a73d01ced49 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ee27c3f9e0ec46caa742ac386d6b7c20 (127.15.255.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } attrs { replace: true } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: false } } }
I20240502 21:27:48.358012 24046 raft_consensus.cc:2932] T 16bb47f03a2f46fdb7344e018f07fdcb P 6c194a6db6c145acafc0f3068b8ce1fd [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ee27c3f9e0ec46caa742ac386d6b7c20 (127.15.255.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } attrs { replace: true } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: false } } }
I20240502 21:27:48.360939 25879 raft_consensus.cc:2932] T 16bb47f03a2f46fdb7344e018f07fdcb P ee27c3f9e0ec46caa742ac386d6b7c20 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ee27c3f9e0ec46caa742ac386d6b7c20 (127.15.255.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } attrs { replace: true } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: false } } }
I20240502 21:27:48.363385 23768 catalog_manager.cc:5484] T 16bb47f03a2f46fdb7344e018f07fdcb P 6c194a6db6c145acafc0f3068b8ce1fd reported cstate change: config changed from index 4 to 5, ee27c3f9e0ec46caa742ac386d6b7c20 (127.15.255.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } attrs { replace: true } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: false } } }
I20240502 21:27:48.366441 23915 raft_consensus.cc:2932] T 16bb47f03a2f46fdb7344e018f07fdcb P 6810a866decb4f5089f7d35c870e4204 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ee27c3f9e0ec46caa742ac386d6b7c20 (127.15.255.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } attrs { replace: true } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: false } } }
I20240502 21:27:48.369602 24185 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f84bbe304597450c978ae28e0c4e7a3e"
dest_uuid: "009d3e7e8a8b4e2497de3a73d01ced49"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:44608
I20240502 21:27:48.369855 24185 raft_consensus.cc:595] T f84bbe304597450c978ae28e0c4e7a3e P 009d3e7e8a8b4e2497de3a73d01ced49 [term 1 LEADER]: Received request to transfer leadership
I20240502 21:27:48.370123 27313 raft_consensus.cc:1050] T 83b95f3ae7564b81b31a2f5673ee22bb P 6810a866decb4f5089f7d35c870e4204: attempting to promote NON_VOTER e117afbf8819424e8a98476453dfde2a to VOTER
I20240502 21:27:48.370581 24183 consensus_queue.cc:237] T 16bb47f03a2f46fdb7344e018f07fdcb P 009d3e7e8a8b4e2497de3a73d01ced49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: false } }
I20240502 21:27:48.370913 27313 consensus_queue.cc:237] T 83b95f3ae7564b81b31a2f5673ee22bb P 6810a866decb4f5089f7d35c870e4204 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } attrs { replace: true } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } }
I20240502 21:27:48.372483 25879 raft_consensus.cc:1261] T 16bb47f03a2f46fdb7344e018f07fdcb P ee27c3f9e0ec46caa742ac386d6b7c20 [term 1 FOLLOWER]: Refusing update from remote peer 009d3e7e8a8b4e2497de3a73d01ced49: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:27:48.372593 24046 raft_consensus.cc:1261] T 16bb47f03a2f46fdb7344e018f07fdcb P 6c194a6db6c145acafc0f3068b8ce1fd [term 1 FOLLOWER]: Refusing update from remote peer 009d3e7e8a8b4e2497de3a73d01ced49: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:27:48.374092 27292 consensus_queue.cc:1035] T 16bb47f03a2f46fdb7344e018f07fdcb P 009d3e7e8a8b4e2497de3a73d01ced49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:27:48.374408 24315 raft_consensus.cc:1261] T 83b95f3ae7564b81b31a2f5673ee22bb P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 1 FOLLOWER]: Refusing update from remote peer 6810a866decb4f5089f7d35c870e4204: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:27:48.374985 27234 consensus_queue.cc:1035] T 83b95f3ae7564b81b31a2f5673ee22bb P 6810a866decb4f5089f7d35c870e4204 [LEADER]: Connected to new peer: Peer: permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:27:48.376619 26147 raft_consensus.cc:1261] T 83b95f3ae7564b81b31a2f5673ee22bb P e117afbf8819424e8a98476453dfde2a [term 1 LEARNER]: Refusing update from remote peer 6810a866decb4f5089f7d35c870e4204: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:27:48.377307 27292 consensus_queue.cc:1035] T 16bb47f03a2f46fdb7344e018f07fdcb P 009d3e7e8a8b4e2497de3a73d01ced49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:27:48.377650 27234 consensus_queue.cc:1035] T 83b95f3ae7564b81b31a2f5673ee22bb P 6810a866decb4f5089f7d35c870e4204 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:27:48.377775 27292 raft_consensus.cc:2932] T 16bb47f03a2f46fdb7344e018f07fdcb P 009d3e7e8a8b4e2497de3a73d01ced49 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 6810a866decb4f5089f7d35c870e4204 (127.15.255.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: false } } }
I20240502 21:27:48.378830 25879 raft_consensus.cc:2932] T 16bb47f03a2f46fdb7344e018f07fdcb P ee27c3f9e0ec46caa742ac386d6b7c20 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 6810a866decb4f5089f7d35c870e4204 (127.15.255.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: false } } }
I20240502 21:27:48.383291 23770 catalog_manager.cc:5484] T 16bb47f03a2f46fdb7344e018f07fdcb P ee27c3f9e0ec46caa742ac386d6b7c20 reported cstate change: config changed from index 5 to 6, VOTER 6810a866decb4f5089f7d35c870e4204 (127.15.255.129) evicted. New cstate: current_term: 1 leader_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: false } } }
I20240502 21:27:48.383957 24046 raft_consensus.cc:1261] T 83b95f3ae7564b81b31a2f5673ee22bb P 6c194a6db6c145acafc0f3068b8ce1fd [term 1 FOLLOWER]: Refusing update from remote peer 6810a866decb4f5089f7d35c870e4204: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:27:48.384536 27234 consensus_queue.cc:1035] T 83b95f3ae7564b81b31a2f5673ee22bb P 6810a866decb4f5089f7d35c870e4204 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:27:48.387997 23756 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 16bb47f03a2f46fdb7344e018f07fdcb with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:27:48.388080 23894 tablet_service.cc:1514] Processing DeleteTablet for tablet 16bb47f03a2f46fdb7344e018f07fdcb with delete_type TABLET_DATA_TOMBSTONED (TS 6810a866decb4f5089f7d35c870e4204 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:33512
I20240502 21:27:48.388439 27301 tablet_replica.cc:330] T 16bb47f03a2f46fdb7344e018f07fdcb P 6810a866decb4f5089f7d35c870e4204: stopping tablet replica
I20240502 21:27:48.388646 27301 raft_consensus.cc:2229] T 16bb47f03a2f46fdb7344e018f07fdcb P 6810a866decb4f5089f7d35c870e4204 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:48.388891 27301 raft_consensus.cc:2258] T 16bb47f03a2f46fdb7344e018f07fdcb P 6810a866decb4f5089f7d35c870e4204 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:48.388638 24047 raft_consensus.cc:2932] T 16bb47f03a2f46fdb7344e018f07fdcb P 6c194a6db6c145acafc0f3068b8ce1fd [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 6810a866decb4f5089f7d35c870e4204 (127.15.255.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: false } } }
I20240502 21:27:48.389973 27301 ts_tablet_manager.cc:1875] T 16bb47f03a2f46fdb7344e018f07fdcb P 6810a866decb4f5089f7d35c870e4204: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:27:48.395689 27234 raft_consensus.cc:2932] T 83b95f3ae7564b81b31a2f5673ee22bb P 6810a866decb4f5089f7d35c870e4204 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e117afbf8819424e8a98476453dfde2a (127.15.255.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } attrs { replace: true } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } } }
I20240502 21:27:48.397706 26147 raft_consensus.cc:2932] T 83b95f3ae7564b81b31a2f5673ee22bb P e117afbf8819424e8a98476453dfde2a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e117afbf8819424e8a98476453dfde2a (127.15.255.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } attrs { replace: true } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } } }
I20240502 21:27:48.398833 27301 ts_tablet_manager.cc:1888] T 16bb47f03a2f46fdb7344e018f07fdcb P 6810a866decb4f5089f7d35c870e4204: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 21:27:48.399005 27301 log.cc:1196] T 16bb47f03a2f46fdb7344e018f07fdcb P 6810a866decb4f5089f7d35c870e4204: Deleting WAL directory at /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-0/wal/wals/16bb47f03a2f46fdb7344e018f07fdcb
I20240502 21:27:48.400158 26560 raft_consensus.cc:1050] T d5697d8b06094211b261f144de9d46f8 P 038c5d9d1d7a49c981c4c34fbc00d9e6: attempting to promote NON_VOTER a3e500d7077a48acb5edb33f7e178c81 to VOTER
I20240502 21:27:48.400462 24047 raft_consensus.cc:2932] T 83b95f3ae7564b81b31a2f5673ee22bb P 6c194a6db6c145acafc0f3068b8ce1fd [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e117afbf8819424e8a98476453dfde2a (127.15.255.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } attrs { replace: true } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } } }
I20240502 21:27:48.401023 26560 consensus_queue.cc:237] T d5697d8b06094211b261f144de9d46f8 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } attrs { replace: true } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } }
I20240502 21:27:48.403075 24317 raft_consensus.cc:2932] T 83b95f3ae7564b81b31a2f5673ee22bb P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e117afbf8819424e8a98476453dfde2a (127.15.255.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } attrs { replace: true } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } } }
I20240502 21:27:48.403486 23770 catalog_manager.cc:5484] T 83b95f3ae7564b81b31a2f5673ee22bb P 6810a866decb4f5089f7d35c870e4204 reported cstate change: config changed from index 4 to 5, e117afbf8819424e8a98476453dfde2a (127.15.255.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "6810a866decb4f5089f7d35c870e4204" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:27:48.408890 23757 catalog_manager.cc:4830] TS 6810a866decb4f5089f7d35c870e4204 (127.15.255.129:40435): tablet 16bb47f03a2f46fdb7344e018f07fdcb (table rebalance_test_table_8 [id=9a4e7bd31db34cf3aff2242876f0de47]) successfully deleted
I20240502 21:27:48.409917 23915 consensus_queue.cc:237] T 83b95f3ae7564b81b31a2f5673ee22bb P 6810a866decb4f5089f7d35c870e4204 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } }
I20240502 21:27:48.412453 24317 raft_consensus.cc:1261] T 83b95f3ae7564b81b31a2f5673ee22bb P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 1 FOLLOWER]: Refusing update from remote peer 6810a866decb4f5089f7d35c870e4204: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:27:48.412972 27327 consensus_queue.cc:1035] T 83b95f3ae7564b81b31a2f5673ee22bb P 6810a866decb4f5089f7d35c870e4204 [LEADER]: Connected to new peer: Peer: permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:27:48.415889 27327 raft_consensus.cc:2932] T 83b95f3ae7564b81b31a2f5673ee22bb P 6810a866decb4f5089f7d35c870e4204 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 6c194a6db6c145acafc0f3068b8ce1fd (127.15.255.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } } }
I20240502 21:27:48.419775 24316 raft_consensus.cc:2932] T 83b95f3ae7564b81b31a2f5673ee22bb P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 6c194a6db6c145acafc0f3068b8ce1fd (127.15.255.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } } }
I20240502 21:27:48.420604 26145 raft_consensus.cc:1261] T 83b95f3ae7564b81b31a2f5673ee22bb P e117afbf8819424e8a98476453dfde2a [term 1 FOLLOWER]: Refusing update from remote peer 6810a866decb4f5089f7d35c870e4204: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:27:48.425382 23770 catalog_manager.cc:5484] T 83b95f3ae7564b81b31a2f5673ee22bb P 038c5d9d1d7a49c981c4c34fbc00d9e6 reported cstate change: config changed from index 5 to 6, VOTER 6c194a6db6c145acafc0f3068b8ce1fd (127.15.255.130) evicted. New cstate: current_term: 1 leader_uuid: "6810a866decb4f5089f7d35c870e4204" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } } }
I20240502 21:27:48.425900 27328 consensus_queue.cc:1035] T 83b95f3ae7564b81b31a2f5673ee22bb P 6810a866decb4f5089f7d35c870e4204 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:27:48.427381 26145 raft_consensus.cc:2932] T 83b95f3ae7564b81b31a2f5673ee22bb P e117afbf8819424e8a98476453dfde2a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 6c194a6db6c145acafc0f3068b8ce1fd (127.15.255.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } } }
I20240502 21:27:48.427953 23757 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 83b95f3ae7564b81b31a2f5673ee22bb with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:27:48.430323 26280 raft_consensus.cc:1261] T d5697d8b06094211b261f144de9d46f8 P a3e500d7077a48acb5edb33f7e178c81 [term 1 LEARNER]: Refusing update from remote peer 038c5d9d1d7a49c981c4c34fbc00d9e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:27:48.430913 24183 raft_consensus.cc:1261] T d5697d8b06094211b261f144de9d46f8 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 1 FOLLOWER]: Refusing update from remote peer 038c5d9d1d7a49c981c4c34fbc00d9e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:27:48.430934 26560 consensus_queue.cc:1035] T d5697d8b06094211b261f144de9d46f8 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:27:48.432288 24047 raft_consensus.cc:1261] T d5697d8b06094211b261f144de9d46f8 P 6c194a6db6c145acafc0f3068b8ce1fd [term 1 FOLLOWER]: Refusing update from remote peer 038c5d9d1d7a49c981c4c34fbc00d9e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:27:48.433650 27348 consensus_queue.cc:1035] T d5697d8b06094211b261f144de9d46f8 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:27:48.434233 27348 consensus_queue.cc:1035] T d5697d8b06094211b261f144de9d46f8 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:27:48.436460 24027 tablet_service.cc:1514] Processing DeleteTablet for tablet 83b95f3ae7564b81b31a2f5673ee22bb with delete_type TABLET_DATA_TOMBSTONED (TS 6c194a6db6c145acafc0f3068b8ce1fd not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:47386
I20240502 21:27:48.441778 26420 raft_consensus.cc:2932] T d5697d8b06094211b261f144de9d46f8 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a3e500d7077a48acb5edb33f7e178c81 (127.15.255.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } attrs { replace: true } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } } }
I20240502 21:27:48.445745 26280 raft_consensus.cc:2932] T d5697d8b06094211b261f144de9d46f8 P a3e500d7077a48acb5edb33f7e178c81 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a3e500d7077a48acb5edb33f7e178c81 (127.15.255.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } attrs { replace: true } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } } }
I20240502 21:27:48.450230 24183 raft_consensus.cc:2932] T d5697d8b06094211b261f144de9d46f8 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a3e500d7077a48acb5edb33f7e178c81 (127.15.255.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } attrs { replace: true } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } } }
I20240502 21:27:48.450727 23770 catalog_manager.cc:5484] T d5697d8b06094211b261f144de9d46f8 P a3e500d7077a48acb5edb33f7e178c81 reported cstate change: config changed from index 4 to 5, a3e500d7077a48acb5edb33f7e178c81 (127.15.255.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } attrs { replace: true } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } } }
I20240502 21:27:48.451588 27349 raft_consensus.cc:1050] T fcc9cbeae6f343af9f84c73349f3dadb P 038c5d9d1d7a49c981c4c34fbc00d9e6: attempting to promote NON_VOTER fb6004b8640a4906aa0cfe63e383d40e to VOTER
I20240502 21:27:48.452340 27349 consensus_queue.cc:237] T fcc9cbeae6f343af9f84c73349f3dadb P 038c5d9d1d7a49c981c4c34fbc00d9e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } attrs { replace: true } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: false } }
I20240502 21:27:48.451762 27557 tablet_replica.cc:330] T 83b95f3ae7564b81b31a2f5673ee22bb P 6c194a6db6c145acafc0f3068b8ce1fd: stopping tablet replica
I20240502 21:27:48.454576 24047 raft_consensus.cc:2932] T d5697d8b06094211b261f144de9d46f8 P 6c194a6db6c145acafc0f3068b8ce1fd [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a3e500d7077a48acb5edb33f7e178c81 (127.15.255.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } attrs { replace: true } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } } }
I20240502 21:27:48.455077 27557 raft_consensus.cc:2229] T 83b95f3ae7564b81b31a2f5673ee22bb P 6c194a6db6c145acafc0f3068b8ce1fd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:48.456506 27557 raft_consensus.cc:2258] T 83b95f3ae7564b81b31a2f5673ee22bb P 6c194a6db6c145acafc0f3068b8ce1fd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:48.459647 27557 ts_tablet_manager.cc:1875] T 83b95f3ae7564b81b31a2f5673ee22bb P 6c194a6db6c145acafc0f3068b8ce1fd: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:27:48.459724 27234 raft_consensus.cc:979] T 89f86d58fe414b579845214534d338d8 P 6810a866decb4f5089f7d35c870e4204: : Instructing follower fb6004b8640a4906aa0cfe63e383d40e to start an election
I20240502 21:27:48.459889 27328 raft_consensus.cc:1067] T 89f86d58fe414b579845214534d338d8 P 6810a866decb4f5089f7d35c870e4204 [term 1 LEADER]: Signalling peer fb6004b8640a4906aa0cfe63e383d40e to start an election
I20240502 21:27:48.460736 26012 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "89f86d58fe414b579845214534d338d8"
dest_uuid: "fb6004b8640a4906aa0cfe63e383d40e"
from {username='slave'} at 127.15.255.129:59107
I20240502 21:27:48.461072 26012 raft_consensus.cc:482] T 89f86d58fe414b579845214534d338d8 P fb6004b8640a4906aa0cfe63e383d40e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:48.461238 26012 raft_consensus.cc:3037] T 89f86d58fe414b579845214534d338d8 P fb6004b8640a4906aa0cfe63e383d40e [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:48.464018 26012 raft_consensus.cc:504] T 89f86d58fe414b579845214534d338d8 P fb6004b8640a4906aa0cfe63e383d40e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } attrs { replace: true } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: false } }
I20240502 21:27:48.466537 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89f86d58fe414b579845214534d338d8" candidate_uuid: "fb6004b8640a4906aa0cfe63e383d40e" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6"
I20240502 21:27:48.466652 23915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89f86d58fe414b579845214534d338d8" candidate_uuid: "fb6004b8640a4906aa0cfe63e383d40e" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "6810a866decb4f5089f7d35c870e4204"
I20240502 21:27:48.466765 24317 raft_consensus.cc:3037] T 89f86d58fe414b579845214534d338d8 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:48.466840 23915 raft_consensus.cc:3032] T 89f86d58fe414b579845214534d338d8 P 6810a866decb4f5089f7d35c870e4204 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 21:27:48.466832 27557 ts_tablet_manager.cc:1888] T 83b95f3ae7564b81b31a2f5673ee22bb P 6c194a6db6c145acafc0f3068b8ce1fd: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 21:27:48.466949 27557 log.cc:1196] T 83b95f3ae7564b81b31a2f5673ee22bb P 6c194a6db6c145acafc0f3068b8ce1fd: Deleting WAL directory at /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-1/wal/wals/83b95f3ae7564b81b31a2f5673ee22bb
I20240502 21:27:48.466935 23915 raft_consensus.cc:729] T 89f86d58fe414b579845214534d338d8 P 6810a866decb4f5089f7d35c870e4204 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6810a866decb4f5089f7d35c870e4204, State: Running, Role: LEADER
I20240502 21:27:48.467231 23915 consensus_queue.cc:260] T 89f86d58fe414b579845214534d338d8 P 6810a866decb4f5089f7d35c870e4204 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } attrs { replace: true } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: false } }
I20240502 21:27:48.467677 23915 raft_consensus.cc:3037] T 89f86d58fe414b579845214534d338d8 P 6810a866decb4f5089f7d35c870e4204 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:48.469432 24317 raft_consensus.cc:2454] T 89f86d58fe414b579845214534d338d8 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb6004b8640a4906aa0cfe63e383d40e in term 2.
I20240502 21:27:48.470220 23915 raft_consensus.cc:2454] T 89f86d58fe414b579845214534d338d8 P 6810a866decb4f5089f7d35c870e4204 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb6004b8640a4906aa0cfe63e383d40e in term 2.
I20240502 21:27:48.470736 23757 catalog_manager.cc:4830] TS 6c194a6db6c145acafc0f3068b8ce1fd (127.15.255.130:41235): tablet 83b95f3ae7564b81b31a2f5673ee22bb (table rebalance_test_table_8 [id=9a4e7bd31db34cf3aff2242876f0de47]) successfully deleted
I20240502 21:27:48.471457 25947 leader_election.cc:302] T 89f86d58fe414b579845214534d338d8 P fb6004b8640a4906aa0cfe63e383d40e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 038c5d9d1d7a49c981c4c34fbc00d9e6, 6810a866decb4f5089f7d35c870e4204, fb6004b8640a4906aa0cfe63e383d40e; no voters:
I20240502 21:27:48.471866 27031 raft_consensus.cc:2781] T 89f86d58fe414b579845214534d338d8 P fb6004b8640a4906aa0cfe63e383d40e [term 2 FOLLOWER]: Leader election won for term 2
I20240502 21:27:48.474668 26013 raft_consensus.cc:1261] T fcc9cbeae6f343af9f84c73349f3dadb P fb6004b8640a4906aa0cfe63e383d40e [term 1 LEARNER]: Refusing update from remote peer 038c5d9d1d7a49c981c4c34fbc00d9e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:27:48.474769 24047 raft_consensus.cc:1261] T fcc9cbeae6f343af9f84c73349f3dadb P 6c194a6db6c145acafc0f3068b8ce1fd [term 1 FOLLOWER]: Refusing update from remote peer 038c5d9d1d7a49c981c4c34fbc00d9e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:27:48.475354 26420 consensus_queue.cc:1035] T fcc9cbeae6f343af9f84c73349f3dadb P 038c5d9d1d7a49c981c4c34fbc00d9e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:27:48.475906 26420 consensus_queue.cc:1035] T fcc9cbeae6f343af9f84c73349f3dadb P 038c5d9d1d7a49c981c4c34fbc00d9e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:27:48.476579 24317 consensus_queue.cc:237] T d5697d8b06094211b261f144de9d46f8 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } }
I20240502 21:27:48.479382 26279 raft_consensus.cc:1261] T d5697d8b06094211b261f144de9d46f8 P a3e500d7077a48acb5edb33f7e178c81 [term 1 FOLLOWER]: Refusing update from remote peer 038c5d9d1d7a49c981c4c34fbc00d9e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:27:48.479941 24185 raft_consensus.cc:1261] T d5697d8b06094211b261f144de9d46f8 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 1 FOLLOWER]: Refusing update from remote peer 038c5d9d1d7a49c981c4c34fbc00d9e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:27:48.480624 26420 consensus_queue.cc:1035] T d5697d8b06094211b261f144de9d46f8 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:27:48.481169 26420 consensus_queue.cc:1035] T d5697d8b06094211b261f144de9d46f8 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:27:48.483193 27563 ts_tablet_manager.cc:902] T 8af15fc41246471db31f7824985b5903 P fb6004b8640a4906aa0cfe63e383d40e: Initiating tablet copy from peer 6c194a6db6c145acafc0f3068b8ce1fd (127.15.255.130:41235)
I20240502 21:27:48.484879 27349 raft_consensus.cc:2932] T d5697d8b06094211b261f144de9d46f8 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 6c194a6db6c145acafc0f3068b8ce1fd (127.15.255.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } } }
I20240502 21:27:48.490087 23755 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet d5697d8b06094211b261f144de9d46f8 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:27:48.490078 24185 raft_consensus.cc:2932] T d5697d8b06094211b261f144de9d46f8 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 6c194a6db6c145acafc0f3068b8ce1fd (127.15.255.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } } }
I20240502 21:27:48.491297 23772 catalog_manager.cc:5484] T d5697d8b06094211b261f144de9d46f8 P 038c5d9d1d7a49c981c4c34fbc00d9e6 reported cstate change: config changed from index 5 to 6, VOTER 6c194a6db6c145acafc0f3068b8ce1fd (127.15.255.130) evicted. New cstate: current_term: 1 leader_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:48.493798 26279 raft_consensus.cc:2932] T d5697d8b06094211b261f144de9d46f8 P a3e500d7077a48acb5edb33f7e178c81 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 6c194a6db6c145acafc0f3068b8ce1fd (127.15.255.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } } }
I20240502 21:27:48.494863 27348 raft_consensus.cc:2932] T fcc9cbeae6f343af9f84c73349f3dadb P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, fb6004b8640a4906aa0cfe63e383d40e (127.15.255.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } attrs { replace: true } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: false } } }
I20240502 21:27:48.496815 24027 tablet_service.cc:1514] Processing DeleteTablet for tablet d5697d8b06094211b261f144de9d46f8 with delete_type TABLET_DATA_TOMBSTONED (TS 6c194a6db6c145acafc0f3068b8ce1fd not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:47386
I20240502 21:27:48.497179 27557 tablet_replica.cc:330] T d5697d8b06094211b261f144de9d46f8 P 6c194a6db6c145acafc0f3068b8ce1fd: stopping tablet replica
I20240502 21:27:48.497365 27557 raft_consensus.cc:2229] T d5697d8b06094211b261f144de9d46f8 P 6c194a6db6c145acafc0f3068b8ce1fd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:48.497589 27557 raft_consensus.cc:2258] T d5697d8b06094211b261f144de9d46f8 P 6c194a6db6c145acafc0f3068b8ce1fd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:48.498713 24046 raft_consensus.cc:2932] T fcc9cbeae6f343af9f84c73349f3dadb P 6c194a6db6c145acafc0f3068b8ce1fd [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, fb6004b8640a4906aa0cfe63e383d40e (127.15.255.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } attrs { replace: true } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: false } } }
I20240502 21:27:48.500097 26011 raft_consensus.cc:2932] T fcc9cbeae6f343af9f84c73349f3dadb P fb6004b8640a4906aa0cfe63e383d40e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, fb6004b8640a4906aa0cfe63e383d40e (127.15.255.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } attrs { replace: true } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: false } } }
I20240502 21:27:48.504106 23772 catalog_manager.cc:5484] T fcc9cbeae6f343af9f84c73349f3dadb P 038c5d9d1d7a49c981c4c34fbc00d9e6 reported cstate change: config changed from index 4 to 5, fb6004b8640a4906aa0cfe63e383d40e (127.15.255.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:27:48.511950 24183 raft_consensus.cc:1261] T fcc9cbeae6f343af9f84c73349f3dadb P 009d3e7e8a8b4e2497de3a73d01ced49 [term 1 FOLLOWER]: Refusing update from remote peer 038c5d9d1d7a49c981c4c34fbc00d9e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:27:48.512603 26560 consensus_queue.cc:1035] T fcc9cbeae6f343af9f84c73349f3dadb P 038c5d9d1d7a49c981c4c34fbc00d9e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:27:48.516252 26012 leader_election.cc:288] T 89f86d58fe414b579845214534d338d8 P fb6004b8640a4906aa0cfe63e383d40e [CANDIDATE]: Term 2 election: Requested vote from peers 038c5d9d1d7a49c981c4c34fbc00d9e6 (127.15.255.132:46813), 6810a866decb4f5089f7d35c870e4204 (127.15.255.129:40435), 009d3e7e8a8b4e2497de3a73d01ced49 (127.15.255.131:37895)
I20240502 21:27:48.516942 24185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89f86d58fe414b579845214534d338d8" candidate_uuid: "fb6004b8640a4906aa0cfe63e383d40e" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "009d3e7e8a8b4e2497de3a73d01ced49"
I20240502 21:27:48.517160 24185 raft_consensus.cc:3037] T 89f86d58fe414b579845214534d338d8 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:48.521620 24183 raft_consensus.cc:2932] T fcc9cbeae6f343af9f84c73349f3dadb P 009d3e7e8a8b4e2497de3a73d01ced49 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, fb6004b8640a4906aa0cfe63e383d40e (127.15.255.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } attrs { replace: true } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: false } } }
I20240502 21:27:48.527639 27573 ts_tablet_manager.cc:902] T 52f621ea3f174490948eae854c706a92 P ee27c3f9e0ec46caa742ac386d6b7c20: Initiating tablet copy from peer 038c5d9d1d7a49c981c4c34fbc00d9e6 (127.15.255.132:46813)
I20240502 21:27:48.519882 24185 raft_consensus.cc:2454] T 89f86d58fe414b579845214534d338d8 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb6004b8640a4906aa0cfe63e383d40e in term 2.
I20240502 21:27:48.535826 27557 ts_tablet_manager.cc:1875] T d5697d8b06094211b261f144de9d46f8 P 6c194a6db6c145acafc0f3068b8ce1fd: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:27:48.542181 27557 ts_tablet_manager.cc:1888] T d5697d8b06094211b261f144de9d46f8 P 6c194a6db6c145acafc0f3068b8ce1fd: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 21:27:48.542305 27557 log.cc:1196] T d5697d8b06094211b261f144de9d46f8 P 6c194a6db6c145acafc0f3068b8ce1fd: Deleting WAL directory at /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-1/wal/wals/d5697d8b06094211b261f144de9d46f8
I20240502 21:27:48.543030 23757 catalog_manager.cc:4830] TS 6c194a6db6c145acafc0f3068b8ce1fd (127.15.255.130:41235): tablet d5697d8b06094211b261f144de9d46f8 (table rebalance_test_table_9 [id=188ff51c46164f47803c423463c2e6e2]) successfully deleted
I20240502 21:27:48.546260 27566 ts_tablet_manager.cc:902] T fef55863beb6435dbb25bfb6d71e3cd7 P a3e500d7077a48acb5edb33f7e178c81: Initiating tablet copy from peer 038c5d9d1d7a49c981c4c34fbc00d9e6 (127.15.255.132:46813)
I20240502 21:27:48.548453 27031 raft_consensus.cc:686] T 89f86d58fe414b579845214534d338d8 P fb6004b8640a4906aa0cfe63e383d40e [term 2 LEADER]: Becoming Leader. State: Replica: fb6004b8640a4906aa0cfe63e383d40e, State: Running, Role: LEADER
I20240502 21:27:48.549064 27031 consensus_queue.cc:237] T 89f86d58fe414b579845214534d338d8 P fb6004b8640a4906aa0cfe63e383d40e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } attrs { replace: true } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: false } }
I20240502 21:27:48.554267 23770 catalog_manager.cc:5484] T 89f86d58fe414b579845214534d338d8 P fb6004b8640a4906aa0cfe63e383d40e reported cstate change: term changed from 1 to 2, leader changed from 6810a866decb4f5089f7d35c870e4204 (127.15.255.129) to fb6004b8640a4906aa0cfe63e383d40e (127.15.255.134). New cstate: current_term: 2 leader_uuid: "fb6004b8640a4906aa0cfe63e383d40e" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:27:48.561050 27566 tablet_copy_client.cc:308] T fef55863beb6435dbb25bfb6d71e3cd7 P a3e500d7077a48acb5edb33f7e178c81: tablet copy: Beginning tablet copy session from remote peer at address 127.15.255.132:46813
I20240502 21:27:48.561779 24338 tablet_copy_service.cc:140] P 038c5d9d1d7a49c981c4c34fbc00d9e6: Received BeginTabletCopySession request for tablet fef55863beb6435dbb25bfb6d71e3cd7 from peer a3e500d7077a48acb5edb33f7e178c81 ({username='slave'} at 127.15.255.136:38895)
I20240502 21:27:48.561981 24338 tablet_copy_service.cc:161] P 038c5d9d1d7a49c981c4c34fbc00d9e6: Beginning new tablet copy session on tablet fef55863beb6435dbb25bfb6d71e3cd7 from peer a3e500d7077a48acb5edb33f7e178c81 at {username='slave'} at 127.15.255.136:38895: session id = a3e500d7077a48acb5edb33f7e178c81-fef55863beb6435dbb25bfb6d71e3cd7
I20240502 21:27:48.565290 24338 tablet_copy_source_session.cc:205] T fef55863beb6435dbb25bfb6d71e3cd7 P 038c5d9d1d7a49c981c4c34fbc00d9e6: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:27:48.566785 27566 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fef55863beb6435dbb25bfb6d71e3cd7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:48.569139 27563 tablet_copy_client.cc:308] T 8af15fc41246471db31f7824985b5903 P fb6004b8640a4906aa0cfe63e383d40e: tablet copy: Beginning tablet copy session from remote peer at address 127.15.255.130:41235
I20240502 21:27:48.578051 27566 tablet_copy_client.cc:783] T fef55863beb6435dbb25bfb6d71e3cd7 P a3e500d7077a48acb5edb33f7e178c81: tablet copy: Starting download of 0 data blocks...
I20240502 21:27:48.578425 27566 tablet_copy_client.cc:647] T fef55863beb6435dbb25bfb6d71e3cd7 P a3e500d7077a48acb5edb33f7e178c81: tablet copy: Starting download of 1 WAL segments...
I20240502 21:27:48.580102 27581 tablet_copy_client.cc:1002] T fef55863beb6435dbb25bfb6d71e3cd7 P a3e500d7077a48acb5edb33f7e178c81: tablet copy: Injecting latency into file download: 1500
I20240502 21:27:48.584939 27573 tablet_copy_client.cc:308] T 52f621ea3f174490948eae854c706a92 P ee27c3f9e0ec46caa742ac386d6b7c20: tablet copy: Beginning tablet copy session from remote peer at address 127.15.255.132:46813
I20240502 21:27:48.585693 24338 tablet_copy_service.cc:140] P 038c5d9d1d7a49c981c4c34fbc00d9e6: Received BeginTabletCopySession request for tablet 52f621ea3f174490948eae854c706a92 from peer ee27c3f9e0ec46caa742ac386d6b7c20 ({username='slave'} at 127.15.255.133:45805)
I20240502 21:27:48.585844 24338 tablet_copy_service.cc:161] P 038c5d9d1d7a49c981c4c34fbc00d9e6: Beginning new tablet copy session on tablet 52f621ea3f174490948eae854c706a92 from peer ee27c3f9e0ec46caa742ac386d6b7c20 at {username='slave'} at 127.15.255.133:45805: session id = ee27c3f9e0ec46caa742ac386d6b7c20-52f621ea3f174490948eae854c706a92
I20240502 21:27:48.588559 27584 ts_tablet_manager.cc:902] T 30236241b670443489240f191d9b9d50 P fb6004b8640a4906aa0cfe63e383d40e: Initiating tablet copy from peer 6810a866decb4f5089f7d35c870e4204 (127.15.255.129:40435)
I20240502 21:27:48.589272 24338 tablet_copy_source_session.cc:205] T 52f621ea3f174490948eae854c706a92 P 038c5d9d1d7a49c981c4c34fbc00d9e6: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:27:48.590595 27573 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 52f621ea3f174490948eae854c706a92. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:48.593024 27292 raft_consensus.cc:979] T f84bbe304597450c978ae28e0c4e7a3e P 009d3e7e8a8b4e2497de3a73d01ced49: : Instructing follower a3e500d7077a48acb5edb33f7e178c81 to start an election
I20240502 21:27:48.593202 27292 raft_consensus.cc:1067] T f84bbe304597450c978ae28e0c4e7a3e P 009d3e7e8a8b4e2497de3a73d01ced49 [term 1 LEADER]: Signalling peer a3e500d7077a48acb5edb33f7e178c81 to start an election
I20240502 21:27:48.594190 26280 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f84bbe304597450c978ae28e0c4e7a3e"
dest_uuid: "a3e500d7077a48acb5edb33f7e178c81"
from {username='slave'} at 127.15.255.131:56011
I20240502 21:27:48.594458 26280 raft_consensus.cc:482] T f84bbe304597450c978ae28e0c4e7a3e P a3e500d7077a48acb5edb33f7e178c81 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:48.594640 26280 raft_consensus.cc:3037] T f84bbe304597450c978ae28e0c4e7a3e P a3e500d7077a48acb5edb33f7e178c81 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:48.600394 26280 raft_consensus.cc:504] T f84bbe304597450c978ae28e0c4e7a3e P a3e500d7077a48acb5edb33f7e178c81 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } attrs { replace: true } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } }
I20240502 21:27:48.602712 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f84bbe304597450c978ae28e0c4e7a3e" candidate_uuid: "a3e500d7077a48acb5edb33f7e178c81" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6"
I20240502 21:27:48.602932 24317 raft_consensus.cc:3037] T f84bbe304597450c978ae28e0c4e7a3e P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:48.603572 27573 tablet_copy_client.cc:783] T 52f621ea3f174490948eae854c706a92 P ee27c3f9e0ec46caa742ac386d6b7c20: tablet copy: Starting download of 0 data blocks...
I20240502 21:27:48.603821 27573 tablet_copy_client.cc:647] T 52f621ea3f174490948eae854c706a92 P ee27c3f9e0ec46caa742ac386d6b7c20: tablet copy: Starting download of 1 WAL segments...
I20240502 21:27:48.605181 27593 ts_tablet_manager.cc:902] T dc0e070513cc46ca82c1946423ad91c5 P e117afbf8819424e8a98476453dfde2a: Initiating tablet copy from peer 6c194a6db6c145acafc0f3068b8ce1fd (127.15.255.130:41235)
I20240502 21:27:48.605598 24317 raft_consensus.cc:2454] T f84bbe304597450c978ae28e0c4e7a3e P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a3e500d7077a48acb5edb33f7e178c81 in term 2.
I20240502 21:27:48.605885 27587 tablet_copy_client.cc:1002] T 52f621ea3f174490948eae854c706a92 P ee27c3f9e0ec46caa742ac386d6b7c20: tablet copy: Injecting latency into file download: 1500
I20240502 21:27:48.608022 26280 leader_election.cc:288] T f84bbe304597450c978ae28e0c4e7a3e P a3e500d7077a48acb5edb33f7e178c81 [CANDIDATE]: Term 2 election: Requested vote from peers 038c5d9d1d7a49c981c4c34fbc00d9e6 (127.15.255.132:46813), 6c194a6db6c145acafc0f3068b8ce1fd (127.15.255.130:41235), 009d3e7e8a8b4e2497de3a73d01ced49 (127.15.255.131:37895)
I20240502 21:27:48.608711 24185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f84bbe304597450c978ae28e0c4e7a3e" candidate_uuid: "a3e500d7077a48acb5edb33f7e178c81" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "009d3e7e8a8b4e2497de3a73d01ced49"
I20240502 21:27:48.608918 24185 raft_consensus.cc:3032] T f84bbe304597450c978ae28e0c4e7a3e P 009d3e7e8a8b4e2497de3a73d01ced49 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 21:27:48.609025 24185 raft_consensus.cc:729] T f84bbe304597450c978ae28e0c4e7a3e P 009d3e7e8a8b4e2497de3a73d01ced49 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 009d3e7e8a8b4e2497de3a73d01ced49, State: Running, Role: LEADER
I20240502 21:27:48.609130 24070 tablet_copy_service.cc:140] P 6c194a6db6c145acafc0f3068b8ce1fd: Received BeginTabletCopySession request for tablet 8af15fc41246471db31f7824985b5903 from peer fb6004b8640a4906aa0cfe63e383d40e ({username='slave'} at 127.15.255.134:44929)
I20240502 21:27:48.609319 24070 tablet_copy_service.cc:161] P 6c194a6db6c145acafc0f3068b8ce1fd: Beginning new tablet copy session on tablet 8af15fc41246471db31f7824985b5903 from peer fb6004b8640a4906aa0cfe63e383d40e at {username='slave'} at 127.15.255.134:44929: session id = fb6004b8640a4906aa0cfe63e383d40e-8af15fc41246471db31f7824985b5903
I20240502 21:27:48.609352 24185 consensus_queue.cc:260] T f84bbe304597450c978ae28e0c4e7a3e P 009d3e7e8a8b4e2497de3a73d01ced49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } attrs { replace: true } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } }
I20240502 21:27:48.609817 24185 raft_consensus.cc:3037] T f84bbe304597450c978ae28e0c4e7a3e P 009d3e7e8a8b4e2497de3a73d01ced49 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:48.612468 24070 tablet_copy_source_session.cc:205] T 8af15fc41246471db31f7824985b5903 P 6c194a6db6c145acafc0f3068b8ce1fd: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:27:48.613246 24185 raft_consensus.cc:2454] T f84bbe304597450c978ae28e0c4e7a3e P 009d3e7e8a8b4e2497de3a73d01ced49 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a3e500d7077a48acb5edb33f7e178c81 in term 2.
I20240502 21:27:48.613916 26213 leader_election.cc:302] T f84bbe304597450c978ae28e0c4e7a3e P a3e500d7077a48acb5edb33f7e178c81 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 009d3e7e8a8b4e2497de3a73d01ced49, 038c5d9d1d7a49c981c4c34fbc00d9e6, a3e500d7077a48acb5edb33f7e178c81; no voters:
I20240502 21:27:48.614580 27045 raft_consensus.cc:2781] T f84bbe304597450c978ae28e0c4e7a3e P a3e500d7077a48acb5edb33f7e178c81 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 21:27:48.615550 27563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8af15fc41246471db31f7824985b5903. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:48.617357 27584 tablet_copy_client.cc:308] T 30236241b670443489240f191d9b9d50 P fb6004b8640a4906aa0cfe63e383d40e: tablet copy: Beginning tablet copy session from remote peer at address 127.15.255.129:40435
I20240502 21:27:48.618484 23935 tablet_copy_service.cc:140] P 6810a866decb4f5089f7d35c870e4204: Received BeginTabletCopySession request for tablet 30236241b670443489240f191d9b9d50 from peer fb6004b8640a4906aa0cfe63e383d40e ({username='slave'} at 127.15.255.134:44215)
I20240502 21:27:48.618661 23935 tablet_copy_service.cc:161] P 6810a866decb4f5089f7d35c870e4204: Beginning new tablet copy session on tablet 30236241b670443489240f191d9b9d50 from peer fb6004b8640a4906aa0cfe63e383d40e at {username='slave'} at 127.15.255.134:44215: session id = fb6004b8640a4906aa0cfe63e383d40e-30236241b670443489240f191d9b9d50
I20240502 21:27:48.620270 24317 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "fcc9cbeae6f343af9f84c73349f3dadb"
dest_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:52418
I20240502 21:27:48.620559 24317 raft_consensus.cc:595] T fcc9cbeae6f343af9f84c73349f3dadb P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 1 LEADER]: Received request to transfer leadership
I20240502 21:27:48.621412 23935 tablet_copy_source_session.cc:205] T 30236241b670443489240f191d9b9d50 P 6810a866decb4f5089f7d35c870e4204: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:27:48.623983 27045 raft_consensus.cc:686] T f84bbe304597450c978ae28e0c4e7a3e P a3e500d7077a48acb5edb33f7e178c81 [term 2 LEADER]: Becoming Leader. State: Replica: a3e500d7077a48acb5edb33f7e178c81, State: Running, Role: LEADER
I20240502 21:27:48.624500 27584 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30236241b670443489240f191d9b9d50. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:48.624516 27045 consensus_queue.cc:237] T f84bbe304597450c978ae28e0c4e7a3e P a3e500d7077a48acb5edb33f7e178c81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } attrs { replace: true } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } }
I20240502 21:27:48.631803 27584 tablet_copy_client.cc:783] T 30236241b670443489240f191d9b9d50 P fb6004b8640a4906aa0cfe63e383d40e: tablet copy: Starting download of 0 data blocks...
I20240502 21:27:48.632086 27584 tablet_copy_client.cc:647] T 30236241b670443489240f191d9b9d50 P fb6004b8640a4906aa0cfe63e383d40e: tablet copy: Starting download of 1 WAL segments...
I20240502 21:27:48.631911 23772 catalog_manager.cc:5484] T f84bbe304597450c978ae28e0c4e7a3e P a3e500d7077a48acb5edb33f7e178c81 reported cstate change: term changed from 1 to 2, leader changed from 009d3e7e8a8b4e2497de3a73d01ced49 (127.15.255.131) to a3e500d7077a48acb5edb33f7e178c81 (127.15.255.136). New cstate: current_term: 2 leader_uuid: "a3e500d7077a48acb5edb33f7e178c81" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:27:48.632253 27563 tablet_copy_client.cc:783] T 8af15fc41246471db31f7824985b5903 P fb6004b8640a4906aa0cfe63e383d40e: tablet copy: Starting download of 0 data blocks...
I20240502 21:27:48.632656 27563 tablet_copy_client.cc:647] T 8af15fc41246471db31f7824985b5903 P fb6004b8640a4906aa0cfe63e383d40e: tablet copy: Starting download of 1 WAL segments...
I20240502 21:27:48.635345 27593 tablet_copy_client.cc:308] T dc0e070513cc46ca82c1946423ad91c5 P e117afbf8819424e8a98476453dfde2a: tablet copy: Beginning tablet copy session from remote peer at address 127.15.255.130:41235
I20240502 21:27:48.636561 27598 tablet_copy_client.cc:1002] T 30236241b670443489240f191d9b9d50 P fb6004b8640a4906aa0cfe63e383d40e: tablet copy: Injecting latency into file download: 1500
I20240502 21:27:48.636847 27582 tablet_copy_client.cc:1002] T 8af15fc41246471db31f7824985b5903 P fb6004b8640a4906aa0cfe63e383d40e: tablet copy: Injecting latency into file download: 1500
I20240502 21:27:48.637032 24070 tablet_copy_service.cc:140] P 6c194a6db6c145acafc0f3068b8ce1fd: Received BeginTabletCopySession request for tablet dc0e070513cc46ca82c1946423ad91c5 from peer e117afbf8819424e8a98476453dfde2a ({username='slave'} at 127.15.255.135:53383)
I20240502 21:27:48.637200 24070 tablet_copy_service.cc:161] P 6c194a6db6c145acafc0f3068b8ce1fd: Beginning new tablet copy session on tablet dc0e070513cc46ca82c1946423ad91c5 from peer e117afbf8819424e8a98476453dfde2a at {username='slave'} at 127.15.255.135:53383: session id = e117afbf8819424e8a98476453dfde2a-dc0e070513cc46ca82c1946423ad91c5
I20240502 21:27:48.639328 24070 tablet_copy_source_session.cc:205] T dc0e070513cc46ca82c1946423ad91c5 P 6c194a6db6c145acafc0f3068b8ce1fd: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:27:48.639942 24046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f84bbe304597450c978ae28e0c4e7a3e" candidate_uuid: "a3e500d7077a48acb5edb33f7e178c81" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "6c194a6db6c145acafc0f3068b8ce1fd"
I20240502 21:27:48.640123 24046 raft_consensus.cc:3037] T f84bbe304597450c978ae28e0c4e7a3e P 6c194a6db6c145acafc0f3068b8ce1fd [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:48.640821 27593 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc0e070513cc46ca82c1946423ad91c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:48.642328 24046 raft_consensus.cc:2454] T f84bbe304597450c978ae28e0c4e7a3e P 6c194a6db6c145acafc0f3068b8ce1fd [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a3e500d7077a48acb5edb33f7e178c81 in term 2.
I20240502 21:27:48.645483 27593 tablet_copy_client.cc:783] T dc0e070513cc46ca82c1946423ad91c5 P e117afbf8819424e8a98476453dfde2a: tablet copy: Starting download of 0 data blocks...
I20240502 21:27:48.645753 27593 tablet_copy_client.cc:647] T dc0e070513cc46ca82c1946423ad91c5 P e117afbf8819424e8a98476453dfde2a: tablet copy: Starting download of 1 WAL segments...
I20240502 21:27:48.647408 27601 tablet_copy_client.cc:1002] T dc0e070513cc46ca82c1946423ad91c5 P e117afbf8819424e8a98476453dfde2a: tablet copy: Injecting latency into file download: 1500
I20240502 21:27:48.687961 24185 consensus_queue.cc:237] T d79eec4f59634df480161815d888bcac P 009d3e7e8a8b4e2497de3a73d01ced49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } attrs { replace: true } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: NON_VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: true } }
I20240502 21:27:48.690268 24046 raft_consensus.cc:1261] T d79eec4f59634df480161815d888bcac P 6c194a6db6c145acafc0f3068b8ce1fd [term 1 FOLLOWER]: Refusing update from remote peer 009d3e7e8a8b4e2497de3a73d01ced49: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 21:27:48.690430 24115 consensus_peers.cc:485] T d79eec4f59634df480161815d888bcac P 009d3e7e8a8b4e2497de3a73d01ced49 -> Peer ee27c3f9e0ec46caa742ac386d6b7c20 (127.15.255.133:40841): Couldn't send request to peer ee27c3f9e0ec46caa742ac386d6b7c20. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d79eec4f59634df480161815d888bcac. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:48.690435 24317 raft_consensus.cc:1261] T d79eec4f59634df480161815d888bcac P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 1 FOLLOWER]: Refusing update from remote peer 009d3e7e8a8b4e2497de3a73d01ced49: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:27:48.690883 26475 consensus_queue.cc:1035] T d79eec4f59634df480161815d888bcac P 009d3e7e8a8b4e2497de3a73d01ced49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:27:48.691139 27292 consensus_queue.cc:1035] T d79eec4f59634df480161815d888bcac P 009d3e7e8a8b4e2497de3a73d01ced49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:27:48.693495 26554 raft_consensus.cc:2932] T d79eec4f59634df480161815d888bcac P 009d3e7e8a8b4e2497de3a73d01ced49 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ee27c3f9e0ec46caa742ac386d6b7c20 (127.15.255.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } attrs { replace: true } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: NON_VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: true } } }
I20240502 21:27:48.694105 24046 raft_consensus.cc:2932] T d79eec4f59634df480161815d888bcac P 6c194a6db6c145acafc0f3068b8ce1fd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ee27c3f9e0ec46caa742ac386d6b7c20 (127.15.255.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } attrs { replace: true } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: NON_VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: true } } }
I20240502 21:27:48.694126 24317 raft_consensus.cc:2932] T d79eec4f59634df480161815d888bcac P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ee27c3f9e0ec46caa742ac386d6b7c20 (127.15.255.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } attrs { replace: true } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: NON_VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: true } } }
I20240502 21:27:48.698597 23772 catalog_manager.cc:5484] T d79eec4f59634df480161815d888bcac P 009d3e7e8a8b4e2497de3a73d01ced49 reported cstate change: config changed from index -1 to 4, NON_VOTER ee27c3f9e0ec46caa742ac386d6b7c20 (127.15.255.133) added. New cstate: current_term: 1 leader_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: NON_VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:48.711658 24046 consensus_queue.cc:237] T 77fc7226878f44d3884ccb1fc41c44a4 P 6c194a6db6c145acafc0f3068b8ce1fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } attrs { replace: true } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: NON_VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: true } }
I20240502 21:27:48.714275 24317 raft_consensus.cc:1261] T 77fc7226878f44d3884ccb1fc41c44a4 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 1 FOLLOWER]: Refusing update from remote peer 6c194a6db6c145acafc0f3068b8ce1fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:27:48.714515 23915 raft_consensus.cc:1261] T 77fc7226878f44d3884ccb1fc41c44a4 P 6810a866decb4f5089f7d35c870e4204 [term 1 FOLLOWER]: Refusing update from remote peer 6c194a6db6c145acafc0f3068b8ce1fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 21:27:48.714965 23982 consensus_peers.cc:485] T 77fc7226878f44d3884ccb1fc41c44a4 P 6c194a6db6c145acafc0f3068b8ce1fd -> Peer e117afbf8819424e8a98476453dfde2a (127.15.255.135:32775): Couldn't send request to peer e117afbf8819424e8a98476453dfde2a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 77fc7226878f44d3884ccb1fc41c44a4. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:48.714988 27449 consensus_queue.cc:1035] T 77fc7226878f44d3884ccb1fc41c44a4 P 6c194a6db6c145acafc0f3068b8ce1fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:27:48.715394 27457 consensus_queue.cc:1035] T 77fc7226878f44d3884ccb1fc41c44a4 P 6c194a6db6c145acafc0f3068b8ce1fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:27:48.717792 27449 raft_consensus.cc:2932] T 77fc7226878f44d3884ccb1fc41c44a4 P 6c194a6db6c145acafc0f3068b8ce1fd [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER e117afbf8819424e8a98476453dfde2a (127.15.255.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } attrs { replace: true } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: NON_VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: true } } }
I20240502 21:27:48.718508 24317 raft_consensus.cc:2932] T 77fc7226878f44d3884ccb1fc41c44a4 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER e117afbf8819424e8a98476453dfde2a (127.15.255.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } attrs { replace: true } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: NON_VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: true } } }
I20240502 21:27:48.723238 23770 catalog_manager.cc:5484] T 77fc7226878f44d3884ccb1fc41c44a4 P 6c194a6db6c145acafc0f3068b8ce1fd reported cstate change: config changed from index -1 to 4, NON_VOTER e117afbf8819424e8a98476453dfde2a (127.15.255.135) added. New cstate: current_term: 1 leader_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: NON_VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:48.735330 23915 raft_consensus.cc:2932] T 77fc7226878f44d3884ccb1fc41c44a4 P 6810a866decb4f5089f7d35c870e4204 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER e117afbf8819424e8a98476453dfde2a (127.15.255.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } attrs { replace: true } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: NON_VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: true } } }
I20240502 21:27:48.744036 24317 consensus_queue.cc:237] T 8eba3872a77046ecb9c9477d4cc0e18b P 038c5d9d1d7a49c981c4c34fbc00d9e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } attrs { replace: true } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: NON_VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: true } }
I20240502 21:27:48.746392 24046 raft_consensus.cc:1261] T 8eba3872a77046ecb9c9477d4cc0e18b P 6c194a6db6c145acafc0f3068b8ce1fd [term 1 FOLLOWER]: Refusing update from remote peer 038c5d9d1d7a49c981c4c34fbc00d9e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:27:48.746454 23915 raft_consensus.cc:1261] T 8eba3872a77046ecb9c9477d4cc0e18b P 6810a866decb4f5089f7d35c870e4204 [term 1 FOLLOWER]: Refusing update from remote peer 038c5d9d1d7a49c981c4c34fbc00d9e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 21:27:48.746670 24250 consensus_peers.cc:485] T 8eba3872a77046ecb9c9477d4cc0e18b P 038c5d9d1d7a49c981c4c34fbc00d9e6 -> Peer ee27c3f9e0ec46caa742ac386d6b7c20 (127.15.255.133:40841): Couldn't send request to peer ee27c3f9e0ec46caa742ac386d6b7c20. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8eba3872a77046ecb9c9477d4cc0e18b. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:48.747052 26560 consensus_queue.cc:1035] T 8eba3872a77046ecb9c9477d4cc0e18b P 038c5d9d1d7a49c981c4c34fbc00d9e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:27:48.747295 27348 consensus_queue.cc:1035] T 8eba3872a77046ecb9c9477d4cc0e18b P 038c5d9d1d7a49c981c4c34fbc00d9e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:27:48.749892 27349 raft_consensus.cc:2932] T 8eba3872a77046ecb9c9477d4cc0e18b P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ee27c3f9e0ec46caa742ac386d6b7c20 (127.15.255.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } attrs { replace: true } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: NON_VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: true } } }
I20240502 21:27:48.750663 24046 raft_consensus.cc:2932] T 8eba3872a77046ecb9c9477d4cc0e18b P 6c194a6db6c145acafc0f3068b8ce1fd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ee27c3f9e0ec46caa742ac386d6b7c20 (127.15.255.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } attrs { replace: true } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: NON_VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: true } } }
I20240502 21:27:48.750854 23915 raft_consensus.cc:2932] T 8eba3872a77046ecb9c9477d4cc0e18b P 6810a866decb4f5089f7d35c870e4204 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ee27c3f9e0ec46caa742ac386d6b7c20 (127.15.255.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } attrs { replace: true } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: NON_VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: true } } }
I20240502 21:27:48.756712 23772 catalog_manager.cc:5484] T 8eba3872a77046ecb9c9477d4cc0e18b P 6c194a6db6c145acafc0f3068b8ce1fd reported cstate change: config changed from index -1 to 4, NON_VOTER ee27c3f9e0ec46caa742ac386d6b7c20 (127.15.255.133) added. New cstate: current_term: 1 leader_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } attrs { replace: true } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: NON_VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: true } } }
I20240502 21:27:48.769487 24046 consensus_queue.cc:237] T c9b5be01b5e941b1848df79c7a9db348 P 6c194a6db6c145acafc0f3068b8ce1fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } attrs { replace: true } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: NON_VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: true } }
I20240502 21:27:48.771418 24317 raft_consensus.cc:1261] T c9b5be01b5e941b1848df79c7a9db348 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 1 FOLLOWER]: Refusing update from remote peer 6c194a6db6c145acafc0f3068b8ce1fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:27:48.771498 23915 raft_consensus.cc:1261] T c9b5be01b5e941b1848df79c7a9db348 P 6810a866decb4f5089f7d35c870e4204 [term 1 FOLLOWER]: Refusing update from remote peer 6c194a6db6c145acafc0f3068b8ce1fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:27:48.771898 27456 consensus_queue.cc:1035] T c9b5be01b5e941b1848df79c7a9db348 P 6c194a6db6c145acafc0f3068b8ce1fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:27:48.772215 27449 consensus_queue.cc:1035] T c9b5be01b5e941b1848df79c7a9db348 P 6c194a6db6c145acafc0f3068b8ce1fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 21:27:48.772544 23982 consensus_peers.cc:485] T c9b5be01b5e941b1848df79c7a9db348 P 6c194a6db6c145acafc0f3068b8ce1fd -> Peer e117afbf8819424e8a98476453dfde2a (127.15.255.135:32775): Couldn't send request to peer e117afbf8819424e8a98476453dfde2a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c9b5be01b5e941b1848df79c7a9db348. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:48.774682 27449 raft_consensus.cc:2932] T c9b5be01b5e941b1848df79c7a9db348 P 6c194a6db6c145acafc0f3068b8ce1fd [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER e117afbf8819424e8a98476453dfde2a (127.15.255.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } attrs { replace: true } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: NON_VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: true } } }
I20240502 21:27:48.775543 24317 raft_consensus.cc:2932] T c9b5be01b5e941b1848df79c7a9db348 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER e117afbf8819424e8a98476453dfde2a (127.15.255.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } attrs { replace: true } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: NON_VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: true } } }
I20240502 21:27:48.775646 23915 raft_consensus.cc:2932] T c9b5be01b5e941b1848df79c7a9db348 P 6810a866decb4f5089f7d35c870e4204 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER e117afbf8819424e8a98476453dfde2a (127.15.255.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } attrs { replace: true } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: NON_VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: true } } }
I20240502 21:27:48.780282 23772 catalog_manager.cc:5484] T c9b5be01b5e941b1848df79c7a9db348 P 038c5d9d1d7a49c981c4c34fbc00d9e6 reported cstate change: config changed from index -1 to 4, NON_VOTER e117afbf8819424e8a98476453dfde2a (127.15.255.135) added. New cstate: current_term: 1 leader_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } attrs { replace: true } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: NON_VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: true } } }
I20240502 21:27:48.906644 24317 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "fcc9cbeae6f343af9f84c73349f3dadb"
dest_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:37706
I20240502 21:27:48.906858 24317 raft_consensus.cc:595] T fcc9cbeae6f343af9f84c73349f3dadb P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 1 LEADER]: Received request to transfer leadership
I20240502 21:27:48.920168 26420 raft_consensus.cc:979] T fcc9cbeae6f343af9f84c73349f3dadb P 038c5d9d1d7a49c981c4c34fbc00d9e6: : Instructing follower 009d3e7e8a8b4e2497de3a73d01ced49 to start an election
I20240502 21:27:48.920379 26420 raft_consensus.cc:1067] T fcc9cbeae6f343af9f84c73349f3dadb P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 1 LEADER]: Signalling peer 009d3e7e8a8b4e2497de3a73d01ced49 to start an election
I20240502 21:27:48.922410 24185 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "fcc9cbeae6f343af9f84c73349f3dadb"
dest_uuid: "009d3e7e8a8b4e2497de3a73d01ced49"
from {username='slave'} at 127.15.255.132:39569
I20240502 21:27:48.922652 24185 raft_consensus.cc:482] T fcc9cbeae6f343af9f84c73349f3dadb P 009d3e7e8a8b4e2497de3a73d01ced49 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:48.922749 24185 raft_consensus.cc:3037] T fcc9cbeae6f343af9f84c73349f3dadb P 009d3e7e8a8b4e2497de3a73d01ced49 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:48.924722 24185 raft_consensus.cc:504] T fcc9cbeae6f343af9f84c73349f3dadb P 009d3e7e8a8b4e2497de3a73d01ced49 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } attrs { replace: true } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: false } }
I20240502 21:27:48.925644 24185 leader_election.cc:288] T fcc9cbeae6f343af9f84c73349f3dadb P 009d3e7e8a8b4e2497de3a73d01ced49 [CANDIDATE]: Term 2 election: Requested vote from peers 6c194a6db6c145acafc0f3068b8ce1fd (127.15.255.130:41235), 038c5d9d1d7a49c981c4c34fbc00d9e6 (127.15.255.132:46813), fb6004b8640a4906aa0cfe63e383d40e (127.15.255.134:32825)
I20240502 21:27:48.925868 24046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcc9cbeae6f343af9f84c73349f3dadb" candidate_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "6c194a6db6c145acafc0f3068b8ce1fd"
I20240502 21:27:48.926132 24046 raft_consensus.cc:3037] T fcc9cbeae6f343af9f84c73349f3dadb P 6c194a6db6c145acafc0f3068b8ce1fd [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:48.926101 26011 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcc9cbeae6f343af9f84c73349f3dadb" candidate_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "fb6004b8640a4906aa0cfe63e383d40e"
I20240502 21:27:48.926334 26011 raft_consensus.cc:3037] T fcc9cbeae6f343af9f84c73349f3dadb P fb6004b8640a4906aa0cfe63e383d40e [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:48.928329 26011 raft_consensus.cc:2454] T fcc9cbeae6f343af9f84c73349f3dadb P fb6004b8640a4906aa0cfe63e383d40e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 009d3e7e8a8b4e2497de3a73d01ced49 in term 2.
I20240502 21:27:48.929137 24046 raft_consensus.cc:2454] T fcc9cbeae6f343af9f84c73349f3dadb P 6c194a6db6c145acafc0f3068b8ce1fd [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 009d3e7e8a8b4e2497de3a73d01ced49 in term 2.
I20240502 21:27:48.929318 24317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcc9cbeae6f343af9f84c73349f3dadb" candidate_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6"
I20240502 21:27:48.929559 24317 raft_consensus.cc:3032] T fcc9cbeae6f343af9f84c73349f3dadb P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 21:27:48.929656 24317 raft_consensus.cc:729] T fcc9cbeae6f343af9f84c73349f3dadb P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 038c5d9d1d7a49c981c4c34fbc00d9e6, State: Running, Role: LEADER
I20240502 21:27:48.930007 24317 consensus_queue.cc:260] T fcc9cbeae6f343af9f84c73349f3dadb P 038c5d9d1d7a49c981c4c34fbc00d9e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } attrs { replace: true } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: false } }
I20240502 21:27:48.930454 24317 raft_consensus.cc:3037] T fcc9cbeae6f343af9f84c73349f3dadb P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:48.932634 24317 raft_consensus.cc:2454] T fcc9cbeae6f343af9f84c73349f3dadb P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 009d3e7e8a8b4e2497de3a73d01ced49 in term 2.
I20240502 21:27:48.932996 24116 leader_election.cc:302] T fcc9cbeae6f343af9f84c73349f3dadb P 009d3e7e8a8b4e2497de3a73d01ced49 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 009d3e7e8a8b4e2497de3a73d01ced49, 038c5d9d1d7a49c981c4c34fbc00d9e6, fb6004b8640a4906aa0cfe63e383d40e; no voters:
I20240502 21:27:48.933322 26475 raft_consensus.cc:2781] T fcc9cbeae6f343af9f84c73349f3dadb P 009d3e7e8a8b4e2497de3a73d01ced49 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 21:27:48.933447 26475 raft_consensus.cc:686] T fcc9cbeae6f343af9f84c73349f3dadb P 009d3e7e8a8b4e2497de3a73d01ced49 [term 2 LEADER]: Becoming Leader. State: Replica: 009d3e7e8a8b4e2497de3a73d01ced49, State: Running, Role: LEADER
I20240502 21:27:48.933678 26475 consensus_queue.cc:237] T fcc9cbeae6f343af9f84c73349f3dadb P 009d3e7e8a8b4e2497de3a73d01ced49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } attrs { replace: true } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: false } }
I20240502 21:27:48.936338 23766 catalog_manager.cc:5484] T fcc9cbeae6f343af9f84c73349f3dadb P 009d3e7e8a8b4e2497de3a73d01ced49 reported cstate change: term changed from 1 to 2, leader changed from 038c5d9d1d7a49c981c4c34fbc00d9e6 (127.15.255.132) to 009d3e7e8a8b4e2497de3a73d01ced49 (127.15.255.131). New cstate: current_term: 2 leader_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:48.941704 23915 consensus_queue.cc:237] T 256ae04a48b849c4a9086b6b1ff5df19 P 6810a866decb4f5089f7d35c870e4204 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } attrs { replace: true } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: NON_VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: true } }
I20240502 21:27:48.944382 24046 raft_consensus.cc:1261] T 256ae04a48b849c4a9086b6b1ff5df19 P 6c194a6db6c145acafc0f3068b8ce1fd [term 1 FOLLOWER]: Refusing update from remote peer 6810a866decb4f5089f7d35c870e4204: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 21:27:48.944912 23850 consensus_peers.cc:485] T 256ae04a48b849c4a9086b6b1ff5df19 P 6810a866decb4f5089f7d35c870e4204 -> Peer e117afbf8819424e8a98476453dfde2a (127.15.255.135:32775): Couldn't send request to peer e117afbf8819424e8a98476453dfde2a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 256ae04a48b849c4a9086b6b1ff5df19. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:48.945042 24185 raft_consensus.cc:1261] T 256ae04a48b849c4a9086b6b1ff5df19 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 1 FOLLOWER]: Refusing update from remote peer 6810a866decb4f5089f7d35c870e4204: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:27:48.944983 27234 consensus_queue.cc:1035] T 256ae04a48b849c4a9086b6b1ff5df19 P 6810a866decb4f5089f7d35c870e4204 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:27:48.945536 27234 consensus_queue.cc:1035] T 256ae04a48b849c4a9086b6b1ff5df19 P 6810a866decb4f5089f7d35c870e4204 [LEADER]: Connected to new peer: Peer: permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:27:48.947383 27327 raft_consensus.cc:2932] T 256ae04a48b849c4a9086b6b1ff5df19 P 6810a866decb4f5089f7d35c870e4204 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER e117afbf8819424e8a98476453dfde2a (127.15.255.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } attrs { replace: true } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: NON_VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: true } } }
I20240502 21:27:48.948249 24046 raft_consensus.cc:2932] T 256ae04a48b849c4a9086b6b1ff5df19 P 6c194a6db6c145acafc0f3068b8ce1fd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER e117afbf8819424e8a98476453dfde2a (127.15.255.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } attrs { replace: true } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: NON_VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: true } } }
I20240502 21:27:48.950676 24185 raft_consensus.cc:2932] T 256ae04a48b849c4a9086b6b1ff5df19 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER e117afbf8819424e8a98476453dfde2a (127.15.255.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } attrs { replace: true } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: NON_VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: true } } }
I20240502 21:27:48.952230 23766 catalog_manager.cc:5484] T 256ae04a48b849c4a9086b6b1ff5df19 P 6810a866decb4f5089f7d35c870e4204 reported cstate change: config changed from index -1 to 4, NON_VOTER e117afbf8819424e8a98476453dfde2a (127.15.255.135) added. New cstate: current_term: 1 leader_uuid: "6810a866decb4f5089f7d35c870e4204" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: NON_VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:49.003288 23915 raft_consensus.cc:1261] T 89f86d58fe414b579845214534d338d8 P 6810a866decb4f5089f7d35c870e4204 [term 2 FOLLOWER]: Refusing update from remote peer fb6004b8640a4906aa0cfe63e383d40e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 21:27:49.004035 27031 consensus_queue.cc:1035] T 89f86d58fe414b579845214534d338d8 P fb6004b8640a4906aa0cfe63e383d40e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:27:49.052620 24046 raft_consensus.cc:1261] T f84bbe304597450c978ae28e0c4e7a3e P 6c194a6db6c145acafc0f3068b8ce1fd [term 2 FOLLOWER]: Refusing update from remote peer a3e500d7077a48acb5edb33f7e178c81: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 21:27:49.053323 27045 consensus_queue.cc:1035] T f84bbe304597450c978ae28e0c4e7a3e P a3e500d7077a48acb5edb33f7e178c81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:27:49.064800 24317 raft_consensus.cc:1261] T 89f86d58fe414b579845214534d338d8 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 2 FOLLOWER]: Refusing update from remote peer fb6004b8640a4906aa0cfe63e383d40e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 21:27:49.065583 27031 consensus_queue.cc:1035] T 89f86d58fe414b579845214534d338d8 P fb6004b8640a4906aa0cfe63e383d40e [LEADER]: Connected to new peer: Peer: permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:27:49.065802 24185 raft_consensus.cc:1261] T 89f86d58fe414b579845214534d338d8 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 2 FOLLOWER]: Refusing update from remote peer fb6004b8640a4906aa0cfe63e383d40e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 21:27:49.066372 27031 consensus_queue.cc:1035] T 89f86d58fe414b579845214534d338d8 P fb6004b8640a4906aa0cfe63e383d40e [LEADER]: Connected to new peer: Peer: permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:27:49.105011 26011 consensus_queue.cc:237] T 89f86d58fe414b579845214534d338d8 P fb6004b8640a4906aa0cfe63e383d40e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: false } }
I20240502 21:27:49.107683 24317 raft_consensus.cc:1261] T 89f86d58fe414b579845214534d338d8 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 2 FOLLOWER]: Refusing update from remote peer fb6004b8640a4906aa0cfe63e383d40e: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 21:27:49.107880 24185 raft_consensus.cc:1261] T 89f86d58fe414b579845214534d338d8 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 2 FOLLOWER]: Refusing update from remote peer fb6004b8640a4906aa0cfe63e383d40e: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 21:27:49.108469 27031 consensus_queue.cc:1035] T 89f86d58fe414b579845214534d338d8 P fb6004b8640a4906aa0cfe63e383d40e [LEADER]: Connected to new peer: Peer: permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:27:49.109000 27031 consensus_queue.cc:1035] T 89f86d58fe414b579845214534d338d8 P fb6004b8640a4906aa0cfe63e383d40e [LEADER]: Connected to new peer: Peer: permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:27:49.111049 27796 raft_consensus.cc:2932] T 89f86d58fe414b579845214534d338d8 P fb6004b8640a4906aa0cfe63e383d40e [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 6810a866decb4f5089f7d35c870e4204 (127.15.255.129) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: false } } }
I20240502 21:27:49.114578 24317 raft_consensus.cc:2932] T 89f86d58fe414b579845214534d338d8 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 6810a866decb4f5089f7d35c870e4204 (127.15.255.129) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: false } } }
I20240502 21:27:49.116896 23754 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 89f86d58fe414b579845214534d338d8 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:27:49.118036 23766 catalog_manager.cc:5484] T 89f86d58fe414b579845214534d338d8 P fb6004b8640a4906aa0cfe63e383d40e reported cstate change: config changed from index 5 to 7, VOTER 6810a866decb4f5089f7d35c870e4204 (127.15.255.129) evicted. New cstate: current_term: 2 leader_uuid: "fb6004b8640a4906aa0cfe63e383d40e" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:27:49.120636 24185 raft_consensus.cc:2932] T 89f86d58fe414b579845214534d338d8 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 6810a866decb4f5089f7d35c870e4204 (127.15.255.129) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: false } } }
I20240502 21:27:49.122294 23894 tablet_service.cc:1514] Processing DeleteTablet for tablet 89f86d58fe414b579845214534d338d8 with delete_type TABLET_DATA_TOMBSTONED (TS 6810a866decb4f5089f7d35c870e4204 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:33512
I20240502 21:27:49.133167 27813 ts_tablet_manager.cc:902] T 77fc7226878f44d3884ccb1fc41c44a4 P e117afbf8819424e8a98476453dfde2a: Initiating tablet copy from peer 6c194a6db6c145acafc0f3068b8ce1fd (127.15.255.130:41235)
I20240502 21:27:49.136852 27811 tablet_replica.cc:330] T 89f86d58fe414b579845214534d338d8 P 6810a866decb4f5089f7d35c870e4204: stopping tablet replica
I20240502 21:27:49.137089 27811 raft_consensus.cc:2229] T 89f86d58fe414b579845214534d338d8 P 6810a866decb4f5089f7d35c870e4204 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:49.137291 27811 raft_consensus.cc:2258] T 89f86d58fe414b579845214534d338d8 P 6810a866decb4f5089f7d35c870e4204 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:49.138592 27811 ts_tablet_manager.cc:1875] T 89f86d58fe414b579845214534d338d8 P 6810a866decb4f5089f7d35c870e4204: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:27:49.141386 27813 tablet_copy_client.cc:308] T 77fc7226878f44d3884ccb1fc41c44a4 P e117afbf8819424e8a98476453dfde2a: tablet copy: Beginning tablet copy session from remote peer at address 127.15.255.130:41235
I20240502 21:27:49.142185 24070 tablet_copy_service.cc:140] P 6c194a6db6c145acafc0f3068b8ce1fd: Received BeginTabletCopySession request for tablet 77fc7226878f44d3884ccb1fc41c44a4 from peer e117afbf8819424e8a98476453dfde2a ({username='slave'} at 127.15.255.135:53383)
I20240502 21:27:49.142354 24070 tablet_copy_service.cc:161] P 6c194a6db6c145acafc0f3068b8ce1fd: Beginning new tablet copy session on tablet 77fc7226878f44d3884ccb1fc41c44a4 from peer e117afbf8819424e8a98476453dfde2a at {username='slave'} at 127.15.255.135:53383: session id = e117afbf8819424e8a98476453dfde2a-77fc7226878f44d3884ccb1fc41c44a4
I20240502 21:27:49.145398 24070 tablet_copy_source_session.cc:205] T 77fc7226878f44d3884ccb1fc41c44a4 P 6c194a6db6c145acafc0f3068b8ce1fd: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:27:49.146620 27811 ts_tablet_manager.cc:1888] T 89f86d58fe414b579845214534d338d8 P 6810a866decb4f5089f7d35c870e4204: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240502 21:27:49.146835 27811 log.cc:1196] T 89f86d58fe414b579845214534d338d8 P 6810a866decb4f5089f7d35c870e4204: Deleting WAL directory at /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-0/wal/wals/89f86d58fe414b579845214534d338d8
I20240502 21:27:49.146989 24185 raft_consensus.cc:1261] T f84bbe304597450c978ae28e0c4e7a3e P 009d3e7e8a8b4e2497de3a73d01ced49 [term 2 FOLLOWER]: Refusing update from remote peer a3e500d7077a48acb5edb33f7e178c81: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 21:27:49.147347 27813 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77fc7226878f44d3884ccb1fc41c44a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:49.147724 23757 catalog_manager.cc:4830] TS 6810a866decb4f5089f7d35c870e4204 (127.15.255.129:40435): tablet 89f86d58fe414b579845214534d338d8 (table rebalance_test_table_3 [id=933ce90acb9f4b91ab6bd41e87a8fe99]) successfully deleted
I20240502 21:27:49.148296 27045 consensus_queue.cc:1035] T f84bbe304597450c978ae28e0c4e7a3e P a3e500d7077a48acb5edb33f7e178c81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:27:49.154713 27813 tablet_copy_client.cc:783] T 77fc7226878f44d3884ccb1fc41c44a4 P e117afbf8819424e8a98476453dfde2a: tablet copy: Starting download of 0 data blocks...
I20240502 21:27:49.155215 27813 tablet_copy_client.cc:647] T 77fc7226878f44d3884ccb1fc41c44a4 P e117afbf8819424e8a98476453dfde2a: tablet copy: Starting download of 1 WAL segments...
I20240502 21:27:49.158073 24317 raft_consensus.cc:1261] T f84bbe304597450c978ae28e0c4e7a3e P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 2 FOLLOWER]: Refusing update from remote peer a3e500d7077a48acb5edb33f7e178c81: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 21:27:49.158460 27817 tablet_copy_client.cc:1002] T 77fc7226878f44d3884ccb1fc41c44a4 P e117afbf8819424e8a98476453dfde2a: tablet copy: Injecting latency into file download: 1500
I20240502 21:27:49.158694 27820 consensus_queue.cc:1035] T f84bbe304597450c978ae28e0c4e7a3e P a3e500d7077a48acb5edb33f7e178c81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:27:49.173404 27827 ts_tablet_manager.cc:902] T d79eec4f59634df480161815d888bcac P ee27c3f9e0ec46caa742ac386d6b7c20: Initiating tablet copy from peer 009d3e7e8a8b4e2497de3a73d01ced49 (127.15.255.131:37895)
I20240502 21:27:49.174466 27827 tablet_copy_client.cc:308] T d79eec4f59634df480161815d888bcac P ee27c3f9e0ec46caa742ac386d6b7c20: tablet copy: Beginning tablet copy session from remote peer at address 127.15.255.131:37895
I20240502 21:27:49.175096 24205 tablet_copy_service.cc:140] P 009d3e7e8a8b4e2497de3a73d01ced49: Received BeginTabletCopySession request for tablet d79eec4f59634df480161815d888bcac from peer ee27c3f9e0ec46caa742ac386d6b7c20 ({username='slave'} at 127.15.255.133:44245)
I20240502 21:27:49.175231 24205 tablet_copy_service.cc:161] P 009d3e7e8a8b4e2497de3a73d01ced49: Beginning new tablet copy session on tablet d79eec4f59634df480161815d888bcac from peer ee27c3f9e0ec46caa742ac386d6b7c20 at {username='slave'} at 127.15.255.133:44245: session id = ee27c3f9e0ec46caa742ac386d6b7c20-d79eec4f59634df480161815d888bcac
I20240502 21:27:49.177088 24205 tablet_copy_source_session.cc:205] T d79eec4f59634df480161815d888bcac P 009d3e7e8a8b4e2497de3a73d01ced49: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:27:49.178292 27827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d79eec4f59634df480161815d888bcac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:49.179154 27832 ts_tablet_manager.cc:902] T c9b5be01b5e941b1848df79c7a9db348 P e117afbf8819424e8a98476453dfde2a: Initiating tablet copy from peer 6c194a6db6c145acafc0f3068b8ce1fd (127.15.255.130:41235)
I20240502 21:27:49.180486 27832 tablet_copy_client.cc:308] T c9b5be01b5e941b1848df79c7a9db348 P e117afbf8819424e8a98476453dfde2a: tablet copy: Beginning tablet copy session from remote peer at address 127.15.255.130:41235
I20240502 21:27:49.181113 24070 tablet_copy_service.cc:140] P 6c194a6db6c145acafc0f3068b8ce1fd: Received BeginTabletCopySession request for tablet c9b5be01b5e941b1848df79c7a9db348 from peer e117afbf8819424e8a98476453dfde2a ({username='slave'} at 127.15.255.135:53383)
I20240502 21:27:49.181267 24070 tablet_copy_service.cc:161] P 6c194a6db6c145acafc0f3068b8ce1fd: Beginning new tablet copy session on tablet c9b5be01b5e941b1848df79c7a9db348 from peer e117afbf8819424e8a98476453dfde2a at {username='slave'} at 127.15.255.135:53383: session id = e117afbf8819424e8a98476453dfde2a-c9b5be01b5e941b1848df79c7a9db348
I20240502 21:27:49.183331 24070 tablet_copy_source_session.cc:205] T c9b5be01b5e941b1848df79c7a9db348 P 6c194a6db6c145acafc0f3068b8ce1fd: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:27:49.183650 27827 tablet_copy_client.cc:783] T d79eec4f59634df480161815d888bcac P ee27c3f9e0ec46caa742ac386d6b7c20: tablet copy: Starting download of 0 data blocks...
I20240502 21:27:49.183920 27827 tablet_copy_client.cc:647] T d79eec4f59634df480161815d888bcac P ee27c3f9e0ec46caa742ac386d6b7c20: tablet copy: Starting download of 1 WAL segments...
I20240502 21:27:49.184401 27832 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9b5be01b5e941b1848df79c7a9db348. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:49.185545 27831 tablet_copy_client.cc:1002] T d79eec4f59634df480161815d888bcac P ee27c3f9e0ec46caa742ac386d6b7c20: tablet copy: Injecting latency into file download: 1500
I20240502 21:27:49.189286 27832 tablet_copy_client.cc:783] T c9b5be01b5e941b1848df79c7a9db348 P e117afbf8819424e8a98476453dfde2a: tablet copy: Starting download of 0 data blocks...
I20240502 21:27:49.189543 27832 tablet_copy_client.cc:647] T c9b5be01b5e941b1848df79c7a9db348 P e117afbf8819424e8a98476453dfde2a: tablet copy: Starting download of 1 WAL segments...
I20240502 21:27:49.190824 27836 tablet_copy_client.cc:1002] T c9b5be01b5e941b1848df79c7a9db348 P e117afbf8819424e8a98476453dfde2a: tablet copy: Injecting latency into file download: 1500
I20240502 21:27:49.236774 27837 ts_tablet_manager.cc:902] T 8eba3872a77046ecb9c9477d4cc0e18b P ee27c3f9e0ec46caa742ac386d6b7c20: Initiating tablet copy from peer 038c5d9d1d7a49c981c4c34fbc00d9e6 (127.15.255.132:46813)
I20240502 21:27:49.238035 27837 tablet_copy_client.cc:308] T 8eba3872a77046ecb9c9477d4cc0e18b P ee27c3f9e0ec46caa742ac386d6b7c20: tablet copy: Beginning tablet copy session from remote peer at address 127.15.255.132:46813
I20240502 21:27:49.238757 24338 tablet_copy_service.cc:140] P 038c5d9d1d7a49c981c4c34fbc00d9e6: Received BeginTabletCopySession request for tablet 8eba3872a77046ecb9c9477d4cc0e18b from peer ee27c3f9e0ec46caa742ac386d6b7c20 ({username='slave'} at 127.15.255.133:45805)
I20240502 21:27:49.238950 24338 tablet_copy_service.cc:161] P 038c5d9d1d7a49c981c4c34fbc00d9e6: Beginning new tablet copy session on tablet 8eba3872a77046ecb9c9477d4cc0e18b from peer ee27c3f9e0ec46caa742ac386d6b7c20 at {username='slave'} at 127.15.255.133:45805: session id = ee27c3f9e0ec46caa742ac386d6b7c20-8eba3872a77046ecb9c9477d4cc0e18b
I20240502 21:27:49.241087 24338 tablet_copy_source_session.cc:205] T 8eba3872a77046ecb9c9477d4cc0e18b P 038c5d9d1d7a49c981c4c34fbc00d9e6: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:27:49.242117 27837 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8eba3872a77046ecb9c9477d4cc0e18b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:49.246336 27837 tablet_copy_client.cc:783] T 8eba3872a77046ecb9c9477d4cc0e18b P ee27c3f9e0ec46caa742ac386d6b7c20: tablet copy: Starting download of 0 data blocks...
I20240502 21:27:49.246534 27837 tablet_copy_client.cc:647] T 8eba3872a77046ecb9c9477d4cc0e18b P ee27c3f9e0ec46caa742ac386d6b7c20: tablet copy: Starting download of 1 WAL segments...
I20240502 21:27:49.247798 27841 tablet_copy_client.cc:1002] T 8eba3872a77046ecb9c9477d4cc0e18b P ee27c3f9e0ec46caa742ac386d6b7c20: tablet copy: Injecting latency into file download: 1500
I20240502 21:27:49.319837 24317 raft_consensus.cc:1261] T fcc9cbeae6f343af9f84c73349f3dadb P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 2 FOLLOWER]: Refusing update from remote peer 009d3e7e8a8b4e2497de3a73d01ced49: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 21:27:49.320339 27292 consensus_queue.cc:1035] T fcc9cbeae6f343af9f84c73349f3dadb P 009d3e7e8a8b4e2497de3a73d01ced49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:27:49.322525 26011 raft_consensus.cc:1261] T fcc9cbeae6f343af9f84c73349f3dadb P fb6004b8640a4906aa0cfe63e383d40e [term 2 FOLLOWER]: Refusing update from remote peer 009d3e7e8a8b4e2497de3a73d01ced49: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 21:27:49.323055 27292 consensus_queue.cc:1035] T fcc9cbeae6f343af9f84c73349f3dadb P 009d3e7e8a8b4e2497de3a73d01ced49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:27:49.326117 24046 raft_consensus.cc:1261] T fcc9cbeae6f343af9f84c73349f3dadb P 6c194a6db6c145acafc0f3068b8ce1fd [term 2 FOLLOWER]: Refusing update from remote peer 009d3e7e8a8b4e2497de3a73d01ced49: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 21:27:49.327685 27842 consensus_queue.cc:1035] T fcc9cbeae6f343af9f84c73349f3dadb P 009d3e7e8a8b4e2497de3a73d01ced49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:27:49.509240 27919 ts_tablet_manager.cc:902] T 256ae04a48b849c4a9086b6b1ff5df19 P e117afbf8819424e8a98476453dfde2a: Initiating tablet copy from peer 6810a866decb4f5089f7d35c870e4204 (127.15.255.129:40435)
I20240502 21:27:49.511469 27919 tablet_copy_client.cc:308] T 256ae04a48b849c4a9086b6b1ff5df19 P e117afbf8819424e8a98476453dfde2a: tablet copy: Beginning tablet copy session from remote peer at address 127.15.255.129:40435
I20240502 21:27:49.512164 23935 tablet_copy_service.cc:140] P 6810a866decb4f5089f7d35c870e4204: Received BeginTabletCopySession request for tablet 256ae04a48b849c4a9086b6b1ff5df19 from peer e117afbf8819424e8a98476453dfde2a ({username='slave'} at 127.15.255.135:39893)
I20240502 21:27:49.512346 23935 tablet_copy_service.cc:161] P 6810a866decb4f5089f7d35c870e4204: Beginning new tablet copy session on tablet 256ae04a48b849c4a9086b6b1ff5df19 from peer e117afbf8819424e8a98476453dfde2a at {username='slave'} at 127.15.255.135:39893: session id = e117afbf8819424e8a98476453dfde2a-256ae04a48b849c4a9086b6b1ff5df19
I20240502 21:27:49.514453 23935 tablet_copy_source_session.cc:205] T 256ae04a48b849c4a9086b6b1ff5df19 P 6810a866decb4f5089f7d35c870e4204: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:27:49.515859 27919 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 256ae04a48b849c4a9086b6b1ff5df19. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:49.522481 27919 tablet_copy_client.cc:783] T 256ae04a48b849c4a9086b6b1ff5df19 P e117afbf8819424e8a98476453dfde2a: tablet copy: Starting download of 0 data blocks...
I20240502 21:27:49.522742 27919 tablet_copy_client.cc:647] T 256ae04a48b849c4a9086b6b1ff5df19 P e117afbf8819424e8a98476453dfde2a: tablet copy: Starting download of 1 WAL segments...
I20240502 21:27:49.524693 27923 tablet_copy_client.cc:1002] T 256ae04a48b849c4a9086b6b1ff5df19 P e117afbf8819424e8a98476453dfde2a: tablet copy: Injecting latency into file download: 1500
I20240502 21:27:49.644709 26280 consensus_queue.cc:237] T f84bbe304597450c978ae28e0c4e7a3e P a3e500d7077a48acb5edb33f7e178c81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } }
I20240502 21:27:49.646368 24317 raft_consensus.cc:1261] T f84bbe304597450c978ae28e0c4e7a3e P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 2 FOLLOWER]: Refusing update from remote peer a3e500d7077a48acb5edb33f7e178c81: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 21:27:49.646371 24046 raft_consensus.cc:1261] T f84bbe304597450c978ae28e0c4e7a3e P 6c194a6db6c145acafc0f3068b8ce1fd [term 2 FOLLOWER]: Refusing update from remote peer a3e500d7077a48acb5edb33f7e178c81: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 21:27:49.646838 27045 consensus_queue.cc:1035] T f84bbe304597450c978ae28e0c4e7a3e P a3e500d7077a48acb5edb33f7e178c81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:27:49.647117 27820 consensus_queue.cc:1035] T f84bbe304597450c978ae28e0c4e7a3e P a3e500d7077a48acb5edb33f7e178c81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:27:49.649570 27045 raft_consensus.cc:2932] T f84bbe304597450c978ae28e0c4e7a3e P a3e500d7077a48acb5edb33f7e178c81 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 009d3e7e8a8b4e2497de3a73d01ced49 (127.15.255.131) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } } }
I20240502 21:27:49.650398 24046 raft_consensus.cc:2932] T f84bbe304597450c978ae28e0c4e7a3e P 6c194a6db6c145acafc0f3068b8ce1fd [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 009d3e7e8a8b4e2497de3a73d01ced49 (127.15.255.131) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } } }
I20240502 21:27:49.652758 24317 raft_consensus.cc:2932] T f84bbe304597450c978ae28e0c4e7a3e P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 009d3e7e8a8b4e2497de3a73d01ced49 (127.15.255.131) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } } }
I20240502 21:27:49.653553 23756 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f84bbe304597450c978ae28e0c4e7a3e with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:27:49.654191 23766 catalog_manager.cc:5484] T f84bbe304597450c978ae28e0c4e7a3e P a3e500d7077a48acb5edb33f7e178c81 reported cstate change: config changed from index 5 to 7, VOTER 009d3e7e8a8b4e2497de3a73d01ced49 (127.15.255.131) evicted. New cstate: current_term: 2 leader_uuid: "a3e500d7077a48acb5edb33f7e178c81" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:27:49.662487 24162 tablet_service.cc:1514] Processing DeleteTablet for tablet f84bbe304597450c978ae28e0c4e7a3e with delete_type TABLET_DATA_TOMBSTONED (TS 009d3e7e8a8b4e2497de3a73d01ced49 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:33102
I20240502 21:27:49.663146 27980 tablet_replica.cc:330] T f84bbe304597450c978ae28e0c4e7a3e P 009d3e7e8a8b4e2497de3a73d01ced49: stopping tablet replica
I20240502 21:27:49.663367 27980 raft_consensus.cc:2229] T f84bbe304597450c978ae28e0c4e7a3e P 009d3e7e8a8b4e2497de3a73d01ced49 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:49.663610 27980 raft_consensus.cc:2258] T f84bbe304597450c978ae28e0c4e7a3e P 009d3e7e8a8b4e2497de3a73d01ced49 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:49.664397 27980 ts_tablet_manager.cc:1875] T f84bbe304597450c978ae28e0c4e7a3e P 009d3e7e8a8b4e2497de3a73d01ced49: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:27:49.668507 27980 ts_tablet_manager.cc:1888] T f84bbe304597450c978ae28e0c4e7a3e P 009d3e7e8a8b4e2497de3a73d01ced49: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240502 21:27:49.668607 27980 log.cc:1196] T f84bbe304597450c978ae28e0c4e7a3e P 009d3e7e8a8b4e2497de3a73d01ced49: Deleting WAL directory at /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-2/wal/wals/f84bbe304597450c978ae28e0c4e7a3e
I20240502 21:27:49.669191 23756 catalog_manager.cc:4830] TS 009d3e7e8a8b4e2497de3a73d01ced49 (127.15.255.131:37895): tablet f84bbe304597450c978ae28e0c4e7a3e (table rebalance_test_table_7 [id=754dbb3bce6a4875a0b642faf5315709]) successfully deleted
I20240502 21:27:49.885990 24185 consensus_queue.cc:237] T fcc9cbeae6f343af9f84c73349f3dadb P 009d3e7e8a8b4e2497de3a73d01ced49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: false } }
I20240502 21:27:49.887794 24046 raft_consensus.cc:1261] T fcc9cbeae6f343af9f84c73349f3dadb P 6c194a6db6c145acafc0f3068b8ce1fd [term 2 FOLLOWER]: Refusing update from remote peer 009d3e7e8a8b4e2497de3a73d01ced49: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 21:27:49.887981 26012 raft_consensus.cc:1261] T fcc9cbeae6f343af9f84c73349f3dadb P fb6004b8640a4906aa0cfe63e383d40e [term 2 FOLLOWER]: Refusing update from remote peer 009d3e7e8a8b4e2497de3a73d01ced49: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 21:27:49.888444 27842 consensus_queue.cc:1035] T fcc9cbeae6f343af9f84c73349f3dadb P 009d3e7e8a8b4e2497de3a73d01ced49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:27:49.888859 27292 consensus_queue.cc:1035] T fcc9cbeae6f343af9f84c73349f3dadb P 009d3e7e8a8b4e2497de3a73d01ced49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:27:49.891353 27842 raft_consensus.cc:2932] T fcc9cbeae6f343af9f84c73349f3dadb P 009d3e7e8a8b4e2497de3a73d01ced49 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 038c5d9d1d7a49c981c4c34fbc00d9e6 (127.15.255.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: false } } }
I20240502 21:27:49.891935 24046 raft_consensus.cc:2932] T fcc9cbeae6f343af9f84c73349f3dadb P 6c194a6db6c145acafc0f3068b8ce1fd [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 038c5d9d1d7a49c981c4c34fbc00d9e6 (127.15.255.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: false } } }
I20240502 21:27:49.892776 26012 raft_consensus.cc:2932] T fcc9cbeae6f343af9f84c73349f3dadb P fb6004b8640a4906aa0cfe63e383d40e [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 038c5d9d1d7a49c981c4c34fbc00d9e6 (127.15.255.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: false } } }
I20240502 21:27:49.896206 23756 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet fcc9cbeae6f343af9f84c73349f3dadb with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:27:49.897460 23770 catalog_manager.cc:5484] T fcc9cbeae6f343af9f84c73349f3dadb P 009d3e7e8a8b4e2497de3a73d01ced49 reported cstate change: config changed from index 5 to 7, VOTER 038c5d9d1d7a49c981c4c34fbc00d9e6 (127.15.255.132) evicted. New cstate: current_term: 2 leader_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:27:49.906509 24297 tablet_service.cc:1514] Processing DeleteTablet for tablet fcc9cbeae6f343af9f84c73349f3dadb with delete_type TABLET_DATA_TOMBSTONED (TS 038c5d9d1d7a49c981c4c34fbc00d9e6 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:58268
I20240502 21:27:49.923395 28027 tablet_replica.cc:330] T fcc9cbeae6f343af9f84c73349f3dadb P 038c5d9d1d7a49c981c4c34fbc00d9e6: stopping tablet replica
I20240502 21:27:49.923770 28027 raft_consensus.cc:2229] T fcc9cbeae6f343af9f84c73349f3dadb P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:49.924067 28027 raft_consensus.cc:2258] T fcc9cbeae6f343af9f84c73349f3dadb P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:49.925482 28027 ts_tablet_manager.cc:1875] T fcc9cbeae6f343af9f84c73349f3dadb P 038c5d9d1d7a49c981c4c34fbc00d9e6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:27:49.931587 28027 ts_tablet_manager.cc:1888] T fcc9cbeae6f343af9f84c73349f3dadb P 038c5d9d1d7a49c981c4c34fbc00d9e6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240502 21:27:49.931865 28027 log.cc:1196] T fcc9cbeae6f343af9f84c73349f3dadb P 038c5d9d1d7a49c981c4c34fbc00d9e6: Deleting WAL directory at /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-3/wal/wals/fcc9cbeae6f343af9f84c73349f3dadb
I20240502 21:27:49.932832 23755 catalog_manager.cc:4830] TS 038c5d9d1d7a49c981c4c34fbc00d9e6 (127.15.255.132:46813): tablet fcc9cbeae6f343af9f84c73349f3dadb (table rebalance_test_table_9 [id=188ff51c46164f47803c423463c2e6e2]) successfully deleted
I20240502 21:27:50.080585 27566 tablet_copy_client.cc:523] T fef55863beb6435dbb25bfb6d71e3cd7 P a3e500d7077a48acb5edb33f7e178c81: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:27:50.084250 27566 tablet_bootstrap.cc:492] T fef55863beb6435dbb25bfb6d71e3cd7 P a3e500d7077a48acb5edb33f7e178c81: Bootstrap starting.
I20240502 21:27:50.106307 27573 tablet_copy_client.cc:523] T 52f621ea3f174490948eae854c706a92 P ee27c3f9e0ec46caa742ac386d6b7c20: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:27:50.109241 27573 tablet_bootstrap.cc:492] T 52f621ea3f174490948eae854c706a92 P ee27c3f9e0ec46caa742ac386d6b7c20: Bootstrap starting.
I20240502 21:27:50.125789 27566 tablet_bootstrap.cc:492] T fef55863beb6435dbb25bfb6d71e3cd7 P a3e500d7077a48acb5edb33f7e178c81: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:27:50.126139 27566 tablet_bootstrap.cc:492] T fef55863beb6435dbb25bfb6d71e3cd7 P a3e500d7077a48acb5edb33f7e178c81: Bootstrap complete.
I20240502 21:27:50.126392 27566 ts_tablet_manager.cc:1367] T fef55863beb6435dbb25bfb6d71e3cd7 P a3e500d7077a48acb5edb33f7e178c81: Time spent bootstrapping tablet: real 0.042s user 0.040s sys 0.004s
I20240502 21:27:50.127099 27566 raft_consensus.cc:348] T fef55863beb6435dbb25bfb6d71e3cd7 P a3e500d7077a48acb5edb33f7e178c81 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } attrs { replace: true } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: true } }
I20240502 21:27:50.127323 27566 raft_consensus.cc:729] T fef55863beb6435dbb25bfb6d71e3cd7 P a3e500d7077a48acb5edb33f7e178c81 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a3e500d7077a48acb5edb33f7e178c81, State: Initialized, Role: LEARNER
I20240502 21:27:50.127516 27566 consensus_queue.cc:260] T fef55863beb6435dbb25bfb6d71e3cd7 P a3e500d7077a48acb5edb33f7e178c81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } attrs { replace: true } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: true } }
I20240502 21:27:50.128015 27566 ts_tablet_manager.cc:1398] T fef55863beb6435dbb25bfb6d71e3cd7 P a3e500d7077a48acb5edb33f7e178c81: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 21:27:50.128858 24338 tablet_copy_service.cc:342] P 038c5d9d1d7a49c981c4c34fbc00d9e6: Request end of tablet copy session a3e500d7077a48acb5edb33f7e178c81-fef55863beb6435dbb25bfb6d71e3cd7 received from {username='slave'} at 127.15.255.136:38895
I20240502 21:27:50.128991 24338 tablet_copy_service.cc:433] P 038c5d9d1d7a49c981c4c34fbc00d9e6: ending tablet copy session a3e500d7077a48acb5edb33f7e178c81-fef55863beb6435dbb25bfb6d71e3cd7 on tablet fef55863beb6435dbb25bfb6d71e3cd7 with peer a3e500d7077a48acb5edb33f7e178c81
I20240502 21:27:50.136950 27584 tablet_copy_client.cc:523] T 30236241b670443489240f191d9b9d50 P fb6004b8640a4906aa0cfe63e383d40e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:27:50.137176 27563 tablet_copy_client.cc:523] T 8af15fc41246471db31f7824985b5903 P fb6004b8640a4906aa0cfe63e383d40e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:27:50.139901 27563 tablet_bootstrap.cc:492] T 8af15fc41246471db31f7824985b5903 P fb6004b8640a4906aa0cfe63e383d40e: Bootstrap starting.
I20240502 21:27:50.140648 27584 tablet_bootstrap.cc:492] T 30236241b670443489240f191d9b9d50 P fb6004b8640a4906aa0cfe63e383d40e: Bootstrap starting.
I20240502 21:27:50.144050 27573 tablet_bootstrap.cc:492] T 52f621ea3f174490948eae854c706a92 P ee27c3f9e0ec46caa742ac386d6b7c20: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:27:50.144482 27573 tablet_bootstrap.cc:492] T 52f621ea3f174490948eae854c706a92 P ee27c3f9e0ec46caa742ac386d6b7c20: Bootstrap complete.
I20240502 21:27:50.144753 27573 ts_tablet_manager.cc:1367] T 52f621ea3f174490948eae854c706a92 P ee27c3f9e0ec46caa742ac386d6b7c20: Time spent bootstrapping tablet: real 0.036s user 0.030s sys 0.005s
I20240502 21:27:50.145526 27573 raft_consensus.cc:348] T 52f621ea3f174490948eae854c706a92 P ee27c3f9e0ec46caa742ac386d6b7c20 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } attrs { replace: true } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: NON_VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: true } }
I20240502 21:27:50.145836 27573 raft_consensus.cc:729] T 52f621ea3f174490948eae854c706a92 P ee27c3f9e0ec46caa742ac386d6b7c20 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ee27c3f9e0ec46caa742ac386d6b7c20, State: Initialized, Role: LEARNER
I20240502 21:27:50.146034 27573 consensus_queue.cc:260] T 52f621ea3f174490948eae854c706a92 P ee27c3f9e0ec46caa742ac386d6b7c20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } attrs { replace: true } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: NON_VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: true } }
I20240502 21:27:50.146875 27573 ts_tablet_manager.cc:1398] T 52f621ea3f174490948eae854c706a92 P ee27c3f9e0ec46caa742ac386d6b7c20: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:27:50.147694 24338 tablet_copy_service.cc:342] P 038c5d9d1d7a49c981c4c34fbc00d9e6: Request end of tablet copy session ee27c3f9e0ec46caa742ac386d6b7c20-52f621ea3f174490948eae854c706a92 received from {username='slave'} at 127.15.255.133:45805
I20240502 21:27:50.147797 27593 tablet_copy_client.cc:523] T dc0e070513cc46ca82c1946423ad91c5 P e117afbf8819424e8a98476453dfde2a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:27:50.147825 24338 tablet_copy_service.cc:433] P 038c5d9d1d7a49c981c4c34fbc00d9e6: ending tablet copy session ee27c3f9e0ec46caa742ac386d6b7c20-52f621ea3f174490948eae854c706a92 on tablet 52f621ea3f174490948eae854c706a92 with peer ee27c3f9e0ec46caa742ac386d6b7c20
I20240502 21:27:50.151353 27593 tablet_bootstrap.cc:492] T dc0e070513cc46ca82c1946423ad91c5 P e117afbf8819424e8a98476453dfde2a: Bootstrap starting.
I20240502 21:27:50.185876 27563 tablet_bootstrap.cc:492] T 8af15fc41246471db31f7824985b5903 P fb6004b8640a4906aa0cfe63e383d40e: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:27:50.186378 27563 tablet_bootstrap.cc:492] T 8af15fc41246471db31f7824985b5903 P fb6004b8640a4906aa0cfe63e383d40e: Bootstrap complete.
I20240502 21:27:50.186702 27563 ts_tablet_manager.cc:1367] T 8af15fc41246471db31f7824985b5903 P fb6004b8640a4906aa0cfe63e383d40e: Time spent bootstrapping tablet: real 0.047s user 0.036s sys 0.010s
I20240502 21:27:50.187601 27563 raft_consensus.cc:348] T 8af15fc41246471db31f7824985b5903 P fb6004b8640a4906aa0cfe63e383d40e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } attrs { replace: true } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: NON_VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: true } }
I20240502 21:27:50.187911 27563 raft_consensus.cc:729] T 8af15fc41246471db31f7824985b5903 P fb6004b8640a4906aa0cfe63e383d40e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: fb6004b8640a4906aa0cfe63e383d40e, State: Initialized, Role: LEARNER
I20240502 21:27:50.188125 27563 consensus_queue.cc:260] T 8af15fc41246471db31f7824985b5903 P fb6004b8640a4906aa0cfe63e383d40e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } attrs { replace: true } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: NON_VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: true } }
I20240502 21:27:50.189517 27563 ts_tablet_manager.cc:1398] T 8af15fc41246471db31f7824985b5903 P fb6004b8640a4906aa0cfe63e383d40e: Time spent starting tablet: real 0.003s user 0.003s sys 0.001s
I20240502 21:27:50.190502 24070 tablet_copy_service.cc:342] P 6c194a6db6c145acafc0f3068b8ce1fd: Request end of tablet copy session fb6004b8640a4906aa0cfe63e383d40e-8af15fc41246471db31f7824985b5903 received from {username='slave'} at 127.15.255.134:44929
I20240502 21:27:50.190629 24070 tablet_copy_service.cc:433] P 6c194a6db6c145acafc0f3068b8ce1fd: ending tablet copy session fb6004b8640a4906aa0cfe63e383d40e-8af15fc41246471db31f7824985b5903 on tablet 8af15fc41246471db31f7824985b5903 with peer fb6004b8640a4906aa0cfe63e383d40e
I20240502 21:27:50.196559 27584 tablet_bootstrap.cc:492] T 30236241b670443489240f191d9b9d50 P fb6004b8640a4906aa0cfe63e383d40e: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:27:50.196934 27584 tablet_bootstrap.cc:492] T 30236241b670443489240f191d9b9d50 P fb6004b8640a4906aa0cfe63e383d40e: Bootstrap complete.
I20240502 21:27:50.197173 27584 ts_tablet_manager.cc:1367] T 30236241b670443489240f191d9b9d50 P fb6004b8640a4906aa0cfe63e383d40e: Time spent bootstrapping tablet: real 0.057s user 0.047s sys 0.001s
I20240502 21:27:50.197857 27584 raft_consensus.cc:348] T 30236241b670443489240f191d9b9d50 P fb6004b8640a4906aa0cfe63e383d40e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } attrs { replace: true } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: NON_VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: true } }
I20240502 21:27:50.198086 27584 raft_consensus.cc:729] T 30236241b670443489240f191d9b9d50 P fb6004b8640a4906aa0cfe63e383d40e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: fb6004b8640a4906aa0cfe63e383d40e, State: Initialized, Role: LEARNER
I20240502 21:27:50.198248 27584 consensus_queue.cc:260] T 30236241b670443489240f191d9b9d50 P fb6004b8640a4906aa0cfe63e383d40e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } attrs { replace: true } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: NON_VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: true } }
I20240502 21:27:50.198726 27584 ts_tablet_manager.cc:1398] T 30236241b670443489240f191d9b9d50 P fb6004b8640a4906aa0cfe63e383d40e: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 21:27:50.199795 23935 tablet_copy_service.cc:342] P 6810a866decb4f5089f7d35c870e4204: Request end of tablet copy session fb6004b8640a4906aa0cfe63e383d40e-30236241b670443489240f191d9b9d50 received from {username='slave'} at 127.15.255.134:44215
I20240502 21:27:50.199925 23935 tablet_copy_service.cc:433] P 6810a866decb4f5089f7d35c870e4204: ending tablet copy session fb6004b8640a4906aa0cfe63e383d40e-30236241b670443489240f191d9b9d50 on tablet 30236241b670443489240f191d9b9d50 with peer fb6004b8640a4906aa0cfe63e383d40e
I20240502 21:27:50.202111 27593 tablet_bootstrap.cc:492] T dc0e070513cc46ca82c1946423ad91c5 P e117afbf8819424e8a98476453dfde2a: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:27:50.202546 27593 tablet_bootstrap.cc:492] T dc0e070513cc46ca82c1946423ad91c5 P e117afbf8819424e8a98476453dfde2a: Bootstrap complete.
I20240502 21:27:50.202788 27593 ts_tablet_manager.cc:1367] T dc0e070513cc46ca82c1946423ad91c5 P e117afbf8819424e8a98476453dfde2a: Time spent bootstrapping tablet: real 0.052s user 0.031s sys 0.011s
I20240502 21:27:50.203500 27593 raft_consensus.cc:348] T dc0e070513cc46ca82c1946423ad91c5 P e117afbf8819424e8a98476453dfde2a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } attrs { replace: true } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: NON_VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: true } }
I20240502 21:27:50.203697 27593 raft_consensus.cc:729] T dc0e070513cc46ca82c1946423ad91c5 P e117afbf8819424e8a98476453dfde2a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e117afbf8819424e8a98476453dfde2a, State: Initialized, Role: LEARNER
I20240502 21:27:50.203887 27593 consensus_queue.cc:260] T dc0e070513cc46ca82c1946423ad91c5 P e117afbf8819424e8a98476453dfde2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } attrs { replace: true } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: NON_VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: true } }
I20240502 21:27:50.204641 27593 ts_tablet_manager.cc:1398] T dc0e070513cc46ca82c1946423ad91c5 P e117afbf8819424e8a98476453dfde2a: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240502 21:27:50.205368 24070 tablet_copy_service.cc:342] P 6c194a6db6c145acafc0f3068b8ce1fd: Request end of tablet copy session e117afbf8819424e8a98476453dfde2a-dc0e070513cc46ca82c1946423ad91c5 received from {username='slave'} at 127.15.255.135:53383
I20240502 21:27:50.205511 24070 tablet_copy_service.cc:433] P 6c194a6db6c145acafc0f3068b8ce1fd: ending tablet copy session e117afbf8819424e8a98476453dfde2a-dc0e070513cc46ca82c1946423ad91c5 on tablet dc0e070513cc46ca82c1946423ad91c5 with peer e117afbf8819424e8a98476453dfde2a
I20240502 21:27:50.347749 26012 raft_consensus.cc:1203] T 30236241b670443489240f191d9b9d50 P fb6004b8640a4906aa0cfe63e383d40e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 21:27:50.394834 25879 raft_consensus.cc:1203] T 52f621ea3f174490948eae854c706a92 P ee27c3f9e0ec46caa742ac386d6b7c20 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 21:27:50.448583 26280 raft_consensus.cc:1203] T fef55863beb6435dbb25bfb6d71e3cd7 P a3e500d7077a48acb5edb33f7e178c81 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 21:27:50.458282 26012 raft_consensus.cc:1203] T 8af15fc41246471db31f7824985b5903 P fb6004b8640a4906aa0cfe63e383d40e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 21:27:50.474092 26145 raft_consensus.cc:1203] T dc0e070513cc46ca82c1946423ad91c5 P e117afbf8819424e8a98476453dfde2a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 21:27:50.658789 27813 tablet_copy_client.cc:523] T 77fc7226878f44d3884ccb1fc41c44a4 P e117afbf8819424e8a98476453dfde2a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:27:50.661361 27813 tablet_bootstrap.cc:492] T 77fc7226878f44d3884ccb1fc41c44a4 P e117afbf8819424e8a98476453dfde2a: Bootstrap starting.
I20240502 21:27:50.685936 27827 tablet_copy_client.cc:523] T d79eec4f59634df480161815d888bcac P ee27c3f9e0ec46caa742ac386d6b7c20: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:27:50.688515 27827 tablet_bootstrap.cc:492] T d79eec4f59634df480161815d888bcac P ee27c3f9e0ec46caa742ac386d6b7c20: Bootstrap starting.
I20240502 21:27:50.691179 27832 tablet_copy_client.cc:523] T c9b5be01b5e941b1848df79c7a9db348 P e117afbf8819424e8a98476453dfde2a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:27:50.692456 27813 tablet_bootstrap.cc:492] T 77fc7226878f44d3884ccb1fc41c44a4 P e117afbf8819424e8a98476453dfde2a: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:27:50.692788 27813 tablet_bootstrap.cc:492] T 77fc7226878f44d3884ccb1fc41c44a4 P e117afbf8819424e8a98476453dfde2a: Bootstrap complete.
I20240502 21:27:50.693029 27813 ts_tablet_manager.cc:1367] T 77fc7226878f44d3884ccb1fc41c44a4 P e117afbf8819424e8a98476453dfde2a: Time spent bootstrapping tablet: real 0.032s user 0.028s sys 0.004s
I20240502 21:27:50.693722 27813 raft_consensus.cc:348] T 77fc7226878f44d3884ccb1fc41c44a4 P e117afbf8819424e8a98476453dfde2a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } attrs { replace: true } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: NON_VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: true } }
I20240502 21:27:50.693951 27813 raft_consensus.cc:729] T 77fc7226878f44d3884ccb1fc41c44a4 P e117afbf8819424e8a98476453dfde2a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e117afbf8819424e8a98476453dfde2a, State: Initialized, Role: LEARNER
I20240502 21:27:50.694185 27813 consensus_queue.cc:260] T 77fc7226878f44d3884ccb1fc41c44a4 P e117afbf8819424e8a98476453dfde2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } attrs { replace: true } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: NON_VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: true } }
I20240502 21:27:50.694701 27832 tablet_bootstrap.cc:492] T c9b5be01b5e941b1848df79c7a9db348 P e117afbf8819424e8a98476453dfde2a: Bootstrap starting.
I20240502 21:27:50.694818 27813 ts_tablet_manager.cc:1398] T 77fc7226878f44d3884ccb1fc41c44a4 P e117afbf8819424e8a98476453dfde2a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:27:50.695896 24070 tablet_copy_service.cc:342] P 6c194a6db6c145acafc0f3068b8ce1fd: Request end of tablet copy session e117afbf8819424e8a98476453dfde2a-77fc7226878f44d3884ccb1fc41c44a4 received from {username='slave'} at 127.15.255.135:53383
I20240502 21:27:50.696027 24070 tablet_copy_service.cc:433] P 6c194a6db6c145acafc0f3068b8ce1fd: ending tablet copy session e117afbf8819424e8a98476453dfde2a-77fc7226878f44d3884ccb1fc41c44a4 on tablet 77fc7226878f44d3884ccb1fc41c44a4 with peer e117afbf8819424e8a98476453dfde2a
I20240502 21:27:50.735141 27832 tablet_bootstrap.cc:492] T c9b5be01b5e941b1848df79c7a9db348 P e117afbf8819424e8a98476453dfde2a: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:27:50.735527 27832 tablet_bootstrap.cc:492] T c9b5be01b5e941b1848df79c7a9db348 P e117afbf8819424e8a98476453dfde2a: Bootstrap complete.
I20240502 21:27:50.735770 27832 ts_tablet_manager.cc:1367] T c9b5be01b5e941b1848df79c7a9db348 P e117afbf8819424e8a98476453dfde2a: Time spent bootstrapping tablet: real 0.041s user 0.032s sys 0.006s
I20240502 21:27:50.736017 27827 tablet_bootstrap.cc:492] T d79eec4f59634df480161815d888bcac P ee27c3f9e0ec46caa742ac386d6b7c20: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:27:50.736330 27827 tablet_bootstrap.cc:492] T d79eec4f59634df480161815d888bcac P ee27c3f9e0ec46caa742ac386d6b7c20: Bootstrap complete.
I20240502 21:27:50.736394 27832 raft_consensus.cc:348] T c9b5be01b5e941b1848df79c7a9db348 P e117afbf8819424e8a98476453dfde2a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } attrs { replace: true } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: NON_VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: true } }
I20240502 21:27:50.736582 27827 ts_tablet_manager.cc:1367] T d79eec4f59634df480161815d888bcac P ee27c3f9e0ec46caa742ac386d6b7c20: Time spent bootstrapping tablet: real 0.048s user 0.049s sys 0.000s
I20240502 21:27:50.736644 27832 raft_consensus.cc:729] T c9b5be01b5e941b1848df79c7a9db348 P e117afbf8819424e8a98476453dfde2a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e117afbf8819424e8a98476453dfde2a, State: Initialized, Role: LEARNER
I20240502 21:27:50.736822 27832 consensus_queue.cc:260] T c9b5be01b5e941b1848df79c7a9db348 P e117afbf8819424e8a98476453dfde2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } attrs { replace: true } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: NON_VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: true } }
I20240502 21:27:50.737288 27832 ts_tablet_manager.cc:1398] T c9b5be01b5e941b1848df79c7a9db348 P e117afbf8819424e8a98476453dfde2a: Time spent starting tablet: real 0.001s user 0.000s sys 0.004s
I20240502 21:27:50.737272 27827 raft_consensus.cc:348] T d79eec4f59634df480161815d888bcac P ee27c3f9e0ec46caa742ac386d6b7c20 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } attrs { replace: true } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: NON_VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: true } }
I20240502 21:27:50.737525 27827 raft_consensus.cc:729] T d79eec4f59634df480161815d888bcac P ee27c3f9e0ec46caa742ac386d6b7c20 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ee27c3f9e0ec46caa742ac386d6b7c20, State: Initialized, Role: LEARNER
I20240502 21:27:50.737738 27827 consensus_queue.cc:260] T d79eec4f59634df480161815d888bcac P ee27c3f9e0ec46caa742ac386d6b7c20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } attrs { replace: true } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: NON_VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: true } }
I20240502 21:27:50.738106 24070 tablet_copy_service.cc:342] P 6c194a6db6c145acafc0f3068b8ce1fd: Request end of tablet copy session e117afbf8819424e8a98476453dfde2a-c9b5be01b5e941b1848df79c7a9db348 received from {username='slave'} at 127.15.255.135:53383
I20240502 21:27:50.738271 24070 tablet_copy_service.cc:433] P 6c194a6db6c145acafc0f3068b8ce1fd: ending tablet copy session e117afbf8819424e8a98476453dfde2a-c9b5be01b5e941b1848df79c7a9db348 on tablet c9b5be01b5e941b1848df79c7a9db348 with peer e117afbf8819424e8a98476453dfde2a
I20240502 21:27:50.738286 27827 ts_tablet_manager.cc:1398] T d79eec4f59634df480161815d888bcac P ee27c3f9e0ec46caa742ac386d6b7c20: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:27:50.739130 24205 tablet_copy_service.cc:342] P 009d3e7e8a8b4e2497de3a73d01ced49: Request end of tablet copy session ee27c3f9e0ec46caa742ac386d6b7c20-d79eec4f59634df480161815d888bcac received from {username='slave'} at 127.15.255.133:44245
I20240502 21:27:50.739284 24205 tablet_copy_service.cc:433] P 009d3e7e8a8b4e2497de3a73d01ced49: ending tablet copy session ee27c3f9e0ec46caa742ac386d6b7c20-d79eec4f59634df480161815d888bcac on tablet d79eec4f59634df480161815d888bcac with peer ee27c3f9e0ec46caa742ac386d6b7c20
I20240502 21:27:50.748088 27837 tablet_copy_client.cc:523] T 8eba3872a77046ecb9c9477d4cc0e18b P ee27c3f9e0ec46caa742ac386d6b7c20: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:27:50.750659 27837 tablet_bootstrap.cc:492] T 8eba3872a77046ecb9c9477d4cc0e18b P ee27c3f9e0ec46caa742ac386d6b7c20: Bootstrap starting.
I20240502 21:27:50.781239 27837 tablet_bootstrap.cc:492] T 8eba3872a77046ecb9c9477d4cc0e18b P ee27c3f9e0ec46caa742ac386d6b7c20: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:27:50.781572 27837 tablet_bootstrap.cc:492] T 8eba3872a77046ecb9c9477d4cc0e18b P ee27c3f9e0ec46caa742ac386d6b7c20: Bootstrap complete.
I20240502 21:27:50.781791 27837 ts_tablet_manager.cc:1367] T 8eba3872a77046ecb9c9477d4cc0e18b P ee27c3f9e0ec46caa742ac386d6b7c20: Time spent bootstrapping tablet: real 0.031s user 0.031s sys 0.001s
I20240502 21:27:50.782395 27837 raft_consensus.cc:348] T 8eba3872a77046ecb9c9477d4cc0e18b P ee27c3f9e0ec46caa742ac386d6b7c20 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } attrs { replace: true } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: NON_VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: true } }
I20240502 21:27:50.782598 27837 raft_consensus.cc:729] T 8eba3872a77046ecb9c9477d4cc0e18b P ee27c3f9e0ec46caa742ac386d6b7c20 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ee27c3f9e0ec46caa742ac386d6b7c20, State: Initialized, Role: LEARNER
I20240502 21:27:50.782765 27837 consensus_queue.cc:260] T 8eba3872a77046ecb9c9477d4cc0e18b P ee27c3f9e0ec46caa742ac386d6b7c20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } attrs { replace: true } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: NON_VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: true } }
I20240502 21:27:50.783228 27837 ts_tablet_manager.cc:1398] T 8eba3872a77046ecb9c9477d4cc0e18b P ee27c3f9e0ec46caa742ac386d6b7c20: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 21:27:50.783876 24338 tablet_copy_service.cc:342] P 038c5d9d1d7a49c981c4c34fbc00d9e6: Request end of tablet copy session ee27c3f9e0ec46caa742ac386d6b7c20-8eba3872a77046ecb9c9477d4cc0e18b received from {username='slave'} at 127.15.255.133:45805
I20240502 21:27:50.784010 24338 tablet_copy_service.cc:433] P 038c5d9d1d7a49c981c4c34fbc00d9e6: ending tablet copy session ee27c3f9e0ec46caa742ac386d6b7c20-8eba3872a77046ecb9c9477d4cc0e18b on tablet 8eba3872a77046ecb9c9477d4cc0e18b with peer ee27c3f9e0ec46caa742ac386d6b7c20
I20240502 21:27:50.788899 27234 raft_consensus.cc:1050] T 30236241b670443489240f191d9b9d50 P 6810a866decb4f5089f7d35c870e4204: attempting to promote NON_VOTER fb6004b8640a4906aa0cfe63e383d40e to VOTER
I20240502 21:27:50.789551 27234 consensus_queue.cc:237] T 30236241b670443489240f191d9b9d50 P 6810a866decb4f5089f7d35c870e4204 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } attrs { replace: true } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: false } }
I20240502 21:27:50.791743 24046 raft_consensus.cc:1261] T 30236241b670443489240f191d9b9d50 P 6c194a6db6c145acafc0f3068b8ce1fd [term 1 FOLLOWER]: Refusing update from remote peer 6810a866decb4f5089f7d35c870e4204: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:27:50.791898 24317 raft_consensus.cc:1261] T 30236241b670443489240f191d9b9d50 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 1 FOLLOWER]: Refusing update from remote peer 6810a866decb4f5089f7d35c870e4204: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:27:50.792474 26012 raft_consensus.cc:1261] T 30236241b670443489240f191d9b9d50 P fb6004b8640a4906aa0cfe63e383d40e [term 1 LEARNER]: Refusing update from remote peer 6810a866decb4f5089f7d35c870e4204: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:27:50.792752 27234 consensus_queue.cc:1035] T 30236241b670443489240f191d9b9d50 P 6810a866decb4f5089f7d35c870e4204 [LEADER]: Connected to new peer: Peer: permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:27:50.793064 28210 consensus_queue.cc:1035] T 30236241b670443489240f191d9b9d50 P 6810a866decb4f5089f7d35c870e4204 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:27:50.793504 27327 consensus_queue.cc:1035] T 30236241b670443489240f191d9b9d50 P 6810a866decb4f5089f7d35c870e4204 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:27:50.802570 27327 raft_consensus.cc:2932] T 30236241b670443489240f191d9b9d50 P 6810a866decb4f5089f7d35c870e4204 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, fb6004b8640a4906aa0cfe63e383d40e (127.15.255.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } attrs { replace: true } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: false } } }
I20240502 21:27:50.803237 24317 raft_consensus.cc:2932] T 30236241b670443489240f191d9b9d50 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, fb6004b8640a4906aa0cfe63e383d40e (127.15.255.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } attrs { replace: true } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: false } } }
I20240502 21:27:50.805402 28153 raft_consensus.cc:1050] T 52f621ea3f174490948eae854c706a92 P 038c5d9d1d7a49c981c4c34fbc00d9e6: attempting to promote NON_VOTER ee27c3f9e0ec46caa742ac386d6b7c20 to VOTER
I20240502 21:27:50.806216 28153 consensus_queue.cc:237] T 52f621ea3f174490948eae854c706a92 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } attrs { replace: true } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: false } }
I20240502 21:27:50.806412 24046 raft_consensus.cc:2932] T 30236241b670443489240f191d9b9d50 P 6c194a6db6c145acafc0f3068b8ce1fd [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, fb6004b8640a4906aa0cfe63e383d40e (127.15.255.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } attrs { replace: true } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: false } } }
I20240502 21:27:50.807508 23766 catalog_manager.cc:5484] T 30236241b670443489240f191d9b9d50 P 6810a866decb4f5089f7d35c870e4204 reported cstate change: config changed from index 4 to 5, fb6004b8640a4906aa0cfe63e383d40e (127.15.255.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "6810a866decb4f5089f7d35c870e4204" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:50.808133 26012 raft_consensus.cc:2932] T 30236241b670443489240f191d9b9d50 P fb6004b8640a4906aa0cfe63e383d40e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, fb6004b8640a4906aa0cfe63e383d40e (127.15.255.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } attrs { replace: true } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: false } } }
I20240502 21:27:50.813612 24047 raft_consensus.cc:1261] T 52f621ea3f174490948eae854c706a92 P 6c194a6db6c145acafc0f3068b8ce1fd [term 1 FOLLOWER]: Refusing update from remote peer 038c5d9d1d7a49c981c4c34fbc00d9e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:27:50.813690 23915 raft_consensus.cc:1261] T 52f621ea3f174490948eae854c706a92 P 6810a866decb4f5089f7d35c870e4204 [term 1 FOLLOWER]: Refusing update from remote peer 038c5d9d1d7a49c981c4c34fbc00d9e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:27:50.813822 25879 raft_consensus.cc:1261] T 52f621ea3f174490948eae854c706a92 P ee27c3f9e0ec46caa742ac386d6b7c20 [term 1 LEARNER]: Refusing update from remote peer 038c5d9d1d7a49c981c4c34fbc00d9e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:27:50.814463 27349 consensus_queue.cc:1035] T 52f621ea3f174490948eae854c706a92 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:27:50.814782 28152 consensus_queue.cc:1035] T 52f621ea3f174490948eae854c706a92 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:27:50.815290 27349 consensus_queue.cc:1035] T 52f621ea3f174490948eae854c706a92 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:27:50.837877 28152 raft_consensus.cc:2932] T 52f621ea3f174490948eae854c706a92 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ee27c3f9e0ec46caa742ac386d6b7c20 (127.15.255.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } attrs { replace: true } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: false } } }
I20240502 21:27:50.838725 25879 raft_consensus.cc:2932] T 52f621ea3f174490948eae854c706a92 P ee27c3f9e0ec46caa742ac386d6b7c20 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ee27c3f9e0ec46caa742ac386d6b7c20 (127.15.255.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } attrs { replace: true } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: false } } }
I20240502 21:27:50.840485 23915 raft_consensus.cc:2932] T 52f621ea3f174490948eae854c706a92 P 6810a866decb4f5089f7d35c870e4204 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ee27c3f9e0ec46caa742ac386d6b7c20 (127.15.255.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } attrs { replace: true } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: false } } }
I20240502 21:27:50.843001 24047 raft_consensus.cc:2932] T 52f621ea3f174490948eae854c706a92 P 6c194a6db6c145acafc0f3068b8ce1fd [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ee27c3f9e0ec46caa742ac386d6b7c20 (127.15.255.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } attrs { replace: true } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: false } } }
I20240502 21:27:50.850278 23765 catalog_manager.cc:5484] T 52f621ea3f174490948eae854c706a92 P ee27c3f9e0ec46caa742ac386d6b7c20 reported cstate change: config changed from index 4 to 5, ee27c3f9e0ec46caa742ac386d6b7c20 (127.15.255.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } attrs { replace: true } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: false } } }
I20240502 21:27:50.856964 23916 consensus_queue.cc:237] T 30236241b670443489240f191d9b9d50 P 6810a866decb4f5089f7d35c870e4204 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: false } }
I20240502 21:27:50.858425 24317 consensus_queue.cc:237] T 52f621ea3f174490948eae854c706a92 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: false } }
I20240502 21:27:50.858915 26012 raft_consensus.cc:1261] T 30236241b670443489240f191d9b9d50 P fb6004b8640a4906aa0cfe63e383d40e [term 1 FOLLOWER]: Refusing update from remote peer 6810a866decb4f5089f7d35c870e4204: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:27:50.859558 24316 raft_consensus.cc:1261] T 30236241b670443489240f191d9b9d50 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 1 FOLLOWER]: Refusing update from remote peer 6810a866decb4f5089f7d35c870e4204: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:27:50.859516 27234 consensus_queue.cc:1035] T 30236241b670443489240f191d9b9d50 P 6810a866decb4f5089f7d35c870e4204 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:27:50.860560 27234 consensus_queue.cc:1035] T 30236241b670443489240f191d9b9d50 P 6810a866decb4f5089f7d35c870e4204 [LEADER]: Connected to new peer: Peer: permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:27:50.861747 27328 raft_consensus.cc:2932] T 30236241b670443489240f191d9b9d50 P 6810a866decb4f5089f7d35c870e4204 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 6c194a6db6c145acafc0f3068b8ce1fd (127.15.255.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: false } } }
I20240502 21:27:50.862469 23916 raft_consensus.cc:1261] T 52f621ea3f174490948eae854c706a92 P 6810a866decb4f5089f7d35c870e4204 [term 1 FOLLOWER]: Refusing update from remote peer 038c5d9d1d7a49c981c4c34fbc00d9e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:27:50.862644 25878 raft_consensus.cc:1203] T d79eec4f59634df480161815d888bcac P ee27c3f9e0ec46caa742ac386d6b7c20 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 21:27:50.862653 25879 raft_consensus.cc:1261] T 52f621ea3f174490948eae854c706a92 P ee27c3f9e0ec46caa742ac386d6b7c20 [term 1 FOLLOWER]: Refusing update from remote peer 038c5d9d1d7a49c981c4c34fbc00d9e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:27:50.863221 28153 consensus_queue.cc:1035] T 52f621ea3f174490948eae854c706a92 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:27:50.864180 28153 consensus_queue.cc:1035] T 52f621ea3f174490948eae854c706a92 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:27:50.865396 23757 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 30236241b670443489240f191d9b9d50 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:27:50.865579 28153 raft_consensus.cc:2932] T 52f621ea3f174490948eae854c706a92 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 6c194a6db6c145acafc0f3068b8ce1fd (127.15.255.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: false } } }
I20240502 21:27:50.865847 26012 raft_consensus.cc:2932] T 30236241b670443489240f191d9b9d50 P fb6004b8640a4906aa0cfe63e383d40e [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 6c194a6db6c145acafc0f3068b8ce1fd (127.15.255.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: false } } }
I20240502 21:27:50.868019 23770 catalog_manager.cc:5484] T 30236241b670443489240f191d9b9d50 P 6810a866decb4f5089f7d35c870e4204 reported cstate change: config changed from index 5 to 6, VOTER 6c194a6db6c145acafc0f3068b8ce1fd (127.15.255.130) evicted. New cstate: current_term: 1 leader_uuid: "6810a866decb4f5089f7d35c870e4204" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:27:50.868624 23755 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 52f621ea3f174490948eae854c706a92 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:27:50.870183 24316 raft_consensus.cc:2932] T 30236241b670443489240f191d9b9d50 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 6c194a6db6c145acafc0f3068b8ce1fd (127.15.255.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: false } } }
I20240502 21:27:50.870903 23768 catalog_manager.cc:5484] T 52f621ea3f174490948eae854c706a92 P 038c5d9d1d7a49c981c4c34fbc00d9e6 reported cstate change: config changed from index 5 to 6, VOTER 6c194a6db6c145acafc0f3068b8ce1fd (127.15.255.130) evicted. New cstate: current_term: 1 leader_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:50.871186 25879 raft_consensus.cc:2932] T 52f621ea3f174490948eae854c706a92 P ee27c3f9e0ec46caa742ac386d6b7c20 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 6c194a6db6c145acafc0f3068b8ce1fd (127.15.255.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: false } } }
I20240502 21:27:50.872323 23916 raft_consensus.cc:2932] T 52f621ea3f174490948eae854c706a92 P 6810a866decb4f5089f7d35c870e4204 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 6c194a6db6c145acafc0f3068b8ce1fd (127.15.255.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: false } } }
I20240502 21:27:50.877090 24027 tablet_service.cc:1514] Processing DeleteTablet for tablet 30236241b670443489240f191d9b9d50 with delete_type TABLET_DATA_TOMBSTONED (TS 6c194a6db6c145acafc0f3068b8ce1fd not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:47386
I20240502 21:27:50.880622 24029 tablet_service.cc:1514] Processing DeleteTablet for tablet 52f621ea3f174490948eae854c706a92 with delete_type TABLET_DATA_TOMBSTONED (TS 6c194a6db6c145acafc0f3068b8ce1fd not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:47386
I20240502 21:27:50.886801 28231 raft_consensus.cc:1050] T dc0e070513cc46ca82c1946423ad91c5 P 6c194a6db6c145acafc0f3068b8ce1fd: attempting to promote NON_VOTER e117afbf8819424e8a98476453dfde2a to VOTER
I20240502 21:27:50.887665 28231 consensus_queue.cc:237] T dc0e070513cc46ca82c1946423ad91c5 P 6c194a6db6c145acafc0f3068b8ce1fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } attrs { replace: true } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } }
I20240502 21:27:50.888978 28228 tablet_replica.cc:330] T 30236241b670443489240f191d9b9d50 P 6c194a6db6c145acafc0f3068b8ce1fd: stopping tablet replica
I20240502 21:27:50.889199 28228 raft_consensus.cc:2229] T 30236241b670443489240f191d9b9d50 P 6c194a6db6c145acafc0f3068b8ce1fd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:50.889523 28228 raft_consensus.cc:2258] T 30236241b670443489240f191d9b9d50 P 6c194a6db6c145acafc0f3068b8ce1fd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:50.890512 28228 ts_tablet_manager.cc:1875] T 30236241b670443489240f191d9b9d50 P 6c194a6db6c145acafc0f3068b8ce1fd: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:27:50.892870 27456 raft_consensus.cc:1050] T 8af15fc41246471db31f7824985b5903 P 6c194a6db6c145acafc0f3068b8ce1fd: attempting to promote NON_VOTER fb6004b8640a4906aa0cfe63e383d40e to VOTER
I20240502 21:27:50.893707 27456 consensus_queue.cc:237] T 8af15fc41246471db31f7824985b5903 P 6c194a6db6c145acafc0f3068b8ce1fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } attrs { replace: true } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: false } }
I20240502 21:27:50.897163 28228 ts_tablet_manager.cc:1888] T 30236241b670443489240f191d9b9d50 P 6c194a6db6c145acafc0f3068b8ce1fd: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 21:27:50.897337 28228 log.cc:1196] T 30236241b670443489240f191d9b9d50 P 6c194a6db6c145acafc0f3068b8ce1fd: Deleting WAL directory at /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-1/wal/wals/30236241b670443489240f191d9b9d50
I20240502 21:27:50.898063 28228 tablet_replica.cc:330] T 52f621ea3f174490948eae854c706a92 P 6c194a6db6c145acafc0f3068b8ce1fd: stopping tablet replica
I20240502 21:27:50.898108 23757 catalog_manager.cc:4830] TS 6c194a6db6c145acafc0f3068b8ce1fd (127.15.255.130:41235): tablet 30236241b670443489240f191d9b9d50 (table rebalance_test_table_4 [id=9d2b330e886e4e83a437506bf99d77a3]) successfully deleted
I20240502 21:27:50.898357 28228 raft_consensus.cc:2229] T 52f621ea3f174490948eae854c706a92 P 6c194a6db6c145acafc0f3068b8ce1fd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:50.898604 28228 raft_consensus.cc:2258] T 52f621ea3f174490948eae854c706a92 P 6c194a6db6c145acafc0f3068b8ce1fd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:50.899340 28228 ts_tablet_manager.cc:1875] T 52f621ea3f174490948eae854c706a92 P 6c194a6db6c145acafc0f3068b8ce1fd: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:27:50.900862 26145 raft_consensus.cc:1261] T dc0e070513cc46ca82c1946423ad91c5 P e117afbf8819424e8a98476453dfde2a [term 1 LEARNER]: Refusing update from remote peer 6c194a6db6c145acafc0f3068b8ce1fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:27:50.903455 23916 raft_consensus.cc:1261] T dc0e070513cc46ca82c1946423ad91c5 P 6810a866decb4f5089f7d35c870e4204 [term 1 FOLLOWER]: Refusing update from remote peer 6c194a6db6c145acafc0f3068b8ce1fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:27:50.904106 28099 consensus_queue.cc:1035] T dc0e070513cc46ca82c1946423ad91c5 P 6c194a6db6c145acafc0f3068b8ce1fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:27:50.904695 28099 consensus_queue.cc:1035] T dc0e070513cc46ca82c1946423ad91c5 P 6c194a6db6c145acafc0f3068b8ce1fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:27:50.905007 28228 ts_tablet_manager.cc:1888] T 52f621ea3f174490948eae854c706a92 P 6c194a6db6c145acafc0f3068b8ce1fd: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 21:27:50.905131 28228 log.cc:1196] T 52f621ea3f174490948eae854c706a92 P 6c194a6db6c145acafc0f3068b8ce1fd: Deleting WAL directory at /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-1/wal/wals/52f621ea3f174490948eae854c706a92
I20240502 21:27:50.906548 23757 catalog_manager.cc:4830] TS 6c194a6db6c145acafc0f3068b8ce1fd (127.15.255.130:41235): tablet 52f621ea3f174490948eae854c706a92 (table rebalance_test_table_5 [id=e10e5576c0ea45a09c0eab75c2023b78]) successfully deleted
I20240502 21:27:50.907016 24185 raft_consensus.cc:1261] T dc0e070513cc46ca82c1946423ad91c5 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 1 FOLLOWER]: Refusing update from remote peer 6c194a6db6c145acafc0f3068b8ce1fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:27:50.907356 26012 raft_consensus.cc:1261] T 8af15fc41246471db31f7824985b5903 P fb6004b8640a4906aa0cfe63e383d40e [term 1 LEARNER]: Refusing update from remote peer 6c194a6db6c145acafc0f3068b8ce1fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:27:50.907364 24316 raft_consensus.cc:1261] T 8af15fc41246471db31f7824985b5903 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 1 FOLLOWER]: Refusing update from remote peer 6c194a6db6c145acafc0f3068b8ce1fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:27:50.907904 27456 consensus_queue.cc:1035] T 8af15fc41246471db31f7824985b5903 P 6c194a6db6c145acafc0f3068b8ce1fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:27:50.908205 28237 consensus_queue.cc:1035] T 8af15fc41246471db31f7824985b5903 P 6c194a6db6c145acafc0f3068b8ce1fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:27:50.908771 24185 raft_consensus.cc:1261] T 8af15fc41246471db31f7824985b5903 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 1 FOLLOWER]: Refusing update from remote peer 6c194a6db6c145acafc0f3068b8ce1fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:27:50.909286 27456 consensus_queue.cc:1035] T dc0e070513cc46ca82c1946423ad91c5 P 6c194a6db6c145acafc0f3068b8ce1fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:27:50.909919 27456 consensus_queue.cc:1035] T 8af15fc41246471db31f7824985b5903 P 6c194a6db6c145acafc0f3068b8ce1fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:27:50.916532 28099 raft_consensus.cc:2932] T dc0e070513cc46ca82c1946423ad91c5 P 6c194a6db6c145acafc0f3068b8ce1fd [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e117afbf8819424e8a98476453dfde2a (127.15.255.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } attrs { replace: true } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } } }
I20240502 21:27:50.921034 26145 raft_consensus.cc:2932] T dc0e070513cc46ca82c1946423ad91c5 P e117afbf8819424e8a98476453dfde2a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e117afbf8819424e8a98476453dfde2a (127.15.255.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } attrs { replace: true } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } } }
I20240502 21:27:50.921269 23916 raft_consensus.cc:2932] T dc0e070513cc46ca82c1946423ad91c5 P 6810a866decb4f5089f7d35c870e4204 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e117afbf8819424e8a98476453dfde2a (127.15.255.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } attrs { replace: true } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } } }
I20240502 21:27:50.921617 23765 catalog_manager.cc:5484] T dc0e070513cc46ca82c1946423ad91c5 P 6c194a6db6c145acafc0f3068b8ce1fd reported cstate change: config changed from index 4 to 5, e117afbf8819424e8a98476453dfde2a (127.15.255.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:27:50.928160 24185 raft_consensus.cc:2932] T dc0e070513cc46ca82c1946423ad91c5 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e117afbf8819424e8a98476453dfde2a (127.15.255.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } attrs { replace: true } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } } }
I20240502 21:27:50.931686 24047 consensus_queue.cc:237] T dc0e070513cc46ca82c1946423ad91c5 P 6c194a6db6c145acafc0f3068b8ce1fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } }
I20240502 21:27:50.937134 24185 raft_consensus.cc:1261] T dc0e070513cc46ca82c1946423ad91c5 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 1 FOLLOWER]: Refusing update from remote peer 6c194a6db6c145acafc0f3068b8ce1fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:27:50.937656 27456 consensus_queue.cc:1035] T dc0e070513cc46ca82c1946423ad91c5 P 6c194a6db6c145acafc0f3068b8ce1fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:27:50.939301 26145 raft_consensus.cc:1261] T dc0e070513cc46ca82c1946423ad91c5 P e117afbf8819424e8a98476453dfde2a [term 1 FOLLOWER]: Refusing update from remote peer 6c194a6db6c145acafc0f3068b8ce1fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:27:50.940339 28231 raft_consensus.cc:2932] T 8af15fc41246471db31f7824985b5903 P 6c194a6db6c145acafc0f3068b8ce1fd [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, fb6004b8640a4906aa0cfe63e383d40e (127.15.255.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } attrs { replace: true } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: false } } }
I20240502 21:27:50.941087 28099 raft_consensus.cc:2932] T dc0e070513cc46ca82c1946423ad91c5 P 6c194a6db6c145acafc0f3068b8ce1fd [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 6810a866decb4f5089f7d35c870e4204 (127.15.255.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } } }
I20240502 21:27:50.942641 26012 raft_consensus.cc:2932] T 8af15fc41246471db31f7824985b5903 P fb6004b8640a4906aa0cfe63e383d40e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, fb6004b8640a4906aa0cfe63e383d40e (127.15.255.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } attrs { replace: true } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: false } } }
I20240502 21:27:50.944542 24184 raft_consensus.cc:2932] T 8af15fc41246471db31f7824985b5903 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, fb6004b8640a4906aa0cfe63e383d40e (127.15.255.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } attrs { replace: true } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: false } } }
I20240502 21:27:50.945899 23768 catalog_manager.cc:5484] T dc0e070513cc46ca82c1946423ad91c5 P 6c194a6db6c145acafc0f3068b8ce1fd reported cstate change: config changed from index 5 to 6, VOTER 6810a866decb4f5089f7d35c870e4204 (127.15.255.129) evicted. New cstate: current_term: 1 leader_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:50.946519 24316 raft_consensus.cc:2932] T 8af15fc41246471db31f7824985b5903 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, fb6004b8640a4906aa0cfe63e383d40e (127.15.255.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } attrs { replace: true } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: false } } }
I20240502 21:27:50.948537 25879 raft_consensus.cc:1203] T 8eba3872a77046ecb9c9477d4cc0e18b P ee27c3f9e0ec46caa742ac386d6b7c20 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 21:27:50.949404 23765 catalog_manager.cc:5484] T 8af15fc41246471db31f7824985b5903 P fb6004b8640a4906aa0cfe63e383d40e reported cstate change: config changed from index 4 to 5, fb6004b8640a4906aa0cfe63e383d40e (127.15.255.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } attrs { replace: true } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: false } } }
I20240502 21:27:50.950078 23894 tablet_service.cc:1514] Processing DeleteTablet for tablet dc0e070513cc46ca82c1946423ad91c5 with delete_type TABLET_DATA_TOMBSTONED (TS 6810a866decb4f5089f7d35c870e4204 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:33512
I20240502 21:27:50.951293 23757 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet dc0e070513cc46ca82c1946423ad91c5 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:27:50.952761 24183 raft_consensus.cc:2932] T dc0e070513cc46ca82c1946423ad91c5 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 6810a866decb4f5089f7d35c870e4204 (127.15.255.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } } }
I20240502 21:27:50.953145 28237 consensus_queue.cc:1035] T dc0e070513cc46ca82c1946423ad91c5 P 6c194a6db6c145acafc0f3068b8ce1fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:27:50.960168 24047 consensus_queue.cc:237] T 8af15fc41246471db31f7824985b5903 P 6c194a6db6c145acafc0f3068b8ce1fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: false } }
I20240502 21:27:50.960296 26145 raft_consensus.cc:2932] T dc0e070513cc46ca82c1946423ad91c5 P e117afbf8819424e8a98476453dfde2a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 6810a866decb4f5089f7d35c870e4204 (127.15.255.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } } }
I20240502 21:27:50.964862 26012 raft_consensus.cc:1261] T 8af15fc41246471db31f7824985b5903 P fb6004b8640a4906aa0cfe63e383d40e [term 1 FOLLOWER]: Refusing update from remote peer 6c194a6db6c145acafc0f3068b8ce1fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:27:50.965232 24183 raft_consensus.cc:1261] T 8af15fc41246471db31f7824985b5903 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 1 FOLLOWER]: Refusing update from remote peer 6c194a6db6c145acafc0f3068b8ce1fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:27:50.965656 28099 consensus_queue.cc:1035] T 8af15fc41246471db31f7824985b5903 P 6c194a6db6c145acafc0f3068b8ce1fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:27:50.966861 28231 consensus_queue.cc:1035] T 8af15fc41246471db31f7824985b5903 P 6c194a6db6c145acafc0f3068b8ce1fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:27:50.968640 28237 raft_consensus.cc:2932] T 8af15fc41246471db31f7824985b5903 P 6c194a6db6c145acafc0f3068b8ce1fd [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 038c5d9d1d7a49c981c4c34fbc00d9e6 (127.15.255.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: false } } }
I20240502 21:27:50.969466 26012 raft_consensus.cc:2932] T 8af15fc41246471db31f7824985b5903 P fb6004b8640a4906aa0cfe63e383d40e [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 038c5d9d1d7a49c981c4c34fbc00d9e6 (127.15.255.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: false } } }
I20240502 21:27:50.970036 28249 tablet_replica.cc:330] T dc0e070513cc46ca82c1946423ad91c5 P 6810a866decb4f5089f7d35c870e4204: stopping tablet replica
I20240502 21:27:50.970631 28249 raft_consensus.cc:2229] T dc0e070513cc46ca82c1946423ad91c5 P 6810a866decb4f5089f7d35c870e4204 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:50.971024 28249 raft_consensus.cc:2258] T dc0e070513cc46ca82c1946423ad91c5 P 6810a866decb4f5089f7d35c870e4204 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:50.973033 23757 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 8af15fc41246471db31f7824985b5903 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:27:50.976114 23768 catalog_manager.cc:5484] T 8af15fc41246471db31f7824985b5903 P fb6004b8640a4906aa0cfe63e383d40e reported cstate change: config changed from index 5 to 6, VOTER 038c5d9d1d7a49c981c4c34fbc00d9e6 (127.15.255.132) evicted. New cstate: current_term: 1 leader_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: false } } }
I20240502 21:27:50.976615 24183 raft_consensus.cc:2932] T 8af15fc41246471db31f7824985b5903 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 038c5d9d1d7a49c981c4c34fbc00d9e6 (127.15.255.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: false } } }
I20240502 21:27:50.980847 24297 tablet_service.cc:1514] Processing DeleteTablet for tablet 8af15fc41246471db31f7824985b5903 with delete_type TABLET_DATA_TOMBSTONED (TS 038c5d9d1d7a49c981c4c34fbc00d9e6 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:58268
I20240502 21:27:50.981835 27349 raft_consensus.cc:1050] T fef55863beb6435dbb25bfb6d71e3cd7 P 038c5d9d1d7a49c981c4c34fbc00d9e6: attempting to promote NON_VOTER a3e500d7077a48acb5edb33f7e178c81 to VOTER
I20240502 21:27:50.982741 27349 consensus_queue.cc:237] T fef55863beb6435dbb25bfb6d71e3cd7 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } attrs { replace: true } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } }
I20240502 21:27:50.986577 23916 raft_consensus.cc:1261] T fef55863beb6435dbb25bfb6d71e3cd7 P 6810a866decb4f5089f7d35c870e4204 [term 1 FOLLOWER]: Refusing update from remote peer 038c5d9d1d7a49c981c4c34fbc00d9e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:27:50.986644 26280 raft_consensus.cc:1261] T fef55863beb6435dbb25bfb6d71e3cd7 P a3e500d7077a48acb5edb33f7e178c81 [term 1 LEARNER]: Refusing update from remote peer 038c5d9d1d7a49c981c4c34fbc00d9e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:27:50.987179 28216 consensus_queue.cc:1035] T fef55863beb6435dbb25bfb6d71e3cd7 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:27:50.987730 24183 raft_consensus.cc:1261] T fef55863beb6435dbb25bfb6d71e3cd7 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 1 FOLLOWER]: Refusing update from remote peer 038c5d9d1d7a49c981c4c34fbc00d9e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:27:50.987915 28216 consensus_queue.cc:1035] T fef55863beb6435dbb25bfb6d71e3cd7 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:27:50.988389 28249 ts_tablet_manager.cc:1875] T dc0e070513cc46ca82c1946423ad91c5 P 6810a866decb4f5089f7d35c870e4204: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:27:50.988600 28216 consensus_queue.cc:1035] T fef55863beb6435dbb25bfb6d71e3cd7 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:27:50.998095 28252 tablet_replica.cc:330] T 8af15fc41246471db31f7824985b5903 P 038c5d9d1d7a49c981c4c34fbc00d9e6: stopping tablet replica
I20240502 21:27:50.998178 28249 ts_tablet_manager.cc:1888] T dc0e070513cc46ca82c1946423ad91c5 P 6810a866decb4f5089f7d35c870e4204: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 21:27:50.998368 28249 log.cc:1196] T dc0e070513cc46ca82c1946423ad91c5 P 6810a866decb4f5089f7d35c870e4204: Deleting WAL directory at /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-0/wal/wals/dc0e070513cc46ca82c1946423ad91c5
I20240502 21:27:50.998839 28252 raft_consensus.cc:2229] T 8af15fc41246471db31f7824985b5903 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:50.999205 28252 raft_consensus.cc:2258] T 8af15fc41246471db31f7824985b5903 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:50.999248 23757 catalog_manager.cc:4830] TS 6810a866decb4f5089f7d35c870e4204 (127.15.255.129:40435): tablet dc0e070513cc46ca82c1946423ad91c5 (table rebalance_test_table_0 [id=1e673d0be4694336bbf65d9c114c614f]) successfully deleted
I20240502 21:27:51.000285 28252 ts_tablet_manager.cc:1875] T 8af15fc41246471db31f7824985b5903 P 038c5d9d1d7a49c981c4c34fbc00d9e6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:27:51.006644 28252 ts_tablet_manager.cc:1888] T 8af15fc41246471db31f7824985b5903 P 038c5d9d1d7a49c981c4c34fbc00d9e6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 21:27:51.006819 28252 log.cc:1196] T 8af15fc41246471db31f7824985b5903 P 038c5d9d1d7a49c981c4c34fbc00d9e6: Deleting WAL directory at /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-3/wal/wals/8af15fc41246471db31f7824985b5903
I20240502 21:27:51.007824 23755 catalog_manager.cc:4830] TS 038c5d9d1d7a49c981c4c34fbc00d9e6 (127.15.255.132:46813): tablet 8af15fc41246471db31f7824985b5903 (table rebalance_test_table_5 [id=e10e5576c0ea45a09c0eab75c2023b78]) successfully deleted
I20240502 21:27:51.008641 28153 raft_consensus.cc:2932] T fef55863beb6435dbb25bfb6d71e3cd7 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a3e500d7077a48acb5edb33f7e178c81 (127.15.255.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } attrs { replace: true } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } } }
I20240502 21:27:51.013260 24184 raft_consensus.cc:2932] T fef55863beb6435dbb25bfb6d71e3cd7 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a3e500d7077a48acb5edb33f7e178c81 (127.15.255.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } attrs { replace: true } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } } }
I20240502 21:27:51.014400 23770 catalog_manager.cc:5484] T fef55863beb6435dbb25bfb6d71e3cd7 P 038c5d9d1d7a49c981c4c34fbc00d9e6 reported cstate change: config changed from index 4 to 5, a3e500d7077a48acb5edb33f7e178c81 (127.15.255.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:27:51.016963 26279 raft_consensus.cc:2932] T fef55863beb6435dbb25bfb6d71e3cd7 P a3e500d7077a48acb5edb33f7e178c81 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a3e500d7077a48acb5edb33f7e178c81 (127.15.255.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } attrs { replace: true } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } } }
I20240502 21:27:51.025059 27919 tablet_copy_client.cc:523] T 256ae04a48b849c4a9086b6b1ff5df19 P e117afbf8819424e8a98476453dfde2a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:27:51.028694 27919 tablet_bootstrap.cc:492] T 256ae04a48b849c4a9086b6b1ff5df19 P e117afbf8819424e8a98476453dfde2a: Bootstrap starting.
I20240502 21:27:51.039806 23916 raft_consensus.cc:2932] T fef55863beb6435dbb25bfb6d71e3cd7 P 6810a866decb4f5089f7d35c870e4204 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, a3e500d7077a48acb5edb33f7e178c81 (127.15.255.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } attrs { replace: true } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } } }
I20240502 21:27:51.042713 24316 consensus_queue.cc:237] T fef55863beb6435dbb25bfb6d71e3cd7 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } }
I20240502 21:27:51.046106 23916 raft_consensus.cc:1261] T fef55863beb6435dbb25bfb6d71e3cd7 P 6810a866decb4f5089f7d35c870e4204 [term 1 FOLLOWER]: Refusing update from remote peer 038c5d9d1d7a49c981c4c34fbc00d9e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:27:51.046211 26280 raft_consensus.cc:1261] T fef55863beb6435dbb25bfb6d71e3cd7 P a3e500d7077a48acb5edb33f7e178c81 [term 1 FOLLOWER]: Refusing update from remote peer 038c5d9d1d7a49c981c4c34fbc00d9e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:27:51.046797 28216 consensus_queue.cc:1035] T fef55863beb6435dbb25bfb6d71e3cd7 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:27:51.047108 28153 consensus_queue.cc:1035] T fef55863beb6435dbb25bfb6d71e3cd7 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:27:51.049856 28216 raft_consensus.cc:2932] T fef55863beb6435dbb25bfb6d71e3cd7 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 009d3e7e8a8b4e2497de3a73d01ced49 (127.15.255.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } } }
I20240502 21:27:51.053555 26280 raft_consensus.cc:2932] T fef55863beb6435dbb25bfb6d71e3cd7 P a3e500d7077a48acb5edb33f7e178c81 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 009d3e7e8a8b4e2497de3a73d01ced49 (127.15.255.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } } }
I20240502 21:27:51.054303 23916 raft_consensus.cc:2932] T fef55863beb6435dbb25bfb6d71e3cd7 P 6810a866decb4f5089f7d35c870e4204 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 009d3e7e8a8b4e2497de3a73d01ced49 (127.15.255.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } } }
I20240502 21:27:51.054955 23755 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet fef55863beb6435dbb25bfb6d71e3cd7 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:27:51.057301 23770 catalog_manager.cc:5484] T fef55863beb6435dbb25bfb6d71e3cd7 P 038c5d9d1d7a49c981c4c34fbc00d9e6 reported cstate change: config changed from index 5 to 6, VOTER 009d3e7e8a8b4e2497de3a73d01ced49 (127.15.255.131) evicted. New cstate: current_term: 1 leader_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:27:51.062170 24162 tablet_service.cc:1514] Processing DeleteTablet for tablet fef55863beb6435dbb25bfb6d71e3cd7 with delete_type TABLET_DATA_TOMBSTONED (TS 009d3e7e8a8b4e2497de3a73d01ced49 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:33102
I20240502 21:27:51.068993 28266 tablet_replica.cc:330] T fef55863beb6435dbb25bfb6d71e3cd7 P 009d3e7e8a8b4e2497de3a73d01ced49: stopping tablet replica
I20240502 21:27:51.069659 28266 raft_consensus.cc:2229] T fef55863beb6435dbb25bfb6d71e3cd7 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:51.069945 28266 raft_consensus.cc:2258] T fef55863beb6435dbb25bfb6d71e3cd7 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:51.072757 28266 ts_tablet_manager.cc:1875] T fef55863beb6435dbb25bfb6d71e3cd7 P 009d3e7e8a8b4e2497de3a73d01ced49: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:27:51.077395 28266 ts_tablet_manager.cc:1888] T fef55863beb6435dbb25bfb6d71e3cd7 P 009d3e7e8a8b4e2497de3a73d01ced49: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 21:27:51.077543 28266 log.cc:1196] T fef55863beb6435dbb25bfb6d71e3cd7 P 009d3e7e8a8b4e2497de3a73d01ced49: Deleting WAL directory at /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-2/wal/wals/fef55863beb6435dbb25bfb6d71e3cd7
I20240502 21:27:51.078286 23756 catalog_manager.cc:4830] TS 009d3e7e8a8b4e2497de3a73d01ced49 (127.15.255.131:37895): tablet fef55863beb6435dbb25bfb6d71e3cd7 (table rebalance_test_table_8 [id=9a4e7bd31db34cf3aff2242876f0de47]) successfully deleted
I20240502 21:27:51.088680 27919 tablet_bootstrap.cc:492] T 256ae04a48b849c4a9086b6b1ff5df19 P e117afbf8819424e8a98476453dfde2a: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:27:51.089066 27919 tablet_bootstrap.cc:492] T 256ae04a48b849c4a9086b6b1ff5df19 P e117afbf8819424e8a98476453dfde2a: Bootstrap complete.
I20240502 21:27:51.089336 27919 ts_tablet_manager.cc:1367] T 256ae04a48b849c4a9086b6b1ff5df19 P e117afbf8819424e8a98476453dfde2a: Time spent bootstrapping tablet: real 0.061s user 0.041s sys 0.005s
I20240502 21:27:51.090493 27919 raft_consensus.cc:348] T 256ae04a48b849c4a9086b6b1ff5df19 P e117afbf8819424e8a98476453dfde2a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } attrs { replace: true } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: NON_VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: true } }
I20240502 21:27:51.090803 27919 raft_consensus.cc:729] T 256ae04a48b849c4a9086b6b1ff5df19 P e117afbf8819424e8a98476453dfde2a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e117afbf8819424e8a98476453dfde2a, State: Initialized, Role: LEARNER
I20240502 21:27:51.091045 27919 consensus_queue.cc:260] T 256ae04a48b849c4a9086b6b1ff5df19 P e117afbf8819424e8a98476453dfde2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } attrs { replace: true } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: NON_VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: true } }
I20240502 21:27:51.092173 27919 ts_tablet_manager.cc:1398] T 256ae04a48b849c4a9086b6b1ff5df19 P e117afbf8819424e8a98476453dfde2a: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 21:27:51.092887 23935 tablet_copy_service.cc:342] P 6810a866decb4f5089f7d35c870e4204: Request end of tablet copy session e117afbf8819424e8a98476453dfde2a-256ae04a48b849c4a9086b6b1ff5df19 received from {username='slave'} at 127.15.255.135:39893
I20240502 21:27:51.093036 23935 tablet_copy_service.cc:433] P 6810a866decb4f5089f7d35c870e4204: ending tablet copy session e117afbf8819424e8a98476453dfde2a-256ae04a48b849c4a9086b6b1ff5df19 on tablet 256ae04a48b849c4a9086b6b1ff5df19 with peer e117afbf8819424e8a98476453dfde2a
I20240502 21:27:51.134017 26145 raft_consensus.cc:1203] T 256ae04a48b849c4a9086b6b1ff5df19 P e117afbf8819424e8a98476453dfde2a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 21:27:51.172991 26145 raft_consensus.cc:1203] T 77fc7226878f44d3884ccb1fc41c44a4 P e117afbf8819424e8a98476453dfde2a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 21:27:51.198411 26145 raft_consensus.cc:1203] T c9b5be01b5e941b1848df79c7a9db348 P e117afbf8819424e8a98476453dfde2a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 21:27:51.344241 28320 raft_consensus.cc:1050] T d79eec4f59634df480161815d888bcac P 009d3e7e8a8b4e2497de3a73d01ced49: attempting to promote NON_VOTER ee27c3f9e0ec46caa742ac386d6b7c20 to VOTER
I20240502 21:27:51.344936 28320 consensus_queue.cc:237] T d79eec4f59634df480161815d888bcac P 009d3e7e8a8b4e2497de3a73d01ced49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } attrs { replace: true } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: false } }
I20240502 21:27:51.347620 24316 raft_consensus.cc:1261] T d79eec4f59634df480161815d888bcac P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 1 FOLLOWER]: Refusing update from remote peer 009d3e7e8a8b4e2497de3a73d01ced49: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:27:51.347718 25879 raft_consensus.cc:1261] T d79eec4f59634df480161815d888bcac P ee27c3f9e0ec46caa742ac386d6b7c20 [term 1 LEARNER]: Refusing update from remote peer 009d3e7e8a8b4e2497de3a73d01ced49: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:27:51.348278 24047 raft_consensus.cc:1261] T d79eec4f59634df480161815d888bcac P 6c194a6db6c145acafc0f3068b8ce1fd [term 1 FOLLOWER]: Refusing update from remote peer 009d3e7e8a8b4e2497de3a73d01ced49: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:27:51.348173 28319 consensus_queue.cc:1035] T d79eec4f59634df480161815d888bcac P 009d3e7e8a8b4e2497de3a73d01ced49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:27:51.348929 28320 consensus_queue.cc:1035] T d79eec4f59634df480161815d888bcac P 009d3e7e8a8b4e2497de3a73d01ced49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:27:51.349241 27842 consensus_queue.cc:1035] T d79eec4f59634df480161815d888bcac P 009d3e7e8a8b4e2497de3a73d01ced49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:27:51.352643 27842 raft_consensus.cc:2932] T d79eec4f59634df480161815d888bcac P 009d3e7e8a8b4e2497de3a73d01ced49 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ee27c3f9e0ec46caa742ac386d6b7c20 (127.15.255.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } attrs { replace: true } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: false } } }
I20240502 21:27:51.353386 24316 raft_consensus.cc:2932] T d79eec4f59634df480161815d888bcac P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ee27c3f9e0ec46caa742ac386d6b7c20 (127.15.255.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } attrs { replace: true } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: false } } }
I20240502 21:27:51.354367 24047 raft_consensus.cc:2932] T d79eec4f59634df480161815d888bcac P 6c194a6db6c145acafc0f3068b8ce1fd [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ee27c3f9e0ec46caa742ac386d6b7c20 (127.15.255.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } attrs { replace: true } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: false } } }
I20240502 21:27:51.356277 25879 raft_consensus.cc:2932] T d79eec4f59634df480161815d888bcac P ee27c3f9e0ec46caa742ac386d6b7c20 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ee27c3f9e0ec46caa742ac386d6b7c20 (127.15.255.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } attrs { replace: true } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: false } } }
I20240502 21:27:51.357558 23770 catalog_manager.cc:5484] T d79eec4f59634df480161815d888bcac P 009d3e7e8a8b4e2497de3a73d01ced49 reported cstate change: config changed from index 4 to 5, ee27c3f9e0ec46caa742ac386d6b7c20 (127.15.255.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:27:51.362152 24183 consensus_queue.cc:237] T d79eec4f59634df480161815d888bcac P 009d3e7e8a8b4e2497de3a73d01ced49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: false } }
I20240502 21:27:51.365886 25879 raft_consensus.cc:1261] T d79eec4f59634df480161815d888bcac P ee27c3f9e0ec46caa742ac386d6b7c20 [term 1 FOLLOWER]: Refusing update from remote peer 009d3e7e8a8b4e2497de3a73d01ced49: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:27:51.366665 28320 consensus_queue.cc:1035] T d79eec4f59634df480161815d888bcac P 009d3e7e8a8b4e2497de3a73d01ced49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:27:51.367663 24316 raft_consensus.cc:1261] T d79eec4f59634df480161815d888bcac P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 1 FOLLOWER]: Refusing update from remote peer 009d3e7e8a8b4e2497de3a73d01ced49: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:27:51.368109 28320 consensus_queue.cc:1035] T d79eec4f59634df480161815d888bcac P 009d3e7e8a8b4e2497de3a73d01ced49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:27:51.369069 27842 raft_consensus.cc:2932] T d79eec4f59634df480161815d888bcac P 009d3e7e8a8b4e2497de3a73d01ced49 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 6c194a6db6c145acafc0f3068b8ce1fd (127.15.255.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: false } } }
I20240502 21:27:51.369670 25879 raft_consensus.cc:2932] T d79eec4f59634df480161815d888bcac P ee27c3f9e0ec46caa742ac386d6b7c20 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 6c194a6db6c145acafc0f3068b8ce1fd (127.15.255.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: false } } }
I20240502 21:27:51.372610 28153 raft_consensus.cc:1050] T 8eba3872a77046ecb9c9477d4cc0e18b P 038c5d9d1d7a49c981c4c34fbc00d9e6: attempting to promote NON_VOTER ee27c3f9e0ec46caa742ac386d6b7c20 to VOTER
I20240502 21:27:51.373440 28153 consensus_queue.cc:237] T 8eba3872a77046ecb9c9477d4cc0e18b P 038c5d9d1d7a49c981c4c34fbc00d9e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } attrs { replace: true } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: false } }
I20240502 21:27:51.373852 24316 raft_consensus.cc:2932] T d79eec4f59634df480161815d888bcac P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 6c194a6db6c145acafc0f3068b8ce1fd (127.15.255.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: false } } }
I20240502 21:27:51.374789 23768 catalog_manager.cc:5484] T d79eec4f59634df480161815d888bcac P ee27c3f9e0ec46caa742ac386d6b7c20 reported cstate change: config changed from index 5 to 6, VOTER 6c194a6db6c145acafc0f3068b8ce1fd (127.15.255.130) evicted. New cstate: current_term: 1 leader_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: false } } }
I20240502 21:27:51.376461 24047 raft_consensus.cc:1261] T 8eba3872a77046ecb9c9477d4cc0e18b P 6c194a6db6c145acafc0f3068b8ce1fd [term 1 FOLLOWER]: Refusing update from remote peer 038c5d9d1d7a49c981c4c34fbc00d9e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:27:51.376631 23916 raft_consensus.cc:1261] T 8eba3872a77046ecb9c9477d4cc0e18b P 6810a866decb4f5089f7d35c870e4204 [term 1 FOLLOWER]: Refusing update from remote peer 038c5d9d1d7a49c981c4c34fbc00d9e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:27:51.376986 27349 consensus_queue.cc:1035] T 8eba3872a77046ecb9c9477d4cc0e18b P 038c5d9d1d7a49c981c4c34fbc00d9e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:27:51.377421 23756 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet d79eec4f59634df480161815d888bcac with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:27:51.377992 25879 raft_consensus.cc:1261] T 8eba3872a77046ecb9c9477d4cc0e18b P ee27c3f9e0ec46caa742ac386d6b7c20 [term 1 LEARNER]: Refusing update from remote peer 038c5d9d1d7a49c981c4c34fbc00d9e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:27:51.379583 27349 consensus_queue.cc:1035] T 8eba3872a77046ecb9c9477d4cc0e18b P 038c5d9d1d7a49c981c4c34fbc00d9e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:27:51.379846 28216 consensus_queue.cc:1035] T 8eba3872a77046ecb9c9477d4cc0e18b P 038c5d9d1d7a49c981c4c34fbc00d9e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:27:51.381801 24027 tablet_service.cc:1514] Processing DeleteTablet for tablet d79eec4f59634df480161815d888bcac with delete_type TABLET_DATA_TOMBSTONED (TS 6c194a6db6c145acafc0f3068b8ce1fd not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:47386
I20240502 21:27:51.382201 28228 tablet_replica.cc:330] T d79eec4f59634df480161815d888bcac P 6c194a6db6c145acafc0f3068b8ce1fd: stopping tablet replica
I20240502 21:27:51.382442 28228 raft_consensus.cc:2229] T d79eec4f59634df480161815d888bcac P 6c194a6db6c145acafc0f3068b8ce1fd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:51.382784 28228 raft_consensus.cc:2258] T d79eec4f59634df480161815d888bcac P 6c194a6db6c145acafc0f3068b8ce1fd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:51.389770 28228 ts_tablet_manager.cc:1875] T d79eec4f59634df480161815d888bcac P 6c194a6db6c145acafc0f3068b8ce1fd: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:27:51.390082 27349 raft_consensus.cc:2932] T 8eba3872a77046ecb9c9477d4cc0e18b P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ee27c3f9e0ec46caa742ac386d6b7c20 (127.15.255.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } attrs { replace: true } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: false } } }
I20240502 21:27:51.392136 23916 raft_consensus.cc:2932] T 8eba3872a77046ecb9c9477d4cc0e18b P 6810a866decb4f5089f7d35c870e4204 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ee27c3f9e0ec46caa742ac386d6b7c20 (127.15.255.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } attrs { replace: true } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: false } } }
I20240502 21:27:51.393502 25878 raft_consensus.cc:2932] T 8eba3872a77046ecb9c9477d4cc0e18b P ee27c3f9e0ec46caa742ac386d6b7c20 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ee27c3f9e0ec46caa742ac386d6b7c20 (127.15.255.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } attrs { replace: true } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: false } } }
I20240502 21:27:51.394042 24046 raft_consensus.cc:2932] T 8eba3872a77046ecb9c9477d4cc0e18b P 6c194a6db6c145acafc0f3068b8ce1fd [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ee27c3f9e0ec46caa742ac386d6b7c20 (127.15.255.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } attrs { replace: true } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: false } } }
I20240502 21:27:51.395586 28228 ts_tablet_manager.cc:1888] T d79eec4f59634df480161815d888bcac P 6c194a6db6c145acafc0f3068b8ce1fd: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 21:27:51.395702 28228 log.cc:1196] T d79eec4f59634df480161815d888bcac P 6c194a6db6c145acafc0f3068b8ce1fd: Deleting WAL directory at /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-1/wal/wals/d79eec4f59634df480161815d888bcac
I20240502 21:27:51.396461 23757 catalog_manager.cc:4830] TS 6c194a6db6c145acafc0f3068b8ce1fd (127.15.255.130:41235): tablet d79eec4f59634df480161815d888bcac (table rebalance_test_table_0 [id=1e673d0be4694336bbf65d9c114c614f]) successfully deleted
I20240502 21:27:51.398315 23765 catalog_manager.cc:5484] T 8eba3872a77046ecb9c9477d4cc0e18b P 038c5d9d1d7a49c981c4c34fbc00d9e6 reported cstate change: config changed from index 4 to 5, ee27c3f9e0ec46caa742ac386d6b7c20 (127.15.255.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:27:51.402810 24316 consensus_queue.cc:237] T 8eba3872a77046ecb9c9477d4cc0e18b P 038c5d9d1d7a49c981c4c34fbc00d9e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: false } }
I20240502 21:27:51.406491 25878 raft_consensus.cc:1261] T 8eba3872a77046ecb9c9477d4cc0e18b P ee27c3f9e0ec46caa742ac386d6b7c20 [term 1 FOLLOWER]: Refusing update from remote peer 038c5d9d1d7a49c981c4c34fbc00d9e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:27:51.406982 28153 consensus_queue.cc:1035] T 8eba3872a77046ecb9c9477d4cc0e18b P 038c5d9d1d7a49c981c4c34fbc00d9e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:27:51.409626 23915 raft_consensus.cc:1261] T 8eba3872a77046ecb9c9477d4cc0e18b P 6810a866decb4f5089f7d35c870e4204 [term 1 FOLLOWER]: Refusing update from remote peer 038c5d9d1d7a49c981c4c34fbc00d9e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:27:51.410498 28153 consensus_queue.cc:1035] T 8eba3872a77046ecb9c9477d4cc0e18b P 038c5d9d1d7a49c981c4c34fbc00d9e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:27:51.413046 28216 raft_consensus.cc:2932] T 8eba3872a77046ecb9c9477d4cc0e18b P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 6c194a6db6c145acafc0f3068b8ce1fd (127.15.255.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: false } } }
I20240502 21:27:51.414052 23915 raft_consensus.cc:2932] T 8eba3872a77046ecb9c9477d4cc0e18b P 6810a866decb4f5089f7d35c870e4204 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 6c194a6db6c145acafc0f3068b8ce1fd (127.15.255.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: false } } }
I20240502 21:27:51.416862 23755 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 8eba3872a77046ecb9c9477d4cc0e18b with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:27:51.417907 25878 raft_consensus.cc:2932] T 8eba3872a77046ecb9c9477d4cc0e18b P ee27c3f9e0ec46caa742ac386d6b7c20 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 6c194a6db6c145acafc0f3068b8ce1fd (127.15.255.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: false } } }
I20240502 21:27:51.419340 23770 catalog_manager.cc:5484] T 8eba3872a77046ecb9c9477d4cc0e18b P 6810a866decb4f5089f7d35c870e4204 reported cstate change: config changed from index 5 to 6, VOTER 6c194a6db6c145acafc0f3068b8ce1fd (127.15.255.130) evicted. New cstate: current_term: 1 leader_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: false } } }
I20240502 21:27:51.424127 24027 tablet_service.cc:1514] Processing DeleteTablet for tablet 8eba3872a77046ecb9c9477d4cc0e18b with delete_type TABLET_DATA_TOMBSTONED (TS 6c194a6db6c145acafc0f3068b8ce1fd not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:47386
I20240502 21:27:51.424489 28228 tablet_replica.cc:330] T 8eba3872a77046ecb9c9477d4cc0e18b P 6c194a6db6c145acafc0f3068b8ce1fd: stopping tablet replica
I20240502 21:27:51.424679 28228 raft_consensus.cc:2229] T 8eba3872a77046ecb9c9477d4cc0e18b P 6c194a6db6c145acafc0f3068b8ce1fd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:51.424999 28228 raft_consensus.cc:2258] T 8eba3872a77046ecb9c9477d4cc0e18b P 6c194a6db6c145acafc0f3068b8ce1fd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:51.425873 28228 ts_tablet_manager.cc:1875] T 8eba3872a77046ecb9c9477d4cc0e18b P 6c194a6db6c145acafc0f3068b8ce1fd: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:27:51.433506 28228 ts_tablet_manager.cc:1888] T 8eba3872a77046ecb9c9477d4cc0e18b P 6c194a6db6c145acafc0f3068b8ce1fd: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 21:27:51.433619 28228 log.cc:1196] T 8eba3872a77046ecb9c9477d4cc0e18b P 6c194a6db6c145acafc0f3068b8ce1fd: Deleting WAL directory at /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-1/wal/wals/8eba3872a77046ecb9c9477d4cc0e18b
I20240502 21:27:51.434345 23757 catalog_manager.cc:4830] TS 6c194a6db6c145acafc0f3068b8ce1fd (127.15.255.130:41235): tablet 8eba3872a77046ecb9c9477d4cc0e18b (table rebalance_test_table_9 [id=188ff51c46164f47803c423463c2e6e2]) successfully deleted
I20240502 21:27:51.523126 28371 raft_consensus.cc:1050] T 256ae04a48b849c4a9086b6b1ff5df19 P 6810a866decb4f5089f7d35c870e4204: attempting to promote NON_VOTER e117afbf8819424e8a98476453dfde2a to VOTER
I20240502 21:27:51.523787 28371 consensus_queue.cc:237] T 256ae04a48b849c4a9086b6b1ff5df19 P 6810a866decb4f5089f7d35c870e4204 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } attrs { replace: true } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } }
I20240502 21:27:51.526116 26145 raft_consensus.cc:1261] T 256ae04a48b849c4a9086b6b1ff5df19 P e117afbf8819424e8a98476453dfde2a [term 1 LEARNER]: Refusing update from remote peer 6810a866decb4f5089f7d35c870e4204: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:27:51.526439 24183 raft_consensus.cc:1261] T 256ae04a48b849c4a9086b6b1ff5df19 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 1 FOLLOWER]: Refusing update from remote peer 6810a866decb4f5089f7d35c870e4204: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:27:51.528260 24046 raft_consensus.cc:1261] T 256ae04a48b849c4a9086b6b1ff5df19 P 6c194a6db6c145acafc0f3068b8ce1fd [term 1 FOLLOWER]: Refusing update from remote peer 6810a866decb4f5089f7d35c870e4204: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:27:51.528630 27234 consensus_queue.cc:1035] T 256ae04a48b849c4a9086b6b1ff5df19 P 6810a866decb4f5089f7d35c870e4204 [LEADER]: Connected to new peer: Peer: permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:27:51.529223 27234 consensus_queue.cc:1035] T 256ae04a48b849c4a9086b6b1ff5df19 P 6810a866decb4f5089f7d35c870e4204 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:27:51.531087 27234 consensus_queue.cc:1035] T 256ae04a48b849c4a9086b6b1ff5df19 P 6810a866decb4f5089f7d35c870e4204 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:27:51.542726 27234 raft_consensus.cc:2932] T 256ae04a48b849c4a9086b6b1ff5df19 P 6810a866decb4f5089f7d35c870e4204 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e117afbf8819424e8a98476453dfde2a (127.15.255.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } attrs { replace: true } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } } }
I20240502 21:27:51.543540 24183 raft_consensus.cc:2932] T 256ae04a48b849c4a9086b6b1ff5df19 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e117afbf8819424e8a98476453dfde2a (127.15.255.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } attrs { replace: true } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } } }
I20240502 21:27:51.548197 26145 raft_consensus.cc:2932] T 256ae04a48b849c4a9086b6b1ff5df19 P e117afbf8819424e8a98476453dfde2a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e117afbf8819424e8a98476453dfde2a (127.15.255.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } attrs { replace: true } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } } }
I20240502 21:27:51.548483 23765 catalog_manager.cc:5484] T 256ae04a48b849c4a9086b6b1ff5df19 P 009d3e7e8a8b4e2497de3a73d01ced49 reported cstate change: config changed from index 4 to 5, e117afbf8819424e8a98476453dfde2a (127.15.255.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "6810a866decb4f5089f7d35c870e4204" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } attrs { replace: true } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } } }
I20240502 21:27:51.553727 23916 consensus_queue.cc:237] T 256ae04a48b849c4a9086b6b1ff5df19 P 6810a866decb4f5089f7d35c870e4204 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } }
I20240502 21:27:51.553717 24046 raft_consensus.cc:2932] T 256ae04a48b849c4a9086b6b1ff5df19 P 6c194a6db6c145acafc0f3068b8ce1fd [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e117afbf8819424e8a98476453dfde2a (127.15.255.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } attrs { replace: true } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } } }
I20240502 21:27:51.557528 26145 raft_consensus.cc:1261] T 256ae04a48b849c4a9086b6b1ff5df19 P e117afbf8819424e8a98476453dfde2a [term 1 FOLLOWER]: Refusing update from remote peer 6810a866decb4f5089f7d35c870e4204: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:27:51.558545 27234 consensus_queue.cc:1035] T 256ae04a48b849c4a9086b6b1ff5df19 P 6810a866decb4f5089f7d35c870e4204 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:27:51.558773 24183 raft_consensus.cc:1261] T 256ae04a48b849c4a9086b6b1ff5df19 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 1 FOLLOWER]: Refusing update from remote peer 6810a866decb4f5089f7d35c870e4204: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:27:51.559247 28372 consensus_queue.cc:1035] T 256ae04a48b849c4a9086b6b1ff5df19 P 6810a866decb4f5089f7d35c870e4204 [LEADER]: Connected to new peer: Peer: permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:27:51.565522 28372 raft_consensus.cc:2932] T 256ae04a48b849c4a9086b6b1ff5df19 P 6810a866decb4f5089f7d35c870e4204 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 6c194a6db6c145acafc0f3068b8ce1fd (127.15.255.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } } }
I20240502 21:27:51.566607 26145 raft_consensus.cc:2932] T 256ae04a48b849c4a9086b6b1ff5df19 P e117afbf8819424e8a98476453dfde2a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 6c194a6db6c145acafc0f3068b8ce1fd (127.15.255.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } } }
I20240502 21:27:51.569636 23757 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 256ae04a48b849c4a9086b6b1ff5df19 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:27:51.570334 24183 raft_consensus.cc:2932] T 256ae04a48b849c4a9086b6b1ff5df19 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 6c194a6db6c145acafc0f3068b8ce1fd (127.15.255.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } } }
I20240502 21:27:51.571722 23765 catalog_manager.cc:5484] T 256ae04a48b849c4a9086b6b1ff5df19 P 6810a866decb4f5089f7d35c870e4204 reported cstate change: config changed from index 5 to 6, VOTER 6c194a6db6c145acafc0f3068b8ce1fd (127.15.255.130) evicted. New cstate: current_term: 1 leader_uuid: "6810a866decb4f5089f7d35c870e4204" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:27:51.578698 24027 tablet_service.cc:1514] Processing DeleteTablet for tablet 256ae04a48b849c4a9086b6b1ff5df19 with delete_type TABLET_DATA_TOMBSTONED (TS 6c194a6db6c145acafc0f3068b8ce1fd not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:47386
I20240502 21:27:51.579071 28228 tablet_replica.cc:330] T 256ae04a48b849c4a9086b6b1ff5df19 P 6c194a6db6c145acafc0f3068b8ce1fd: stopping tablet replica
I20240502 21:27:51.579270 28228 raft_consensus.cc:2229] T 256ae04a48b849c4a9086b6b1ff5df19 P 6c194a6db6c145acafc0f3068b8ce1fd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:51.579602 28228 raft_consensus.cc:2258] T 256ae04a48b849c4a9086b6b1ff5df19 P 6c194a6db6c145acafc0f3068b8ce1fd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:51.581180 28228 ts_tablet_manager.cc:1875] T 256ae04a48b849c4a9086b6b1ff5df19 P 6c194a6db6c145acafc0f3068b8ce1fd: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:27:51.586196 28228 ts_tablet_manager.cc:1888] T 256ae04a48b849c4a9086b6b1ff5df19 P 6c194a6db6c145acafc0f3068b8ce1fd: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 21:27:51.586308 28228 log.cc:1196] T 256ae04a48b849c4a9086b6b1ff5df19 P 6c194a6db6c145acafc0f3068b8ce1fd: Deleting WAL directory at /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-1/wal/wals/256ae04a48b849c4a9086b6b1ff5df19
I20240502 21:27:51.587074 23757 catalog_manager.cc:4830] TS 6c194a6db6c145acafc0f3068b8ce1fd (127.15.255.130:41235): tablet 256ae04a48b849c4a9086b6b1ff5df19 (table rebalance_test_table_6 [id=9aaf56e5a31f42a79b45624cc2c43779]) successfully deleted
I20240502 21:27:51.602171 27456 raft_consensus.cc:1050] T c9b5be01b5e941b1848df79c7a9db348 P 6c194a6db6c145acafc0f3068b8ce1fd: attempting to promote NON_VOTER e117afbf8819424e8a98476453dfde2a to VOTER
I20240502 21:27:51.602790 27456 consensus_queue.cc:237] T c9b5be01b5e941b1848df79c7a9db348 P 6c194a6db6c145acafc0f3068b8ce1fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } attrs { replace: true } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } }
I20240502 21:27:51.604887 26145 raft_consensus.cc:1261] T c9b5be01b5e941b1848df79c7a9db348 P e117afbf8819424e8a98476453dfde2a [term 1 LEARNER]: Refusing update from remote peer 6c194a6db6c145acafc0f3068b8ce1fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:27:51.605166 24316 raft_consensus.cc:1261] T c9b5be01b5e941b1848df79c7a9db348 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 1 FOLLOWER]: Refusing update from remote peer 6c194a6db6c145acafc0f3068b8ce1fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:27:51.605319 23916 raft_consensus.cc:1261] T c9b5be01b5e941b1848df79c7a9db348 P 6810a866decb4f5089f7d35c870e4204 [term 1 FOLLOWER]: Refusing update from remote peer 6c194a6db6c145acafc0f3068b8ce1fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:27:51.605433 28379 consensus_queue.cc:1035] T c9b5be01b5e941b1848df79c7a9db348 P 6c194a6db6c145acafc0f3068b8ce1fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:27:51.605847 28231 consensus_queue.cc:1035] T c9b5be01b5e941b1848df79c7a9db348 P 6c194a6db6c145acafc0f3068b8ce1fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:27:51.606115 27456 consensus_queue.cc:1035] T c9b5be01b5e941b1848df79c7a9db348 P 6c194a6db6c145acafc0f3068b8ce1fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:27:51.620152 27456 raft_consensus.cc:2932] T c9b5be01b5e941b1848df79c7a9db348 P 6c194a6db6c145acafc0f3068b8ce1fd [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e117afbf8819424e8a98476453dfde2a (127.15.255.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } attrs { replace: true } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } } }
I20240502 21:27:51.620826 24317 raft_consensus.cc:2932] T c9b5be01b5e941b1848df79c7a9db348 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e117afbf8819424e8a98476453dfde2a (127.15.255.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } attrs { replace: true } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } } }
I20240502 21:27:51.621441 23916 raft_consensus.cc:2932] T c9b5be01b5e941b1848df79c7a9db348 P 6810a866decb4f5089f7d35c870e4204 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e117afbf8819424e8a98476453dfde2a (127.15.255.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } attrs { replace: true } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } } }
I20240502 21:27:51.624673 26145 raft_consensus.cc:2932] T c9b5be01b5e941b1848df79c7a9db348 P e117afbf8819424e8a98476453dfde2a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e117afbf8819424e8a98476453dfde2a (127.15.255.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } attrs { replace: true } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } } }
I20240502 21:27:51.625734 23768 catalog_manager.cc:5484] T c9b5be01b5e941b1848df79c7a9db348 P 038c5d9d1d7a49c981c4c34fbc00d9e6 reported cstate change: config changed from index 4 to 5, e117afbf8819424e8a98476453dfde2a (127.15.255.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } attrs { replace: true } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } } }
I20240502 21:27:51.630923 24046 consensus_queue.cc:237] T c9b5be01b5e941b1848df79c7a9db348 P 6c194a6db6c145acafc0f3068b8ce1fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } }
I20240502 21:27:51.632829 26145 raft_consensus.cc:1261] T c9b5be01b5e941b1848df79c7a9db348 P e117afbf8819424e8a98476453dfde2a [term 1 FOLLOWER]: Refusing update from remote peer 6c194a6db6c145acafc0f3068b8ce1fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:27:51.632925 23916 raft_consensus.cc:1261] T c9b5be01b5e941b1848df79c7a9db348 P 6810a866decb4f5089f7d35c870e4204 [term 1 FOLLOWER]: Refusing update from remote peer 6c194a6db6c145acafc0f3068b8ce1fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:27:51.633318 27456 consensus_queue.cc:1035] T c9b5be01b5e941b1848df79c7a9db348 P 6c194a6db6c145acafc0f3068b8ce1fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:27:51.633591 28231 consensus_queue.cc:1035] T c9b5be01b5e941b1848df79c7a9db348 P 6c194a6db6c145acafc0f3068b8ce1fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:27:51.635465 28379 raft_consensus.cc:2932] T c9b5be01b5e941b1848df79c7a9db348 P 6c194a6db6c145acafc0f3068b8ce1fd [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 038c5d9d1d7a49c981c4c34fbc00d9e6 (127.15.255.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } } }
I20240502 21:27:51.636085 26145 raft_consensus.cc:2932] T c9b5be01b5e941b1848df79c7a9db348 P e117afbf8819424e8a98476453dfde2a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 038c5d9d1d7a49c981c4c34fbc00d9e6 (127.15.255.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } } }
I20240502 21:27:51.636070 23916 raft_consensus.cc:2932] T c9b5be01b5e941b1848df79c7a9db348 P 6810a866decb4f5089f7d35c870e4204 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 038c5d9d1d7a49c981c4c34fbc00d9e6 (127.15.255.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } } }
I20240502 21:27:51.639251 23757 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c9b5be01b5e941b1848df79c7a9db348 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:27:51.641211 23765 catalog_manager.cc:5484] T c9b5be01b5e941b1848df79c7a9db348 P e117afbf8819424e8a98476453dfde2a reported cstate change: config changed from index 5 to 6, VOTER 038c5d9d1d7a49c981c4c34fbc00d9e6 (127.15.255.132) evicted. New cstate: current_term: 1 leader_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } } }
I20240502 21:27:51.644740 24297 tablet_service.cc:1514] Processing DeleteTablet for tablet c9b5be01b5e941b1848df79c7a9db348 with delete_type TABLET_DATA_TOMBSTONED (TS 038c5d9d1d7a49c981c4c34fbc00d9e6 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:58268
I20240502 21:27:51.645524 28384 tablet_replica.cc:330] T c9b5be01b5e941b1848df79c7a9db348 P 038c5d9d1d7a49c981c4c34fbc00d9e6: stopping tablet replica
I20240502 21:27:51.645764 28384 raft_consensus.cc:2229] T c9b5be01b5e941b1848df79c7a9db348 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:51.646021 28384 raft_consensus.cc:2258] T c9b5be01b5e941b1848df79c7a9db348 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:51.646952 28384 ts_tablet_manager.cc:1875] T c9b5be01b5e941b1848df79c7a9db348 P 038c5d9d1d7a49c981c4c34fbc00d9e6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:27:51.651475 28384 ts_tablet_manager.cc:1888] T c9b5be01b5e941b1848df79c7a9db348 P 038c5d9d1d7a49c981c4c34fbc00d9e6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 21:27:51.651587 28384 log.cc:1196] T c9b5be01b5e941b1848df79c7a9db348 P 038c5d9d1d7a49c981c4c34fbc00d9e6: Deleting WAL directory at /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-3/wal/wals/c9b5be01b5e941b1848df79c7a9db348
I20240502 21:27:51.652321 23755 catalog_manager.cc:4830] TS 038c5d9d1d7a49c981c4c34fbc00d9e6 (127.15.255.132:46813): tablet c9b5be01b5e941b1848df79c7a9db348 (table rebalance_test_table_2 [id=85ca987b38d84a7db5119dfc88b4feaf]) successfully deleted
I20240502 21:27:51.703526 28379 raft_consensus.cc:1050] T 77fc7226878f44d3884ccb1fc41c44a4 P 6c194a6db6c145acafc0f3068b8ce1fd: attempting to promote NON_VOTER e117afbf8819424e8a98476453dfde2a to VOTER
I20240502 21:27:51.704291 28379 consensus_queue.cc:237] T 77fc7226878f44d3884ccb1fc41c44a4 P 6c194a6db6c145acafc0f3068b8ce1fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } attrs { replace: true } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } }
I20240502 21:27:51.707175 24317 raft_consensus.cc:1261] T 77fc7226878f44d3884ccb1fc41c44a4 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 1 FOLLOWER]: Refusing update from remote peer 6c194a6db6c145acafc0f3068b8ce1fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:27:51.707203 23916 raft_consensus.cc:1261] T 77fc7226878f44d3884ccb1fc41c44a4 P 6810a866decb4f5089f7d35c870e4204 [term 1 FOLLOWER]: Refusing update from remote peer 6c194a6db6c145acafc0f3068b8ce1fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:27:51.707388 26145 raft_consensus.cc:1261] T 77fc7226878f44d3884ccb1fc41c44a4 P e117afbf8819424e8a98476453dfde2a [term 1 LEARNER]: Refusing update from remote peer 6c194a6db6c145acafc0f3068b8ce1fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:27:51.707986 28379 consensus_queue.cc:1035] T 77fc7226878f44d3884ccb1fc41c44a4 P 6c194a6db6c145acafc0f3068b8ce1fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:27:51.708284 28231 consensus_queue.cc:1035] T 77fc7226878f44d3884ccb1fc41c44a4 P 6c194a6db6c145acafc0f3068b8ce1fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:27:51.708580 27456 consensus_queue.cc:1035] T 77fc7226878f44d3884ccb1fc41c44a4 P 6c194a6db6c145acafc0f3068b8ce1fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:27:51.725751 27456 raft_consensus.cc:2932] T 77fc7226878f44d3884ccb1fc41c44a4 P 6c194a6db6c145acafc0f3068b8ce1fd [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e117afbf8819424e8a98476453dfde2a (127.15.255.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } attrs { replace: true } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } } }
I20240502 21:27:51.728967 26145 raft_consensus.cc:2932] T 77fc7226878f44d3884ccb1fc41c44a4 P e117afbf8819424e8a98476453dfde2a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e117afbf8819424e8a98476453dfde2a (127.15.255.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } attrs { replace: true } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } } }
I20240502 21:27:51.729108 24317 raft_consensus.cc:2932] T 77fc7226878f44d3884ccb1fc41c44a4 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e117afbf8819424e8a98476453dfde2a (127.15.255.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } attrs { replace: true } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } } }
I20240502 21:27:51.730273 23765 catalog_manager.cc:5484] T 77fc7226878f44d3884ccb1fc41c44a4 P 6c194a6db6c145acafc0f3068b8ce1fd reported cstate change: config changed from index 4 to 5, e117afbf8819424e8a98476453dfde2a (127.15.255.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:27:51.738586 24046 consensus_queue.cc:237] T 77fc7226878f44d3884ccb1fc41c44a4 P 6c194a6db6c145acafc0f3068b8ce1fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } }
I20240502 21:27:51.736690 23916 raft_consensus.cc:2932] T 77fc7226878f44d3884ccb1fc41c44a4 P 6810a866decb4f5089f7d35c870e4204 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e117afbf8819424e8a98476453dfde2a (127.15.255.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } attrs { replace: true } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } } }
I20240502 21:27:51.740190 26145 raft_consensus.cc:1261] T 77fc7226878f44d3884ccb1fc41c44a4 P e117afbf8819424e8a98476453dfde2a [term 1 FOLLOWER]: Refusing update from remote peer 6c194a6db6c145acafc0f3068b8ce1fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:27:51.740748 28379 consensus_queue.cc:1035] T 77fc7226878f44d3884ccb1fc41c44a4 P 6c194a6db6c145acafc0f3068b8ce1fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:27:51.743264 28379 raft_consensus.cc:2932] T 77fc7226878f44d3884ccb1fc41c44a4 P 6c194a6db6c145acafc0f3068b8ce1fd [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 038c5d9d1d7a49c981c4c34fbc00d9e6 (127.15.255.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } } }
I20240502 21:27:51.745769 23915 raft_consensus.cc:1261] T 77fc7226878f44d3884ccb1fc41c44a4 P 6810a866decb4f5089f7d35c870e4204 [term 1 FOLLOWER]: Refusing update from remote peer 6c194a6db6c145acafc0f3068b8ce1fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:27:51.746547 27456 consensus_queue.cc:1035] T 77fc7226878f44d3884ccb1fc41c44a4 P 6c194a6db6c145acafc0f3068b8ce1fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:27:51.747471 26147 raft_consensus.cc:2932] T 77fc7226878f44d3884ccb1fc41c44a4 P e117afbf8819424e8a98476453dfde2a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 038c5d9d1d7a49c981c4c34fbc00d9e6 (127.15.255.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } } }
I20240502 21:27:51.749336 23916 raft_consensus.cc:2932] T 77fc7226878f44d3884ccb1fc41c44a4 P 6810a866decb4f5089f7d35c870e4204 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 038c5d9d1d7a49c981c4c34fbc00d9e6 (127.15.255.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } } }
I20240502 21:27:51.750389 23757 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 77fc7226878f44d3884ccb1fc41c44a4 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:27:51.751128 23770 catalog_manager.cc:5484] T 77fc7226878f44d3884ccb1fc41c44a4 P 6c194a6db6c145acafc0f3068b8ce1fd reported cstate change: config changed from index 5 to 6, VOTER 038c5d9d1d7a49c981c4c34fbc00d9e6 (127.15.255.132) evicted. New cstate: current_term: 1 leader_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:27:51.755281 24297 tablet_service.cc:1514] Processing DeleteTablet for tablet 77fc7226878f44d3884ccb1fc41c44a4 with delete_type TABLET_DATA_TOMBSTONED (TS 038c5d9d1d7a49c981c4c34fbc00d9e6 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:58268
I20240502 21:27:51.755649 28384 tablet_replica.cc:330] T 77fc7226878f44d3884ccb1fc41c44a4 P 038c5d9d1d7a49c981c4c34fbc00d9e6: stopping tablet replica
I20240502 21:27:51.755856 28384 raft_consensus.cc:2229] T 77fc7226878f44d3884ccb1fc41c44a4 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:51.756186 28384 raft_consensus.cc:2258] T 77fc7226878f44d3884ccb1fc41c44a4 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:51.757090 28384 ts_tablet_manager.cc:1875] T 77fc7226878f44d3884ccb1fc41c44a4 P 038c5d9d1d7a49c981c4c34fbc00d9e6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:27:51.761189 28384 ts_tablet_manager.cc:1888] T 77fc7226878f44d3884ccb1fc41c44a4 P 038c5d9d1d7a49c981c4c34fbc00d9e6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 21:27:51.761305 28384 log.cc:1196] T 77fc7226878f44d3884ccb1fc41c44a4 P 038c5d9d1d7a49c981c4c34fbc00d9e6: Deleting WAL directory at /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-3/wal/wals/77fc7226878f44d3884ccb1fc41c44a4
I20240502 21:27:51.762120 23755 catalog_manager.cc:4830] TS 038c5d9d1d7a49c981c4c34fbc00d9e6 (127.15.255.132:46813): tablet 77fc7226878f44d3884ccb1fc41c44a4 (table rebalance_test_table_7 [id=754dbb3bce6a4875a0b642faf5315709]) successfully deleted
I20240502 21:27:51.891309 24162 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:27:51.917794 24297 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:27:51.921425 23894 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 21:27:51.937824 26260 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:27:51.942088 27104 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:51.958917 24027 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:27:51.998656 25859 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:52.015645 26127 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:27:52.030427 25992 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:27:52.112538 24046 consensus_queue.cc:237] T c9b5be01b5e941b1848df79c7a9db348 P 6c194a6db6c145acafc0f3068b8ce1fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } attrs { replace: true } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } } peers { permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: NON_VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: true } }
I20240502 21:27:52.114728 26147 raft_consensus.cc:1261] T c9b5be01b5e941b1848df79c7a9db348 P e117afbf8819424e8a98476453dfde2a [term 1 FOLLOWER]: Refusing update from remote peer 6c194a6db6c145acafc0f3068b8ce1fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:27:52.115263 28099 consensus_queue.cc:1035] T c9b5be01b5e941b1848df79c7a9db348 P 6c194a6db6c145acafc0f3068b8ce1fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:27:52.116041 23916 raft_consensus.cc:1261] T c9b5be01b5e941b1848df79c7a9db348 P 6810a866decb4f5089f7d35c870e4204 [term 1 FOLLOWER]: Refusing update from remote peer 6c194a6db6c145acafc0f3068b8ce1fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:27:52.117205 28099 consensus_queue.cc:1035] T c9b5be01b5e941b1848df79c7a9db348 P 6c194a6db6c145acafc0f3068b8ce1fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:27:52.118814 28379 raft_consensus.cc:2932] T c9b5be01b5e941b1848df79c7a9db348 P 6c194a6db6c145acafc0f3068b8ce1fd [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 4f58800ba2ed45eabd81c2f2e691aacd (127.15.255.137) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } attrs { replace: true } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } } peers { permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: NON_VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: true } } }
I20240502 21:27:52.122515 26147 raft_consensus.cc:2932] T c9b5be01b5e941b1848df79c7a9db348 P e117afbf8819424e8a98476453dfde2a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 4f58800ba2ed45eabd81c2f2e691aacd (127.15.255.137) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } attrs { replace: true } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } } peers { permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: NON_VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: true } } }
I20240502 21:27:52.122989 23916 raft_consensus.cc:2932] T c9b5be01b5e941b1848df79c7a9db348 P 6810a866decb4f5089f7d35c870e4204 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 4f58800ba2ed45eabd81c2f2e691aacd (127.15.255.137) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } attrs { replace: true } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } } peers { permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: NON_VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: true } } }
W20240502 21:27:52.123484 23981 consensus_peers.cc:485] T c9b5be01b5e941b1848df79c7a9db348 P 6c194a6db6c145acafc0f3068b8ce1fd -> Peer 4f58800ba2ed45eabd81c2f2e691aacd (127.15.255.137:40161): Couldn't send request to peer 4f58800ba2ed45eabd81c2f2e691aacd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c9b5be01b5e941b1848df79c7a9db348. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:52.128623 23770 catalog_manager.cc:5484] T c9b5be01b5e941b1848df79c7a9db348 P 6810a866decb4f5089f7d35c870e4204 reported cstate change: config changed from index 6 to 7, NON_VOTER 4f58800ba2ed45eabd81c2f2e691aacd (127.15.255.137) added. New cstate: current_term: 1 leader_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } attrs { replace: true } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } } peers { permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: NON_VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: true } } }
I20240502 21:27:52.137991 24317 consensus_queue.cc:237] T 11cb9a2fbda946fb9f303f2c465bccb6 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } attrs { replace: true } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: true } }
I20240502 21:27:52.140139 26147 raft_consensus.cc:1261] T 11cb9a2fbda946fb9f303f2c465bccb6 P e117afbf8819424e8a98476453dfde2a [term 1 FOLLOWER]: Refusing update from remote peer 038c5d9d1d7a49c981c4c34fbc00d9e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:27:52.140414 24183 raft_consensus.cc:1261] T 11cb9a2fbda946fb9f303f2c465bccb6 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 1 FOLLOWER]: Refusing update from remote peer 038c5d9d1d7a49c981c4c34fbc00d9e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:27:52.140640 28460 consensus_queue.cc:1035] T 11cb9a2fbda946fb9f303f2c465bccb6 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:27:52.141024 28216 consensus_queue.cc:1035] T 11cb9a2fbda946fb9f303f2c465bccb6 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240502 21:27:52.141465 24252 consensus_peers.cc:485] T 11cb9a2fbda946fb9f303f2c465bccb6 P 038c5d9d1d7a49c981c4c34fbc00d9e6 -> Peer a3e500d7077a48acb5edb33f7e178c81 (127.15.255.136:43595): Couldn't send request to peer a3e500d7077a48acb5edb33f7e178c81. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 11cb9a2fbda946fb9f303f2c465bccb6. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:52.143796 28460 raft_consensus.cc:2932] T 11cb9a2fbda946fb9f303f2c465bccb6 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER a3e500d7077a48acb5edb33f7e178c81 (127.15.255.136) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } attrs { replace: true } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: true } } }
I20240502 21:27:52.144160 24183 raft_consensus.cc:2932] T 11cb9a2fbda946fb9f303f2c465bccb6 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER a3e500d7077a48acb5edb33f7e178c81 (127.15.255.136) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } attrs { replace: true } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: true } } }
I20240502 21:27:52.144924 26147 raft_consensus.cc:2932] T 11cb9a2fbda946fb9f303f2c465bccb6 P e117afbf8819424e8a98476453dfde2a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER a3e500d7077a48acb5edb33f7e178c81 (127.15.255.136) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } attrs { replace: true } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: true } } }
I20240502 21:27:52.148989 23766 catalog_manager.cc:5484] T 11cb9a2fbda946fb9f303f2c465bccb6 P 038c5d9d1d7a49c981c4c34fbc00d9e6 reported cstate change: config changed from index 6 to 7, NON_VOTER a3e500d7077a48acb5edb33f7e178c81 (127.15.255.136) added. New cstate: current_term: 1 leader_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:52.162508 23915 consensus_queue.cc:237] T adf80b9d71314010a7a4cf052fd8a511 P 6810a866decb4f5089f7d35c870e4204 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } attrs { replace: true } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: NON_VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: true } }
I20240502 21:27:52.164746 26280 raft_consensus.cc:1261] T adf80b9d71314010a7a4cf052fd8a511 P a3e500d7077a48acb5edb33f7e178c81 [term 1 FOLLOWER]: Refusing update from remote peer 6810a866decb4f5089f7d35c870e4204: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:27:52.165300 28374 consensus_queue.cc:1035] T adf80b9d71314010a7a4cf052fd8a511 P 6810a866decb4f5089f7d35c870e4204 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240502 21:27:52.165640 23847 consensus_peers.cc:485] T adf80b9d71314010a7a4cf052fd8a511 P 6810a866decb4f5089f7d35c870e4204 -> Peer fb6004b8640a4906aa0cfe63e383d40e (127.15.255.134:32825): Couldn't send request to peer fb6004b8640a4906aa0cfe63e383d40e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: adf80b9d71314010a7a4cf052fd8a511. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:52.165912 24317 raft_consensus.cc:1261] T adf80b9d71314010a7a4cf052fd8a511 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 1 FOLLOWER]: Refusing update from remote peer 6810a866decb4f5089f7d35c870e4204: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:27:52.166402 28374 consensus_queue.cc:1035] T adf80b9d71314010a7a4cf052fd8a511 P 6810a866decb4f5089f7d35c870e4204 [LEADER]: Connected to new peer: Peer: permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:27:52.168246 28374 raft_consensus.cc:2932] T adf80b9d71314010a7a4cf052fd8a511 P 6810a866decb4f5089f7d35c870e4204 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER fb6004b8640a4906aa0cfe63e383d40e (127.15.255.134) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } attrs { replace: true } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: NON_VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: true } } }
I20240502 21:27:52.174432 24317 raft_consensus.cc:2932] T adf80b9d71314010a7a4cf052fd8a511 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER fb6004b8640a4906aa0cfe63e383d40e (127.15.255.134) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } attrs { replace: true } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: NON_VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: true } } }
I20240502 21:27:52.178150 26280 raft_consensus.cc:2932] T adf80b9d71314010a7a4cf052fd8a511 P a3e500d7077a48acb5edb33f7e178c81 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER fb6004b8640a4906aa0cfe63e383d40e (127.15.255.134) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } attrs { replace: true } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: NON_VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: true } } }
I20240502 21:27:52.179834 23771 catalog_manager.cc:5484] T adf80b9d71314010a7a4cf052fd8a511 P 038c5d9d1d7a49c981c4c34fbc00d9e6 reported cstate change: config changed from index 6 to 7, NON_VOTER fb6004b8640a4906aa0cfe63e383d40e (127.15.255.134) added. New cstate: current_term: 1 leader_uuid: "6810a866decb4f5089f7d35c870e4204" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } attrs { replace: true } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: NON_VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: true } } }
I20240502 21:27:52.191452 23915 consensus_queue.cc:237] T 549666d19e724a96801db183a07809d8 P 6810a866decb4f5089f7d35c870e4204 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } attrs { replace: true } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: NON_VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: true } }
I20240502 21:27:52.193850 26280 raft_consensus.cc:1261] T 549666d19e724a96801db183a07809d8 P a3e500d7077a48acb5edb33f7e178c81 [term 1 FOLLOWER]: Refusing update from remote peer 6810a866decb4f5089f7d35c870e4204: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:27:52.194046 24046 raft_consensus.cc:1261] T 549666d19e724a96801db183a07809d8 P 6c194a6db6c145acafc0f3068b8ce1fd [term 1 FOLLOWER]: Refusing update from remote peer 6810a866decb4f5089f7d35c870e4204: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:27:52.194453 28470 consensus_queue.cc:1035] T 549666d19e724a96801db183a07809d8 P 6810a866decb4f5089f7d35c870e4204 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240502 21:27:52.194834 23847 consensus_peers.cc:485] T 549666d19e724a96801db183a07809d8 P 6810a866decb4f5089f7d35c870e4204 -> Peer ee27c3f9e0ec46caa742ac386d6b7c20 (127.15.255.133:40841): Couldn't send request to peer ee27c3f9e0ec46caa742ac386d6b7c20. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 549666d19e724a96801db183a07809d8. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:52.195001 28374 consensus_queue.cc:1035] T 549666d19e724a96801db183a07809d8 P 6810a866decb4f5089f7d35c870e4204 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:27:52.197533 28374 raft_consensus.cc:2932] T 549666d19e724a96801db183a07809d8 P 6810a866decb4f5089f7d35c870e4204 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER ee27c3f9e0ec46caa742ac386d6b7c20 (127.15.255.133) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } attrs { replace: true } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: NON_VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: true } } }
I20240502 21:27:52.198050 24046 raft_consensus.cc:2932] T 549666d19e724a96801db183a07809d8 P 6c194a6db6c145acafc0f3068b8ce1fd [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER ee27c3f9e0ec46caa742ac386d6b7c20 (127.15.255.133) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } attrs { replace: true } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: NON_VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: true } } }
I20240502 21:27:52.203964 23766 catalog_manager.cc:5484] T 549666d19e724a96801db183a07809d8 P 6810a866decb4f5089f7d35c870e4204 reported cstate change: config changed from index 6 to 7, NON_VOTER ee27c3f9e0ec46caa742ac386d6b7c20 (127.15.255.133) added. New cstate: current_term: 1 leader_uuid: "6810a866decb4f5089f7d35c870e4204" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: NON_VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:52.220890 26280 raft_consensus.cc:2932] T 549666d19e724a96801db183a07809d8 P a3e500d7077a48acb5edb33f7e178c81 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER ee27c3f9e0ec46caa742ac386d6b7c20 (127.15.255.133) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } attrs { replace: true } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: NON_VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: true } } }
I20240502 21:27:52.233822 24046 consensus_queue.cc:237] T 77fc7226878f44d3884ccb1fc41c44a4 P 6c194a6db6c145acafc0f3068b8ce1fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } attrs { replace: true } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } } peers { permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: NON_VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: true } }
I20240502 21:27:52.235807 23915 raft_consensus.cc:1261] T 77fc7226878f44d3884ccb1fc41c44a4 P 6810a866decb4f5089f7d35c870e4204 [term 1 FOLLOWER]: Refusing update from remote peer 6c194a6db6c145acafc0f3068b8ce1fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:27:52.235870 26145 raft_consensus.cc:1261] T 77fc7226878f44d3884ccb1fc41c44a4 P e117afbf8819424e8a98476453dfde2a [term 1 FOLLOWER]: Refusing update from remote peer 6c194a6db6c145acafc0f3068b8ce1fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240502 21:27:52.236280 23981 consensus_peers.cc:485] T 77fc7226878f44d3884ccb1fc41c44a4 P 6c194a6db6c145acafc0f3068b8ce1fd -> Peer 4f58800ba2ed45eabd81c2f2e691aacd (127.15.255.137:40161): Couldn't send request to peer 4f58800ba2ed45eabd81c2f2e691aacd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 77fc7226878f44d3884ccb1fc41c44a4. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:52.236373 28379 consensus_queue.cc:1035] T 77fc7226878f44d3884ccb1fc41c44a4 P 6c194a6db6c145acafc0f3068b8ce1fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:27:52.236629 28099 consensus_queue.cc:1035] T 77fc7226878f44d3884ccb1fc41c44a4 P 6c194a6db6c145acafc0f3068b8ce1fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:27:52.238639 28231 raft_consensus.cc:2932] T 77fc7226878f44d3884ccb1fc41c44a4 P 6c194a6db6c145acafc0f3068b8ce1fd [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 4f58800ba2ed45eabd81c2f2e691aacd (127.15.255.137) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } attrs { replace: true } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } } peers { permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: NON_VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: true } } }
I20240502 21:27:52.239279 23915 raft_consensus.cc:2932] T 77fc7226878f44d3884ccb1fc41c44a4 P 6810a866decb4f5089f7d35c870e4204 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 4f58800ba2ed45eabd81c2f2e691aacd (127.15.255.137) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } attrs { replace: true } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } } peers { permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: NON_VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: true } } }
I20240502 21:27:52.239228 26145 raft_consensus.cc:2932] T 77fc7226878f44d3884ccb1fc41c44a4 P e117afbf8819424e8a98476453dfde2a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 4f58800ba2ed45eabd81c2f2e691aacd (127.15.255.137) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } attrs { replace: true } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } } peers { permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: NON_VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: true } } }
I20240502 21:27:52.243376 23771 catalog_manager.cc:5484] T 77fc7226878f44d3884ccb1fc41c44a4 P 6c194a6db6c145acafc0f3068b8ce1fd reported cstate change: config changed from index 6 to 7, NON_VOTER 4f58800ba2ed45eabd81c2f2e691aacd (127.15.255.137) added. New cstate: current_term: 1 leader_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: NON_VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:52.255398 24183 consensus_queue.cc:237] T 0b3c514213bb40ca98955d6409a3b372 P 009d3e7e8a8b4e2497de3a73d01ced49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } attrs { replace: true } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: false } } peers { permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: NON_VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: true } }
I20240502 21:27:52.257670 26012 raft_consensus.cc:1261] T 0b3c514213bb40ca98955d6409a3b372 P fb6004b8640a4906aa0cfe63e383d40e [term 1 FOLLOWER]: Refusing update from remote peer 009d3e7e8a8b4e2497de3a73d01ced49: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:27:52.258392 28433 consensus_queue.cc:1035] T 0b3c514213bb40ca98955d6409a3b372 P 009d3e7e8a8b4e2497de3a73d01ced49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:27:52.262012 28433 raft_consensus.cc:2932] T 0b3c514213bb40ca98955d6409a3b372 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 4f58800ba2ed45eabd81c2f2e691aacd (127.15.255.137) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } attrs { replace: true } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: false } } peers { permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: NON_VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: true } } }
I20240502 21:27:52.265285 24317 raft_consensus.cc:1261] T 0b3c514213bb40ca98955d6409a3b372 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 1 FOLLOWER]: Refusing update from remote peer 009d3e7e8a8b4e2497de3a73d01ced49: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:27:52.266247 26012 raft_consensus.cc:2932] T 0b3c514213bb40ca98955d6409a3b372 P fb6004b8640a4906aa0cfe63e383d40e [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 4f58800ba2ed45eabd81c2f2e691aacd (127.15.255.137) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } attrs { replace: true } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: false } } peers { permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: NON_VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: true } } }
I20240502 21:27:52.272245 28433 consensus_queue.cc:1035] T 0b3c514213bb40ca98955d6409a3b372 P 009d3e7e8a8b4e2497de3a73d01ced49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:27:52.272605 23766 catalog_manager.cc:5484] T 0b3c514213bb40ca98955d6409a3b372 P fb6004b8640a4906aa0cfe63e383d40e reported cstate change: config changed from index 6 to 7, NON_VOTER 4f58800ba2ed45eabd81c2f2e691aacd (127.15.255.137) added. New cstate: current_term: 1 leader_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } attrs { replace: true } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: false } } peers { permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: NON_VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: true } } }
I20240502 21:27:52.274669 24317 raft_consensus.cc:2932] T 0b3c514213bb40ca98955d6409a3b372 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 4f58800ba2ed45eabd81c2f2e691aacd (127.15.255.137) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } attrs { replace: true } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: false } } peers { permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: NON_VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: true } } }
W20240502 21:27:52.275386 24116 consensus_peers.cc:485] T 0b3c514213bb40ca98955d6409a3b372 P 009d3e7e8a8b4e2497de3a73d01ced49 -> Peer 4f58800ba2ed45eabd81c2f2e691aacd (127.15.255.137:40161): Couldn't send request to peer 4f58800ba2ed45eabd81c2f2e691aacd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0b3c514213bb40ca98955d6409a3b372. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:52.293202 23915 consensus_queue.cc:237] T 4eebf4508766421eb6c547a0fbbc28b1 P 6810a866decb4f5089f7d35c870e4204 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } attrs { replace: true } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: true } }
I20240502 21:27:52.295327 24317 raft_consensus.cc:1261] T 4eebf4508766421eb6c547a0fbbc28b1 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 1 FOLLOWER]: Refusing update from remote peer 6810a866decb4f5089f7d35c870e4204: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:27:52.295459 26145 raft_consensus.cc:1261] T 4eebf4508766421eb6c547a0fbbc28b1 P e117afbf8819424e8a98476453dfde2a [term 1 FOLLOWER]: Refusing update from remote peer 6810a866decb4f5089f7d35c870e4204: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:27:52.295964 27234 consensus_queue.cc:1035] T 4eebf4508766421eb6c547a0fbbc28b1 P 6810a866decb4f5089f7d35c870e4204 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240502 21:27:52.296284 23850 consensus_peers.cc:485] T 4eebf4508766421eb6c547a0fbbc28b1 P 6810a866decb4f5089f7d35c870e4204 -> Peer a3e500d7077a48acb5edb33f7e178c81 (127.15.255.136:43595): Couldn't send request to peer a3e500d7077a48acb5edb33f7e178c81. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4eebf4508766421eb6c547a0fbbc28b1. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:52.296306 28484 consensus_queue.cc:1035] T 4eebf4508766421eb6c547a0fbbc28b1 P 6810a866decb4f5089f7d35c870e4204 [LEADER]: Connected to new peer: Peer: permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:27:52.298858 28374 raft_consensus.cc:2932] T 4eebf4508766421eb6c547a0fbbc28b1 P 6810a866decb4f5089f7d35c870e4204 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER a3e500d7077a48acb5edb33f7e178c81 (127.15.255.136) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } attrs { replace: true } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: true } } }
I20240502 21:27:52.299449 24317 raft_consensus.cc:2932] T 4eebf4508766421eb6c547a0fbbc28b1 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER a3e500d7077a48acb5edb33f7e178c81 (127.15.255.136) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } attrs { replace: true } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: true } } }
I20240502 21:27:52.302039 26145 raft_consensus.cc:2932] T 4eebf4508766421eb6c547a0fbbc28b1 P e117afbf8819424e8a98476453dfde2a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER a3e500d7077a48acb5edb33f7e178c81 (127.15.255.136) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } attrs { replace: true } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: true } } }
I20240502 21:27:52.303572 23766 catalog_manager.cc:5484] T 4eebf4508766421eb6c547a0fbbc28b1 P 6810a866decb4f5089f7d35c870e4204 reported cstate change: config changed from index 6 to 7, NON_VOTER a3e500d7077a48acb5edb33f7e178c81 (127.15.255.136) added. New cstate: current_term: 1 leader_uuid: "6810a866decb4f5089f7d35c870e4204" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:52.318426 26279 consensus_queue.cc:237] T f84bbe304597450c978ae28e0c4e7a3e P a3e500d7077a48acb5edb33f7e178c81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } attrs { replace: true } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: NON_VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: true } }
I20240502 21:27:52.320772 24046 raft_consensus.cc:1261] T f84bbe304597450c978ae28e0c4e7a3e P 6c194a6db6c145acafc0f3068b8ce1fd [term 2 FOLLOWER]: Refusing update from remote peer a3e500d7077a48acb5edb33f7e178c81: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240502 21:27:52.320961 24317 raft_consensus.cc:1261] T f84bbe304597450c978ae28e0c4e7a3e P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 2 FOLLOWER]: Refusing update from remote peer a3e500d7077a48acb5edb33f7e178c81: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240502 21:27:52.321368 28511 consensus_queue.cc:1035] T f84bbe304597450c978ae28e0c4e7a3e P a3e500d7077a48acb5edb33f7e178c81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:27:52.321691 27820 consensus_queue.cc:1035] T f84bbe304597450c978ae28e0c4e7a3e P a3e500d7077a48acb5edb33f7e178c81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:27:52.324247 27820 raft_consensus.cc:2932] T f84bbe304597450c978ae28e0c4e7a3e P a3e500d7077a48acb5edb33f7e178c81 [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 7 to 8, NON_VOTER fb6004b8640a4906aa0cfe63e383d40e (127.15.255.134) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } attrs { replace: true } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: NON_VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: true } } }
I20240502 21:27:52.324950 24046 raft_consensus.cc:2932] T f84bbe304597450c978ae28e0c4e7a3e P 6c194a6db6c145acafc0f3068b8ce1fd [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, NON_VOTER fb6004b8640a4906aa0cfe63e383d40e (127.15.255.134) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } attrs { replace: true } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: NON_VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: true } } }
I20240502 21:27:52.329881 23766 catalog_manager.cc:5484] T f84bbe304597450c978ae28e0c4e7a3e P a3e500d7077a48acb5edb33f7e178c81 reported cstate change: config changed from index 7 to 8, NON_VOTER fb6004b8640a4906aa0cfe63e383d40e (127.15.255.134) added. New cstate: current_term: 2 leader_uuid: "a3e500d7077a48acb5edb33f7e178c81" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: NON_VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:52.342514 24317 raft_consensus.cc:2932] T f84bbe304597450c978ae28e0c4e7a3e P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, NON_VOTER fb6004b8640a4906aa0cfe63e383d40e (127.15.255.134) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } attrs { replace: true } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: NON_VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: true } } }
W20240502 21:27:52.345062 26211 consensus_peers.cc:485] T f84bbe304597450c978ae28e0c4e7a3e P a3e500d7077a48acb5edb33f7e178c81 -> Peer fb6004b8640a4906aa0cfe63e383d40e (127.15.255.134:32825): Couldn't send request to peer fb6004b8640a4906aa0cfe63e383d40e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f84bbe304597450c978ae28e0c4e7a3e. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:52.360237 24183 consensus_queue.cc:237] T 85e77d5faa8346cd8a99dcbcdf75f250 P 009d3e7e8a8b4e2497de3a73d01ced49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } attrs { replace: true } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: NON_VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: true } }
I20240502 21:27:52.362486 23915 raft_consensus.cc:1261] T 85e77d5faa8346cd8a99dcbcdf75f250 P 6810a866decb4f5089f7d35c870e4204 [term 1 FOLLOWER]: Refusing update from remote peer 009d3e7e8a8b4e2497de3a73d01ced49: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:27:52.362573 24046 raft_consensus.cc:1261] T 85e77d5faa8346cd8a99dcbcdf75f250 P 6c194a6db6c145acafc0f3068b8ce1fd [term 1 FOLLOWER]: Refusing update from remote peer 009d3e7e8a8b4e2497de3a73d01ced49: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:27:52.363023 28493 consensus_queue.cc:1035] T 85e77d5faa8346cd8a99dcbcdf75f250 P 009d3e7e8a8b4e2497de3a73d01ced49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 21:27:52.363385 24115 consensus_peers.cc:485] T 85e77d5faa8346cd8a99dcbcdf75f250 P 009d3e7e8a8b4e2497de3a73d01ced49 -> Peer ee27c3f9e0ec46caa742ac386d6b7c20 (127.15.255.133:40841): Couldn't send request to peer ee27c3f9e0ec46caa742ac386d6b7c20. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 85e77d5faa8346cd8a99dcbcdf75f250. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:52.363422 27842 consensus_queue.cc:1035] T 85e77d5faa8346cd8a99dcbcdf75f250 P 009d3e7e8a8b4e2497de3a73d01ced49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:27:52.365990 28493 raft_consensus.cc:2932] T 85e77d5faa8346cd8a99dcbcdf75f250 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ee27c3f9e0ec46caa742ac386d6b7c20 (127.15.255.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } attrs { replace: true } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: NON_VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: true } } }
I20240502 21:27:52.366719 24046 raft_consensus.cc:2932] T 85e77d5faa8346cd8a99dcbcdf75f250 P 6c194a6db6c145acafc0f3068b8ce1fd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ee27c3f9e0ec46caa742ac386d6b7c20 (127.15.255.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } attrs { replace: true } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: NON_VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: true } } }
I20240502 21:27:52.366669 23915 raft_consensus.cc:2932] T 85e77d5faa8346cd8a99dcbcdf75f250 P 6810a866decb4f5089f7d35c870e4204 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER ee27c3f9e0ec46caa742ac386d6b7c20 (127.15.255.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } attrs { replace: true } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: NON_VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: true } } }
I20240502 21:27:52.372300 23766 catalog_manager.cc:5484] T 85e77d5faa8346cd8a99dcbcdf75f250 P 6810a866decb4f5089f7d35c870e4204 reported cstate change: config changed from index -1 to 4, NON_VOTER ee27c3f9e0ec46caa742ac386d6b7c20 (127.15.255.133) added. New cstate: current_term: 1 leader_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } attrs { replace: true } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: NON_VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: true } } }
I20240502 21:27:52.382678 24046 consensus_queue.cc:237] T a274546743354e79bffa26ae7000e895 P 6c194a6db6c145acafc0f3068b8ce1fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } attrs { replace: true } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } } peers { permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: NON_VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: true } }
I20240502 21:27:52.384958 26145 raft_consensus.cc:1261] T a274546743354e79bffa26ae7000e895 P e117afbf8819424e8a98476453dfde2a [term 1 FOLLOWER]: Refusing update from remote peer 6c194a6db6c145acafc0f3068b8ce1fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240502 21:27:52.385146 23981 consensus_peers.cc:485] T a274546743354e79bffa26ae7000e895 P 6c194a6db6c145acafc0f3068b8ce1fd -> Peer 4f58800ba2ed45eabd81c2f2e691aacd (127.15.255.137:40161): Couldn't send request to peer 4f58800ba2ed45eabd81c2f2e691aacd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a274546743354e79bffa26ae7000e895. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:52.385538 27456 consensus_queue.cc:1035] T a274546743354e79bffa26ae7000e895 P 6c194a6db6c145acafc0f3068b8ce1fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:27:52.385993 24183 raft_consensus.cc:1261] T a274546743354e79bffa26ae7000e895 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 1 FOLLOWER]: Refusing update from remote peer 6c194a6db6c145acafc0f3068b8ce1fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:27:52.386503 27456 consensus_queue.cc:1035] T a274546743354e79bffa26ae7000e895 P 6c194a6db6c145acafc0f3068b8ce1fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:27:52.388314 28099 raft_consensus.cc:2932] T a274546743354e79bffa26ae7000e895 P 6c194a6db6c145acafc0f3068b8ce1fd [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 4f58800ba2ed45eabd81c2f2e691aacd (127.15.255.137) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } attrs { replace: true } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } } peers { permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: NON_VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: true } } }
I20240502 21:27:52.388914 26145 raft_consensus.cc:2932] T a274546743354e79bffa26ae7000e895 P e117afbf8819424e8a98476453dfde2a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 4f58800ba2ed45eabd81c2f2e691aacd (127.15.255.137) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } attrs { replace: true } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } } peers { permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: NON_VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: true } } }
I20240502 21:27:52.389449 24183 raft_consensus.cc:2932] T a274546743354e79bffa26ae7000e895 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 4f58800ba2ed45eabd81c2f2e691aacd (127.15.255.137) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } attrs { replace: true } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } } peers { permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: NON_VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: true } } }
I20240502 21:27:52.393410 23771 catalog_manager.cc:5484] T a274546743354e79bffa26ae7000e895 P e117afbf8819424e8a98476453dfde2a reported cstate change: config changed from index 6 to 7, NON_VOTER 4f58800ba2ed45eabd81c2f2e691aacd (127.15.255.137) added. New cstate: current_term: 1 leader_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } attrs { replace: true } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } } peers { permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: NON_VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: true } } }
I20240502 21:27:52.413148 24317 consensus_queue.cc:237] T 52f621ea3f174490948eae854c706a92 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } attrs { replace: true } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: false } } peers { permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: NON_VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: true } }
I20240502 21:27:52.415513 25878 raft_consensus.cc:1261] T 52f621ea3f174490948eae854c706a92 P ee27c3f9e0ec46caa742ac386d6b7c20 [term 1 FOLLOWER]: Refusing update from remote peer 038c5d9d1d7a49c981c4c34fbc00d9e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:27:52.415578 23915 raft_consensus.cc:1261] T 52f621ea3f174490948eae854c706a92 P 6810a866decb4f5089f7d35c870e4204 [term 1 FOLLOWER]: Refusing update from remote peer 038c5d9d1d7a49c981c4c34fbc00d9e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:27:52.416160 28460 consensus_queue.cc:1035] T 52f621ea3f174490948eae854c706a92 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:27:52.416512 28216 consensus_queue.cc:1035] T 52f621ea3f174490948eae854c706a92 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240502 21:27:52.424028 24251 consensus_peers.cc:485] T 52f621ea3f174490948eae854c706a92 P 038c5d9d1d7a49c981c4c34fbc00d9e6 -> Peer 4f58800ba2ed45eabd81c2f2e691aacd (127.15.255.137:40161): Couldn't send request to peer 4f58800ba2ed45eabd81c2f2e691aacd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 52f621ea3f174490948eae854c706a92. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:52.426183 28216 raft_consensus.cc:2932] T 52f621ea3f174490948eae854c706a92 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 4f58800ba2ed45eabd81c2f2e691aacd (127.15.255.137) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } attrs { replace: true } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: false } } peers { permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: NON_VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: true } } }
I20240502 21:27:52.429193 23915 raft_consensus.cc:2932] T 52f621ea3f174490948eae854c706a92 P 6810a866decb4f5089f7d35c870e4204 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 4f58800ba2ed45eabd81c2f2e691aacd (127.15.255.137) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } attrs { replace: true } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: false } } peers { permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: NON_VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: true } } }
I20240502 21:27:52.431469 23766 catalog_manager.cc:5484] T 52f621ea3f174490948eae854c706a92 P 038c5d9d1d7a49c981c4c34fbc00d9e6 reported cstate change: config changed from index 6 to 7, NON_VOTER 4f58800ba2ed45eabd81c2f2e691aacd (127.15.255.137) added. New cstate: current_term: 1 leader_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: NON_VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:52.432612 25879 raft_consensus.cc:2932] T 52f621ea3f174490948eae854c706a92 P ee27c3f9e0ec46caa742ac386d6b7c20 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 4f58800ba2ed45eabd81c2f2e691aacd (127.15.255.137) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } attrs { replace: true } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: false } } peers { permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: NON_VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: true } } }
I20240502 21:27:52.449445 24183 consensus_queue.cc:237] T cca45c89b9fe40e7800d95c1598a9e52 P 009d3e7e8a8b4e2497de3a73d01ced49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } attrs { replace: true } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: false } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: true } }
I20240502 21:27:52.451514 25878 raft_consensus.cc:1261] T cca45c89b9fe40e7800d95c1598a9e52 P ee27c3f9e0ec46caa742ac386d6b7c20 [term 1 FOLLOWER]: Refusing update from remote peer 009d3e7e8a8b4e2497de3a73d01ced49: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:27:52.451730 24046 raft_consensus.cc:1261] T cca45c89b9fe40e7800d95c1598a9e52 P 6c194a6db6c145acafc0f3068b8ce1fd [term 1 FOLLOWER]: Refusing update from remote peer 009d3e7e8a8b4e2497de3a73d01ced49: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:27:52.451989 27842 consensus_queue.cc:1035] T cca45c89b9fe40e7800d95c1598a9e52 P 009d3e7e8a8b4e2497de3a73d01ced49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240502 21:27:52.452303 24117 consensus_peers.cc:485] T cca45c89b9fe40e7800d95c1598a9e52 P 009d3e7e8a8b4e2497de3a73d01ced49 -> Peer a3e500d7077a48acb5edb33f7e178c81 (127.15.255.136:43595): Couldn't send request to peer a3e500d7077a48acb5edb33f7e178c81. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cca45c89b9fe40e7800d95c1598a9e52. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:52.452337 28433 consensus_queue.cc:1035] T cca45c89b9fe40e7800d95c1598a9e52 P 009d3e7e8a8b4e2497de3a73d01ced49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:27:52.454465 27842 raft_consensus.cc:2932] T cca45c89b9fe40e7800d95c1598a9e52 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER a3e500d7077a48acb5edb33f7e178c81 (127.15.255.136) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } attrs { replace: true } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: false } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: true } } }
I20240502 21:27:52.454982 25878 raft_consensus.cc:2932] T cca45c89b9fe40e7800d95c1598a9e52 P ee27c3f9e0ec46caa742ac386d6b7c20 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER a3e500d7077a48acb5edb33f7e178c81 (127.15.255.136) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } attrs { replace: true } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: false } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: true } } }
I20240502 21:27:52.459182 24046 raft_consensus.cc:2932] T cca45c89b9fe40e7800d95c1598a9e52 P 6c194a6db6c145acafc0f3068b8ce1fd [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER a3e500d7077a48acb5edb33f7e178c81 (127.15.255.136) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } attrs { replace: true } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: false } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: true } } }
I20240502 21:27:52.459919 23771 catalog_manager.cc:5484] T cca45c89b9fe40e7800d95c1598a9e52 P 009d3e7e8a8b4e2497de3a73d01ced49 reported cstate change: config changed from index 6 to 7, NON_VOTER a3e500d7077a48acb5edb33f7e178c81 (127.15.255.136) added. New cstate: current_term: 1 leader_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:52.471273 24317 consensus_queue.cc:237] T fe044374ad7b4999afc72ab812466749 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } attrs { replace: true } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: false } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: NON_VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: true } }
I20240502 21:27:52.473520 25878 raft_consensus.cc:1261] T fe044374ad7b4999afc72ab812466749 P ee27c3f9e0ec46caa742ac386d6b7c20 [term 1 FOLLOWER]: Refusing update from remote peer 038c5d9d1d7a49c981c4c34fbc00d9e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:27:52.474027 23915 raft_consensus.cc:1261] T fe044374ad7b4999afc72ab812466749 P 6810a866decb4f5089f7d35c870e4204 [term 1 FOLLOWER]: Refusing update from remote peer 038c5d9d1d7a49c981c4c34fbc00d9e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:27:52.474597 28216 consensus_queue.cc:1035] T fe044374ad7b4999afc72ab812466749 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:27:52.474893 28463 consensus_queue.cc:1035] T fe044374ad7b4999afc72ab812466749 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240502 21:27:52.475185 24250 consensus_peers.cc:485] T fe044374ad7b4999afc72ab812466749 P 038c5d9d1d7a49c981c4c34fbc00d9e6 -> Peer fb6004b8640a4906aa0cfe63e383d40e (127.15.255.134:32825): Couldn't send request to peer fb6004b8640a4906aa0cfe63e383d40e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fe044374ad7b4999afc72ab812466749. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:52.477631 28460 raft_consensus.cc:2932] T fe044374ad7b4999afc72ab812466749 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER fb6004b8640a4906aa0cfe63e383d40e (127.15.255.134) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } attrs { replace: true } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: false } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: NON_VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: true } } }
I20240502 21:27:52.478281 25878 raft_consensus.cc:2932] T fe044374ad7b4999afc72ab812466749 P ee27c3f9e0ec46caa742ac386d6b7c20 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER fb6004b8640a4906aa0cfe63e383d40e (127.15.255.134) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } attrs { replace: true } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: false } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: NON_VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: true } } }
I20240502 21:27:52.478345 23915 raft_consensus.cc:2932] T fe044374ad7b4999afc72ab812466749 P 6810a866decb4f5089f7d35c870e4204 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER fb6004b8640a4906aa0cfe63e383d40e (127.15.255.134) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } attrs { replace: true } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: false } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: NON_VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: true } } }
I20240502 21:27:52.482376 23771 catalog_manager.cc:5484] T fe044374ad7b4999afc72ab812466749 P ee27c3f9e0ec46caa742ac386d6b7c20 reported cstate change: config changed from index 6 to 7, NON_VOTER fb6004b8640a4906aa0cfe63e383d40e (127.15.255.134) added. New cstate: current_term: 1 leader_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } attrs { replace: true } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: false } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: NON_VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: true } } }
I20240502 21:27:52.498978 23915 consensus_queue.cc:237] T 30236241b670443489240f191d9b9d50 P 6810a866decb4f5089f7d35c870e4204 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } attrs { replace: true } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: false } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: true } }
I20240502 21:27:52.501035 24317 raft_consensus.cc:1261] T 30236241b670443489240f191d9b9d50 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 1 FOLLOWER]: Refusing update from remote peer 6810a866decb4f5089f7d35c870e4204: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:27:52.501129 26012 raft_consensus.cc:1261] T 30236241b670443489240f191d9b9d50 P fb6004b8640a4906aa0cfe63e383d40e [term 1 FOLLOWER]: Refusing update from remote peer 6810a866decb4f5089f7d35c870e4204: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:27:52.501600 27234 consensus_queue.cc:1035] T 30236241b670443489240f191d9b9d50 P 6810a866decb4f5089f7d35c870e4204 [LEADER]: Connected to new peer: Peer: permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240502 21:27:52.502046 23850 consensus_peers.cc:485] T 30236241b670443489240f191d9b9d50 P 6810a866decb4f5089f7d35c870e4204 -> Peer a3e500d7077a48acb5edb33f7e178c81 (127.15.255.136:43595): Couldn't send request to peer a3e500d7077a48acb5edb33f7e178c81. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 30236241b670443489240f191d9b9d50. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:52.502445 27234 consensus_queue.cc:1035] T 30236241b670443489240f191d9b9d50 P 6810a866decb4f5089f7d35c870e4204 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:27:52.504185 28470 raft_consensus.cc:2932] T 30236241b670443489240f191d9b9d50 P 6810a866decb4f5089f7d35c870e4204 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER a3e500d7077a48acb5edb33f7e178c81 (127.15.255.136) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } attrs { replace: true } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: false } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: true } } }
I20240502 21:27:52.504907 24317 raft_consensus.cc:2932] T 30236241b670443489240f191d9b9d50 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER a3e500d7077a48acb5edb33f7e178c81 (127.15.255.136) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } attrs { replace: true } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: false } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: true } } }
I20240502 21:27:52.509330 23771 catalog_manager.cc:5484] T 30236241b670443489240f191d9b9d50 P 6810a866decb4f5089f7d35c870e4204 reported cstate change: config changed from index 6 to 7, NON_VOTER a3e500d7077a48acb5edb33f7e178c81 (127.15.255.136) added. New cstate: current_term: 1 leader_uuid: "6810a866decb4f5089f7d35c870e4204" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:52.509562 26012 raft_consensus.cc:2932] T 30236241b670443489240f191d9b9d50 P fb6004b8640a4906aa0cfe63e383d40e [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER a3e500d7077a48acb5edb33f7e178c81 (127.15.255.136) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } attrs { replace: true } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: false } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: true } } }
I20240502 21:27:52.659802 28642 ts_tablet_manager.cc:902] T 549666d19e724a96801db183a07809d8 P ee27c3f9e0ec46caa742ac386d6b7c20: Initiating tablet copy from peer 6810a866decb4f5089f7d35c870e4204 (127.15.255.129:40435)
I20240502 21:27:52.661119 28642 tablet_copy_client.cc:308] T 549666d19e724a96801db183a07809d8 P ee27c3f9e0ec46caa742ac386d6b7c20: tablet copy: Beginning tablet copy session from remote peer at address 127.15.255.129:40435
I20240502 21:27:52.661775 23935 tablet_copy_service.cc:140] P 6810a866decb4f5089f7d35c870e4204: Received BeginTabletCopySession request for tablet 549666d19e724a96801db183a07809d8 from peer ee27c3f9e0ec46caa742ac386d6b7c20 ({username='slave'} at 127.15.255.133:36415)
I20240502 21:27:52.661908 23935 tablet_copy_service.cc:161] P 6810a866decb4f5089f7d35c870e4204: Beginning new tablet copy session on tablet 549666d19e724a96801db183a07809d8 from peer ee27c3f9e0ec46caa742ac386d6b7c20 at {username='slave'} at 127.15.255.133:36415: session id = ee27c3f9e0ec46caa742ac386d6b7c20-549666d19e724a96801db183a07809d8
I20240502 21:27:52.663807 23935 tablet_copy_source_session.cc:205] T 549666d19e724a96801db183a07809d8 P 6810a866decb4f5089f7d35c870e4204: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:27:52.665056 28642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 549666d19e724a96801db183a07809d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:52.669435 28642 tablet_copy_client.cc:783] T 549666d19e724a96801db183a07809d8 P ee27c3f9e0ec46caa742ac386d6b7c20: tablet copy: Starting download of 0 data blocks...
I20240502 21:27:52.669644 28642 tablet_copy_client.cc:647] T 549666d19e724a96801db183a07809d8 P ee27c3f9e0ec46caa742ac386d6b7c20: tablet copy: Starting download of 1 WAL segments...
I20240502 21:27:52.670959 28646 tablet_copy_client.cc:1002] T 549666d19e724a96801db183a07809d8 P ee27c3f9e0ec46caa742ac386d6b7c20: tablet copy: Injecting latency into file download: 1500
I20240502 21:27:52.674314 28647 ts_tablet_manager.cc:902] T adf80b9d71314010a7a4cf052fd8a511 P fb6004b8640a4906aa0cfe63e383d40e: Initiating tablet copy from peer 6810a866decb4f5089f7d35c870e4204 (127.15.255.129:40435)
I20240502 21:27:52.675475 28647 tablet_copy_client.cc:308] T adf80b9d71314010a7a4cf052fd8a511 P fb6004b8640a4906aa0cfe63e383d40e: tablet copy: Beginning tablet copy session from remote peer at address 127.15.255.129:40435
I20240502 21:27:52.676024 23935 tablet_copy_service.cc:140] P 6810a866decb4f5089f7d35c870e4204: Received BeginTabletCopySession request for tablet adf80b9d71314010a7a4cf052fd8a511 from peer fb6004b8640a4906aa0cfe63e383d40e ({username='slave'} at 127.15.255.134:44215)
I20240502 21:27:52.676153 23935 tablet_copy_service.cc:161] P 6810a866decb4f5089f7d35c870e4204: Beginning new tablet copy session on tablet adf80b9d71314010a7a4cf052fd8a511 from peer fb6004b8640a4906aa0cfe63e383d40e at {username='slave'} at 127.15.255.134:44215: session id = fb6004b8640a4906aa0cfe63e383d40e-adf80b9d71314010a7a4cf052fd8a511
I20240502 21:27:52.677971 23935 tablet_copy_source_session.cc:205] T adf80b9d71314010a7a4cf052fd8a511 P 6810a866decb4f5089f7d35c870e4204: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:27:52.678939 28647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet adf80b9d71314010a7a4cf052fd8a511. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:52.682960 28647 tablet_copy_client.cc:783] T adf80b9d71314010a7a4cf052fd8a511 P fb6004b8640a4906aa0cfe63e383d40e: tablet copy: Starting download of 0 data blocks...
I20240502 21:27:52.683159 28647 tablet_copy_client.cc:647] T adf80b9d71314010a7a4cf052fd8a511 P fb6004b8640a4906aa0cfe63e383d40e: tablet copy: Starting download of 1 WAL segments...
I20240502 21:27:52.684367 28651 tablet_copy_client.cc:1002] T adf80b9d71314010a7a4cf052fd8a511 P fb6004b8640a4906aa0cfe63e383d40e: tablet copy: Injecting latency into file download: 1500
I20240502 21:27:52.698325 28652 ts_tablet_manager.cc:902] T c9b5be01b5e941b1848df79c7a9db348 P 4f58800ba2ed45eabd81c2f2e691aacd: Initiating tablet copy from peer 6c194a6db6c145acafc0f3068b8ce1fd (127.15.255.130:41235)
I20240502 21:27:52.699900 28652 tablet_copy_client.cc:308] T c9b5be01b5e941b1848df79c7a9db348 P 4f58800ba2ed45eabd81c2f2e691aacd: tablet copy: Beginning tablet copy session from remote peer at address 127.15.255.130:41235
I20240502 21:27:52.705715 24070 tablet_copy_service.cc:140] P 6c194a6db6c145acafc0f3068b8ce1fd: Received BeginTabletCopySession request for tablet c9b5be01b5e941b1848df79c7a9db348 from peer 4f58800ba2ed45eabd81c2f2e691aacd ({username='slave'} at 127.15.255.137:60133)
I20240502 21:27:52.705852 24070 tablet_copy_service.cc:161] P 6c194a6db6c145acafc0f3068b8ce1fd: Beginning new tablet copy session on tablet c9b5be01b5e941b1848df79c7a9db348 from peer 4f58800ba2ed45eabd81c2f2e691aacd at {username='slave'} at 127.15.255.137:60133: session id = 4f58800ba2ed45eabd81c2f2e691aacd-c9b5be01b5e941b1848df79c7a9db348
I20240502 21:27:52.707828 24070 tablet_copy_source_session.cc:205] T c9b5be01b5e941b1848df79c7a9db348 P 6c194a6db6c145acafc0f3068b8ce1fd: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:27:52.712410 28652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9b5be01b5e941b1848df79c7a9db348. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:52.725179 28652 tablet_copy_client.cc:783] T c9b5be01b5e941b1848df79c7a9db348 P 4f58800ba2ed45eabd81c2f2e691aacd: tablet copy: Starting download of 0 data blocks...
I20240502 21:27:52.725498 28652 tablet_copy_client.cc:647] T c9b5be01b5e941b1848df79c7a9db348 P 4f58800ba2ed45eabd81c2f2e691aacd: tablet copy: Starting download of 1 WAL segments...
I20240502 21:27:52.727269 28656 tablet_copy_client.cc:1002] T c9b5be01b5e941b1848df79c7a9db348 P 4f58800ba2ed45eabd81c2f2e691aacd: tablet copy: Injecting latency into file download: 1500
I20240502 21:27:52.740106 28658 ts_tablet_manager.cc:902] T 11cb9a2fbda946fb9f303f2c465bccb6 P a3e500d7077a48acb5edb33f7e178c81: Initiating tablet copy from peer 038c5d9d1d7a49c981c4c34fbc00d9e6 (127.15.255.132:46813)
I20240502 21:27:52.741411 28658 tablet_copy_client.cc:308] T 11cb9a2fbda946fb9f303f2c465bccb6 P a3e500d7077a48acb5edb33f7e178c81: tablet copy: Beginning tablet copy session from remote peer at address 127.15.255.132:46813
I20240502 21:27:52.742053 24338 tablet_copy_service.cc:140] P 038c5d9d1d7a49c981c4c34fbc00d9e6: Received BeginTabletCopySession request for tablet 11cb9a2fbda946fb9f303f2c465bccb6 from peer a3e500d7077a48acb5edb33f7e178c81 ({username='slave'} at 127.15.255.136:38895)
I20240502 21:27:52.742197 24338 tablet_copy_service.cc:161] P 038c5d9d1d7a49c981c4c34fbc00d9e6: Beginning new tablet copy session on tablet 11cb9a2fbda946fb9f303f2c465bccb6 from peer a3e500d7077a48acb5edb33f7e178c81 at {username='slave'} at 127.15.255.136:38895: session id = a3e500d7077a48acb5edb33f7e178c81-11cb9a2fbda946fb9f303f2c465bccb6
I20240502 21:27:52.744179 24338 tablet_copy_source_session.cc:205] T 11cb9a2fbda946fb9f303f2c465bccb6 P 038c5d9d1d7a49c981c4c34fbc00d9e6: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:27:52.745312 28658 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11cb9a2fbda946fb9f303f2c465bccb6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:52.749684 28658 tablet_copy_client.cc:783] T 11cb9a2fbda946fb9f303f2c465bccb6 P a3e500d7077a48acb5edb33f7e178c81: tablet copy: Starting download of 0 data blocks...
I20240502 21:27:52.749884 28658 tablet_copy_client.cc:647] T 11cb9a2fbda946fb9f303f2c465bccb6 P a3e500d7077a48acb5edb33f7e178c81: tablet copy: Starting download of 1 WAL segments...
I20240502 21:27:52.751183 28663 ts_tablet_manager.cc:902] T 77fc7226878f44d3884ccb1fc41c44a4 P 4f58800ba2ed45eabd81c2f2e691aacd: Initiating tablet copy from peer 6c194a6db6c145acafc0f3068b8ce1fd (127.15.255.130:41235)
I20240502 21:27:52.751335 28662 tablet_copy_client.cc:1002] T 11cb9a2fbda946fb9f303f2c465bccb6 P a3e500d7077a48acb5edb33f7e178c81: tablet copy: Injecting latency into file download: 1500
I20240502 21:27:52.752514 28663 tablet_copy_client.cc:308] T 77fc7226878f44d3884ccb1fc41c44a4 P 4f58800ba2ed45eabd81c2f2e691aacd: tablet copy: Beginning tablet copy session from remote peer at address 127.15.255.130:41235
I20240502 21:27:52.753154 24070 tablet_copy_service.cc:140] P 6c194a6db6c145acafc0f3068b8ce1fd: Received BeginTabletCopySession request for tablet 77fc7226878f44d3884ccb1fc41c44a4 from peer 4f58800ba2ed45eabd81c2f2e691aacd ({username='slave'} at 127.15.255.137:60133)
I20240502 21:27:52.753312 24070 tablet_copy_service.cc:161] P 6c194a6db6c145acafc0f3068b8ce1fd: Beginning new tablet copy session on tablet 77fc7226878f44d3884ccb1fc41c44a4 from peer 4f58800ba2ed45eabd81c2f2e691aacd at {username='slave'} at 127.15.255.137:60133: session id = 4f58800ba2ed45eabd81c2f2e691aacd-77fc7226878f44d3884ccb1fc41c44a4
I20240502 21:27:52.755092 24070 tablet_copy_source_session.cc:205] T 77fc7226878f44d3884ccb1fc41c44a4 P 6c194a6db6c145acafc0f3068b8ce1fd: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:27:52.756160 28663 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77fc7226878f44d3884ccb1fc41c44a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:52.760829 28663 tablet_copy_client.cc:783] T 77fc7226878f44d3884ccb1fc41c44a4 P 4f58800ba2ed45eabd81c2f2e691aacd: tablet copy: Starting download of 0 data blocks...
I20240502 21:27:52.761029 28663 tablet_copy_client.cc:647] T 77fc7226878f44d3884ccb1fc41c44a4 P 4f58800ba2ed45eabd81c2f2e691aacd: tablet copy: Starting download of 1 WAL segments...
I20240502 21:27:52.762725 28667 tablet_copy_client.cc:1002] T 77fc7226878f44d3884ccb1fc41c44a4 P 4f58800ba2ed45eabd81c2f2e691aacd: tablet copy: Injecting latency into file download: 1500
I20240502 21:27:52.770130 28668 ts_tablet_manager.cc:902] T f84bbe304597450c978ae28e0c4e7a3e P fb6004b8640a4906aa0cfe63e383d40e: Initiating tablet copy from peer a3e500d7077a48acb5edb33f7e178c81 (127.15.255.136:43595)
I20240502 21:27:52.771476 28668 tablet_copy_client.cc:308] T f84bbe304597450c978ae28e0c4e7a3e P fb6004b8640a4906aa0cfe63e383d40e: tablet copy: Beginning tablet copy session from remote peer at address 127.15.255.136:43595
I20240502 21:27:52.777534 26300 tablet_copy_service.cc:140] P a3e500d7077a48acb5edb33f7e178c81: Received BeginTabletCopySession request for tablet f84bbe304597450c978ae28e0c4e7a3e from peer fb6004b8640a4906aa0cfe63e383d40e ({username='slave'} at 127.15.255.134:54653)
I20240502 21:27:52.777784 26300 tablet_copy_service.cc:161] P a3e500d7077a48acb5edb33f7e178c81: Beginning new tablet copy session on tablet f84bbe304597450c978ae28e0c4e7a3e from peer fb6004b8640a4906aa0cfe63e383d40e at {username='slave'} at 127.15.255.134:54653: session id = fb6004b8640a4906aa0cfe63e383d40e-f84bbe304597450c978ae28e0c4e7a3e
I20240502 21:27:52.780834 26300 tablet_copy_source_session.cc:205] T f84bbe304597450c978ae28e0c4e7a3e P a3e500d7077a48acb5edb33f7e178c81: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:27:52.782462 28668 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f84bbe304597450c978ae28e0c4e7a3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:52.786651 28668 tablet_copy_client.cc:783] T f84bbe304597450c978ae28e0c4e7a3e P fb6004b8640a4906aa0cfe63e383d40e: tablet copy: Starting download of 0 data blocks...
I20240502 21:27:52.786863 28668 tablet_copy_client.cc:647] T f84bbe304597450c978ae28e0c4e7a3e P fb6004b8640a4906aa0cfe63e383d40e: tablet copy: Starting download of 1 WAL segments...
I20240502 21:27:52.788544 28672 tablet_copy_client.cc:1002] T f84bbe304597450c978ae28e0c4e7a3e P fb6004b8640a4906aa0cfe63e383d40e: tablet copy: Injecting latency into file download: 1500
I20240502 21:27:52.831140 28674 ts_tablet_manager.cc:902] T 0b3c514213bb40ca98955d6409a3b372 P 4f58800ba2ed45eabd81c2f2e691aacd: Initiating tablet copy from peer 009d3e7e8a8b4e2497de3a73d01ced49 (127.15.255.131:37895)
I20240502 21:27:52.832630 28674 tablet_copy_client.cc:308] T 0b3c514213bb40ca98955d6409a3b372 P 4f58800ba2ed45eabd81c2f2e691aacd: tablet copy: Beginning tablet copy session from remote peer at address 127.15.255.131:37895
I20240502 21:27:52.838279 24205 tablet_copy_service.cc:140] P 009d3e7e8a8b4e2497de3a73d01ced49: Received BeginTabletCopySession request for tablet 0b3c514213bb40ca98955d6409a3b372 from peer 4f58800ba2ed45eabd81c2f2e691aacd ({username='slave'} at 127.15.255.137:51277)
I20240502 21:27:52.838472 24205 tablet_copy_service.cc:161] P 009d3e7e8a8b4e2497de3a73d01ced49: Beginning new tablet copy session on tablet 0b3c514213bb40ca98955d6409a3b372 from peer 4f58800ba2ed45eabd81c2f2e691aacd at {username='slave'} at 127.15.255.137:51277: session id = 4f58800ba2ed45eabd81c2f2e691aacd-0b3c514213bb40ca98955d6409a3b372
I20240502 21:27:52.840945 24205 tablet_copy_source_session.cc:205] T 0b3c514213bb40ca98955d6409a3b372 P 009d3e7e8a8b4e2497de3a73d01ced49: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:27:52.842275 28674 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b3c514213bb40ca98955d6409a3b372. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:52.846817 28674 tablet_copy_client.cc:783] T 0b3c514213bb40ca98955d6409a3b372 P 4f58800ba2ed45eabd81c2f2e691aacd: tablet copy: Starting download of 0 data blocks...
I20240502 21:27:52.847056 28674 tablet_copy_client.cc:647] T 0b3c514213bb40ca98955d6409a3b372 P 4f58800ba2ed45eabd81c2f2e691aacd: tablet copy: Starting download of 1 WAL segments...
I20240502 21:27:52.848388 28678 tablet_copy_client.cc:1002] T 0b3c514213bb40ca98955d6409a3b372 P 4f58800ba2ed45eabd81c2f2e691aacd: tablet copy: Injecting latency into file download: 1500
I20240502 21:27:52.859984 28684 ts_tablet_manager.cc:902] T a274546743354e79bffa26ae7000e895 P 4f58800ba2ed45eabd81c2f2e691aacd: Initiating tablet copy from peer 6c194a6db6c145acafc0f3068b8ce1fd (127.15.255.130:41235)
I20240502 21:27:52.865275 28684 tablet_copy_client.cc:308] T a274546743354e79bffa26ae7000e895 P 4f58800ba2ed45eabd81c2f2e691aacd: tablet copy: Beginning tablet copy session from remote peer at address 127.15.255.130:41235
I20240502 21:27:52.865933 24070 tablet_copy_service.cc:140] P 6c194a6db6c145acafc0f3068b8ce1fd: Received BeginTabletCopySession request for tablet a274546743354e79bffa26ae7000e895 from peer 4f58800ba2ed45eabd81c2f2e691aacd ({username='slave'} at 127.15.255.137:60133)
I20240502 21:27:52.866104 24070 tablet_copy_service.cc:161] P 6c194a6db6c145acafc0f3068b8ce1fd: Beginning new tablet copy session on tablet a274546743354e79bffa26ae7000e895 from peer 4f58800ba2ed45eabd81c2f2e691aacd at {username='slave'} at 127.15.255.137:60133: session id = 4f58800ba2ed45eabd81c2f2e691aacd-a274546743354e79bffa26ae7000e895
I20240502 21:27:52.868783 24070 tablet_copy_source_session.cc:205] T a274546743354e79bffa26ae7000e895 P 6c194a6db6c145acafc0f3068b8ce1fd: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:27:52.869999 28684 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a274546743354e79bffa26ae7000e895. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:52.875571 28684 tablet_copy_client.cc:783] T a274546743354e79bffa26ae7000e895 P 4f58800ba2ed45eabd81c2f2e691aacd: tablet copy: Starting download of 0 data blocks...
I20240502 21:27:52.875810 28684 tablet_copy_client.cc:647] T a274546743354e79bffa26ae7000e895 P 4f58800ba2ed45eabd81c2f2e691aacd: tablet copy: Starting download of 1 WAL segments...
I20240502 21:27:52.877211 28693 tablet_copy_client.cc:1002] T a274546743354e79bffa26ae7000e895 P 4f58800ba2ed45eabd81c2f2e691aacd: tablet copy: Injecting latency into file download: 1500
I20240502 21:27:52.901361 28710 ts_tablet_manager.cc:902] T 4eebf4508766421eb6c547a0fbbc28b1 P a3e500d7077a48acb5edb33f7e178c81: Initiating tablet copy from peer 6810a866decb4f5089f7d35c870e4204 (127.15.255.129:40435)
I20240502 21:27:52.905346 28710 tablet_copy_client.cc:308] T 4eebf4508766421eb6c547a0fbbc28b1 P a3e500d7077a48acb5edb33f7e178c81: tablet copy: Beginning tablet copy session from remote peer at address 127.15.255.129:40435
I20240502 21:27:52.906033 23935 tablet_copy_service.cc:140] P 6810a866decb4f5089f7d35c870e4204: Received BeginTabletCopySession request for tablet 4eebf4508766421eb6c547a0fbbc28b1 from peer a3e500d7077a48acb5edb33f7e178c81 ({username='slave'} at 127.15.255.136:46499)
I20240502 21:27:52.906180 23935 tablet_copy_service.cc:161] P 6810a866decb4f5089f7d35c870e4204: Beginning new tablet copy session on tablet 4eebf4508766421eb6c547a0fbbc28b1 from peer a3e500d7077a48acb5edb33f7e178c81 at {username='slave'} at 127.15.255.136:46499: session id = a3e500d7077a48acb5edb33f7e178c81-4eebf4508766421eb6c547a0fbbc28b1
I20240502 21:27:52.908749 23935 tablet_copy_source_session.cc:205] T 4eebf4508766421eb6c547a0fbbc28b1 P 6810a866decb4f5089f7d35c870e4204: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:27:52.909881 28710 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4eebf4508766421eb6c547a0fbbc28b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:52.914424 28710 tablet_copy_client.cc:783] T 4eebf4508766421eb6c547a0fbbc28b1 P a3e500d7077a48acb5edb33f7e178c81: tablet copy: Starting download of 0 data blocks...
I20240502 21:27:52.914688 28710 tablet_copy_client.cc:647] T 4eebf4508766421eb6c547a0fbbc28b1 P a3e500d7077a48acb5edb33f7e178c81: tablet copy: Starting download of 1 WAL segments...
I20240502 21:27:52.916155 28719 tablet_copy_client.cc:1002] T 4eebf4508766421eb6c547a0fbbc28b1 P a3e500d7077a48acb5edb33f7e178c81: tablet copy: Injecting latency into file download: 1500
I20240502 21:27:52.917315 28724 ts_tablet_manager.cc:902] T 30236241b670443489240f191d9b9d50 P a3e500d7077a48acb5edb33f7e178c81: Initiating tablet copy from peer 6810a866decb4f5089f7d35c870e4204 (127.15.255.129:40435)
I20240502 21:27:52.919143 28724 tablet_copy_client.cc:308] T 30236241b670443489240f191d9b9d50 P a3e500d7077a48acb5edb33f7e178c81: tablet copy: Beginning tablet copy session from remote peer at address 127.15.255.129:40435
I20240502 21:27:52.919791 23935 tablet_copy_service.cc:140] P 6810a866decb4f5089f7d35c870e4204: Received BeginTabletCopySession request for tablet 30236241b670443489240f191d9b9d50 from peer a3e500d7077a48acb5edb33f7e178c81 ({username='slave'} at 127.15.255.136:46499)
I20240502 21:27:52.919950 23935 tablet_copy_service.cc:161] P 6810a866decb4f5089f7d35c870e4204: Beginning new tablet copy session on tablet 30236241b670443489240f191d9b9d50 from peer a3e500d7077a48acb5edb33f7e178c81 at {username='slave'} at 127.15.255.136:46499: session id = a3e500d7077a48acb5edb33f7e178c81-30236241b670443489240f191d9b9d50
I20240502 21:27:52.922420 23935 tablet_copy_source_session.cc:205] T 30236241b670443489240f191d9b9d50 P 6810a866decb4f5089f7d35c870e4204: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:27:52.923650 28724 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30236241b670443489240f191d9b9d50. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:52.928565 28724 tablet_copy_client.cc:783] T 30236241b670443489240f191d9b9d50 P a3e500d7077a48acb5edb33f7e178c81: tablet copy: Starting download of 0 data blocks...
I20240502 21:27:52.928812 28734 ts_tablet_manager.cc:902] T fe044374ad7b4999afc72ab812466749 P fb6004b8640a4906aa0cfe63e383d40e: Initiating tablet copy from peer 038c5d9d1d7a49c981c4c34fbc00d9e6 (127.15.255.132:46813)
I20240502 21:27:52.928822 28724 tablet_copy_client.cc:647] T 30236241b670443489240f191d9b9d50 P a3e500d7077a48acb5edb33f7e178c81: tablet copy: Starting download of 1 WAL segments...
I20240502 21:27:52.930399 28729 tablet_copy_client.cc:1002] T 30236241b670443489240f191d9b9d50 P a3e500d7077a48acb5edb33f7e178c81: tablet copy: Injecting latency into file download: 1500
I20240502 21:27:52.930931 28734 tablet_copy_client.cc:308] T fe044374ad7b4999afc72ab812466749 P fb6004b8640a4906aa0cfe63e383d40e: tablet copy: Beginning tablet copy session from remote peer at address 127.15.255.132:46813
I20240502 21:27:52.931627 24338 tablet_copy_service.cc:140] P 038c5d9d1d7a49c981c4c34fbc00d9e6: Received BeginTabletCopySession request for tablet fe044374ad7b4999afc72ab812466749 from peer fb6004b8640a4906aa0cfe63e383d40e ({username='slave'} at 127.15.255.134:51733)
I20240502 21:27:52.931813 24338 tablet_copy_service.cc:161] P 038c5d9d1d7a49c981c4c34fbc00d9e6: Beginning new tablet copy session on tablet fe044374ad7b4999afc72ab812466749 from peer fb6004b8640a4906aa0cfe63e383d40e at {username='slave'} at 127.15.255.134:51733: session id = fb6004b8640a4906aa0cfe63e383d40e-fe044374ad7b4999afc72ab812466749
I20240502 21:27:52.934476 24338 tablet_copy_source_session.cc:205] T fe044374ad7b4999afc72ab812466749 P 038c5d9d1d7a49c981c4c34fbc00d9e6: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:27:52.935886 28734 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe044374ad7b4999afc72ab812466749. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:52.941447 28734 tablet_copy_client.cc:783] T fe044374ad7b4999afc72ab812466749 P fb6004b8640a4906aa0cfe63e383d40e: tablet copy: Starting download of 0 data blocks...
I20240502 21:27:52.941659 28734 tablet_copy_client.cc:647] T fe044374ad7b4999afc72ab812466749 P fb6004b8640a4906aa0cfe63e383d40e: tablet copy: Starting download of 1 WAL segments...
I20240502 21:27:52.943181 28739 tablet_copy_client.cc:1002] T fe044374ad7b4999afc72ab812466749 P fb6004b8640a4906aa0cfe63e383d40e: tablet copy: Injecting latency into file download: 1500
I20240502 21:27:52.944254 28745 ts_tablet_manager.cc:902] T 52f621ea3f174490948eae854c706a92 P 4f58800ba2ed45eabd81c2f2e691aacd: Initiating tablet copy from peer 038c5d9d1d7a49c981c4c34fbc00d9e6 (127.15.255.132:46813)
I20240502 21:27:52.952272 28745 tablet_copy_client.cc:308] T 52f621ea3f174490948eae854c706a92 P 4f58800ba2ed45eabd81c2f2e691aacd: tablet copy: Beginning tablet copy session from remote peer at address 127.15.255.132:46813
I20240502 21:27:52.959143 24338 tablet_copy_service.cc:140] P 038c5d9d1d7a49c981c4c34fbc00d9e6: Received BeginTabletCopySession request for tablet 52f621ea3f174490948eae854c706a92 from peer 4f58800ba2ed45eabd81c2f2e691aacd ({username='slave'} at 127.15.255.137:48505)
I20240502 21:27:52.959314 24338 tablet_copy_service.cc:161] P 038c5d9d1d7a49c981c4c34fbc00d9e6: Beginning new tablet copy session on tablet 52f621ea3f174490948eae854c706a92 from peer 4f58800ba2ed45eabd81c2f2e691aacd at {username='slave'} at 127.15.255.137:48505: session id = 4f58800ba2ed45eabd81c2f2e691aacd-52f621ea3f174490948eae854c706a92
I20240502 21:27:52.961133 28756 ts_tablet_manager.cc:902] T 85e77d5faa8346cd8a99dcbcdf75f250 P ee27c3f9e0ec46caa742ac386d6b7c20: Initiating tablet copy from peer 009d3e7e8a8b4e2497de3a73d01ced49 (127.15.255.131:37895)
I20240502 21:27:52.962122 24338 tablet_copy_source_session.cc:205] T 52f621ea3f174490948eae854c706a92 P 038c5d9d1d7a49c981c4c34fbc00d9e6: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:27:52.963205 28745 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 52f621ea3f174490948eae854c706a92. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:52.967715 28745 tablet_copy_client.cc:783] T 52f621ea3f174490948eae854c706a92 P 4f58800ba2ed45eabd81c2f2e691aacd: tablet copy: Starting download of 0 data blocks...
I20240502 21:27:52.968001 28745 tablet_copy_client.cc:647] T 52f621ea3f174490948eae854c706a92 P 4f58800ba2ed45eabd81c2f2e691aacd: tablet copy: Starting download of 1 WAL segments...
I20240502 21:27:52.969712 28749 tablet_copy_client.cc:1002] T 52f621ea3f174490948eae854c706a92 P 4f58800ba2ed45eabd81c2f2e691aacd: tablet copy: Injecting latency into file download: 1500
I20240502 21:27:52.975162 28756 tablet_copy_client.cc:308] T 85e77d5faa8346cd8a99dcbcdf75f250 P ee27c3f9e0ec46caa742ac386d6b7c20: tablet copy: Beginning tablet copy session from remote peer at address 127.15.255.131:37895
I20240502 21:27:52.975968 24205 tablet_copy_service.cc:140] P 009d3e7e8a8b4e2497de3a73d01ced49: Received BeginTabletCopySession request for tablet 85e77d5faa8346cd8a99dcbcdf75f250 from peer ee27c3f9e0ec46caa742ac386d6b7c20 ({username='slave'} at 127.15.255.133:44245)
I20240502 21:27:52.976109 24205 tablet_copy_service.cc:161] P 009d3e7e8a8b4e2497de3a73d01ced49: Beginning new tablet copy session on tablet 85e77d5faa8346cd8a99dcbcdf75f250 from peer ee27c3f9e0ec46caa742ac386d6b7c20 at {username='slave'} at 127.15.255.133:44245: session id = ee27c3f9e0ec46caa742ac386d6b7c20-85e77d5faa8346cd8a99dcbcdf75f250
I20240502 21:27:52.979070 24205 tablet_copy_source_session.cc:205] T 85e77d5faa8346cd8a99dcbcdf75f250 P 009d3e7e8a8b4e2497de3a73d01ced49: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:27:52.980193 28756 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85e77d5faa8346cd8a99dcbcdf75f250. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:52.985831 28756 tablet_copy_client.cc:783] T 85e77d5faa8346cd8a99dcbcdf75f250 P ee27c3f9e0ec46caa742ac386d6b7c20: tablet copy: Starting download of 0 data blocks...
I20240502 21:27:52.986056 28756 tablet_copy_client.cc:647] T 85e77d5faa8346cd8a99dcbcdf75f250 P ee27c3f9e0ec46caa742ac386d6b7c20: tablet copy: Starting download of 1 WAL segments...
I20240502 21:27:52.987793 28761 tablet_copy_client.cc:1002] T 85e77d5faa8346cd8a99dcbcdf75f250 P ee27c3f9e0ec46caa742ac386d6b7c20: tablet copy: Injecting latency into file download: 1500
I20240502 21:27:53.022922 28780 ts_tablet_manager.cc:902] T cca45c89b9fe40e7800d95c1598a9e52 P a3e500d7077a48acb5edb33f7e178c81: Initiating tablet copy from peer 009d3e7e8a8b4e2497de3a73d01ced49 (127.15.255.131:37895)
I20240502 21:27:53.024127 28780 tablet_copy_client.cc:308] T cca45c89b9fe40e7800d95c1598a9e52 P a3e500d7077a48acb5edb33f7e178c81: tablet copy: Beginning tablet copy session from remote peer at address 127.15.255.131:37895
I20240502 21:27:53.024770 24205 tablet_copy_service.cc:140] P 009d3e7e8a8b4e2497de3a73d01ced49: Received BeginTabletCopySession request for tablet cca45c89b9fe40e7800d95c1598a9e52 from peer a3e500d7077a48acb5edb33f7e178c81 ({username='slave'} at 127.15.255.136:51951)
I20240502 21:27:53.024916 24205 tablet_copy_service.cc:161] P 009d3e7e8a8b4e2497de3a73d01ced49: Beginning new tablet copy session on tablet cca45c89b9fe40e7800d95c1598a9e52 from peer a3e500d7077a48acb5edb33f7e178c81 at {username='slave'} at 127.15.255.136:51951: session id = a3e500d7077a48acb5edb33f7e178c81-cca45c89b9fe40e7800d95c1598a9e52
I20240502 21:27:53.026741 24205 tablet_copy_source_session.cc:205] T cca45c89b9fe40e7800d95c1598a9e52 P 009d3e7e8a8b4e2497de3a73d01ced49: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:27:53.027751 28780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cca45c89b9fe40e7800d95c1598a9e52. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:53.031891 28780 tablet_copy_client.cc:783] T cca45c89b9fe40e7800d95c1598a9e52 P a3e500d7077a48acb5edb33f7e178c81: tablet copy: Starting download of 0 data blocks...
I20240502 21:27:53.032085 28780 tablet_copy_client.cc:647] T cca45c89b9fe40e7800d95c1598a9e52 P a3e500d7077a48acb5edb33f7e178c81: tablet copy: Starting download of 1 WAL segments...
I20240502 21:27:53.033439 28784 tablet_copy_client.cc:1002] T cca45c89b9fe40e7800d95c1598a9e52 P a3e500d7077a48acb5edb33f7e178c81: tablet copy: Injecting latency into file download: 1500
I20240502 21:27:54.171303 28642 tablet_copy_client.cc:523] T 549666d19e724a96801db183a07809d8 P ee27c3f9e0ec46caa742ac386d6b7c20: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:27:54.173957 28642 tablet_bootstrap.cc:492] T 549666d19e724a96801db183a07809d8 P ee27c3f9e0ec46caa742ac386d6b7c20: Bootstrap starting.
I20240502 21:27:54.184731 28647 tablet_copy_client.cc:523] T adf80b9d71314010a7a4cf052fd8a511 P fb6004b8640a4906aa0cfe63e383d40e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:27:54.188258 28647 tablet_bootstrap.cc:492] T adf80b9d71314010a7a4cf052fd8a511 P fb6004b8640a4906aa0cfe63e383d40e: Bootstrap starting.
I20240502 21:27:54.220664 28642 tablet_bootstrap.cc:492] T 549666d19e724a96801db183a07809d8 P ee27c3f9e0ec46caa742ac386d6b7c20: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:27:54.221282 28642 tablet_bootstrap.cc:492] T 549666d19e724a96801db183a07809d8 P ee27c3f9e0ec46caa742ac386d6b7c20: Bootstrap complete.
I20240502 21:27:54.221661 28642 ts_tablet_manager.cc:1367] T 549666d19e724a96801db183a07809d8 P ee27c3f9e0ec46caa742ac386d6b7c20: Time spent bootstrapping tablet: real 0.048s user 0.037s sys 0.009s
I20240502 21:27:54.222685 28642 raft_consensus.cc:348] T 549666d19e724a96801db183a07809d8 P ee27c3f9e0ec46caa742ac386d6b7c20 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } attrs { replace: true } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: NON_VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: true } }
I20240502 21:27:54.223078 28642 raft_consensus.cc:729] T 549666d19e724a96801db183a07809d8 P ee27c3f9e0ec46caa742ac386d6b7c20 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ee27c3f9e0ec46caa742ac386d6b7c20, State: Initialized, Role: LEARNER
I20240502 21:27:54.223345 28642 consensus_queue.cc:260] T 549666d19e724a96801db183a07809d8 P ee27c3f9e0ec46caa742ac386d6b7c20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } attrs { replace: true } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: NON_VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: true } }
I20240502 21:27:54.227784 28652 tablet_copy_client.cc:523] T c9b5be01b5e941b1848df79c7a9db348 P 4f58800ba2ed45eabd81c2f2e691aacd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:27:54.228161 28642 ts_tablet_manager.cc:1398] T 549666d19e724a96801db183a07809d8 P ee27c3f9e0ec46caa742ac386d6b7c20: Time spent starting tablet: real 0.006s user 0.001s sys 0.004s
I20240502 21:27:54.228842 23935 tablet_copy_service.cc:342] P 6810a866decb4f5089f7d35c870e4204: Request end of tablet copy session ee27c3f9e0ec46caa742ac386d6b7c20-549666d19e724a96801db183a07809d8 received from {username='slave'} at 127.15.255.133:36415
I20240502 21:27:54.228967 23935 tablet_copy_service.cc:433] P 6810a866decb4f5089f7d35c870e4204: ending tablet copy session ee27c3f9e0ec46caa742ac386d6b7c20-549666d19e724a96801db183a07809d8 on tablet 549666d19e724a96801db183a07809d8 with peer ee27c3f9e0ec46caa742ac386d6b7c20
I20240502 21:27:54.233390 28652 tablet_bootstrap.cc:492] T c9b5be01b5e941b1848df79c7a9db348 P 4f58800ba2ed45eabd81c2f2e691aacd: Bootstrap starting.
I20240502 21:27:54.245519 28652 log.cc:824] T c9b5be01b5e941b1848df79c7a9db348 P 4f58800ba2ed45eabd81c2f2e691aacd: Log is configured to *not* fsync() on all Append() calls
I20240502 21:27:54.249069 28647 tablet_bootstrap.cc:492] T adf80b9d71314010a7a4cf052fd8a511 P fb6004b8640a4906aa0cfe63e383d40e: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:27:54.249564 28647 tablet_bootstrap.cc:492] T adf80b9d71314010a7a4cf052fd8a511 P fb6004b8640a4906aa0cfe63e383d40e: Bootstrap complete.
I20240502 21:27:54.249874 28647 ts_tablet_manager.cc:1367] T adf80b9d71314010a7a4cf052fd8a511 P fb6004b8640a4906aa0cfe63e383d40e: Time spent bootstrapping tablet: real 0.062s user 0.045s sys 0.009s
I20240502 21:27:54.250758 28647 raft_consensus.cc:348] T adf80b9d71314010a7a4cf052fd8a511 P fb6004b8640a4906aa0cfe63e383d40e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } attrs { replace: true } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: NON_VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: true } }
I20240502 21:27:54.251037 28647 raft_consensus.cc:729] T adf80b9d71314010a7a4cf052fd8a511 P fb6004b8640a4906aa0cfe63e383d40e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: fb6004b8640a4906aa0cfe63e383d40e, State: Initialized, Role: LEARNER
I20240502 21:27:54.251237 28647 consensus_queue.cc:260] T adf80b9d71314010a7a4cf052fd8a511 P fb6004b8640a4906aa0cfe63e383d40e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } attrs { replace: true } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: NON_VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: true } }
I20240502 21:27:54.251694 28658 tablet_copy_client.cc:523] T 11cb9a2fbda946fb9f303f2c465bccb6 P a3e500d7077a48acb5edb33f7e178c81: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:27:54.251765 28647 ts_tablet_manager.cc:1398] T adf80b9d71314010a7a4cf052fd8a511 P fb6004b8640a4906aa0cfe63e383d40e: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 21:27:54.252921 23935 tablet_copy_service.cc:342] P 6810a866decb4f5089f7d35c870e4204: Request end of tablet copy session fb6004b8640a4906aa0cfe63e383d40e-adf80b9d71314010a7a4cf052fd8a511 received from {username='slave'} at 127.15.255.134:44215
I20240502 21:27:54.253046 23935 tablet_copy_service.cc:433] P 6810a866decb4f5089f7d35c870e4204: ending tablet copy session fb6004b8640a4906aa0cfe63e383d40e-adf80b9d71314010a7a4cf052fd8a511 on tablet adf80b9d71314010a7a4cf052fd8a511 with peer fb6004b8640a4906aa0cfe63e383d40e
I20240502 21:27:54.255069 28658 tablet_bootstrap.cc:492] T 11cb9a2fbda946fb9f303f2c465bccb6 P a3e500d7077a48acb5edb33f7e178c81: Bootstrap starting.
I20240502 21:27:54.263109 28663 tablet_copy_client.cc:523] T 77fc7226878f44d3884ccb1fc41c44a4 P 4f58800ba2ed45eabd81c2f2e691aacd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:27:54.266896 28663 tablet_bootstrap.cc:492] T 77fc7226878f44d3884ccb1fc41c44a4 P 4f58800ba2ed45eabd81c2f2e691aacd: Bootstrap starting.
I20240502 21:27:54.283629 26012 raft_consensus.cc:1203] T adf80b9d71314010a7a4cf052fd8a511 P fb6004b8640a4906aa0cfe63e383d40e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240502 21:27:54.289273 28668 tablet_copy_client.cc:523] T f84bbe304597450c978ae28e0c4e7a3e P fb6004b8640a4906aa0cfe63e383d40e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:27:54.292899 28668 tablet_bootstrap.cc:492] T f84bbe304597450c978ae28e0c4e7a3e P fb6004b8640a4906aa0cfe63e383d40e: Bootstrap starting.
I20240502 21:27:54.308413 28652 tablet_bootstrap.cc:492] T c9b5be01b5e941b1848df79c7a9db348 P 4f58800ba2ed45eabd81c2f2e691aacd: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:27:54.309216 28652 tablet_bootstrap.cc:492] T c9b5be01b5e941b1848df79c7a9db348 P 4f58800ba2ed45eabd81c2f2e691aacd: Bootstrap complete.
I20240502 21:27:54.309732 28652 ts_tablet_manager.cc:1367] T c9b5be01b5e941b1848df79c7a9db348 P 4f58800ba2ed45eabd81c2f2e691aacd: Time spent bootstrapping tablet: real 0.077s user 0.057s sys 0.009s
I20240502 21:27:54.316874 28652 raft_consensus.cc:348] T c9b5be01b5e941b1848df79c7a9db348 P 4f58800ba2ed45eabd81c2f2e691aacd [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } attrs { replace: true } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } } peers { permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: NON_VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: true } }
I20240502 21:27:54.317505 28652 raft_consensus.cc:729] T c9b5be01b5e941b1848df79c7a9db348 P 4f58800ba2ed45eabd81c2f2e691aacd [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4f58800ba2ed45eabd81c2f2e691aacd, State: Initialized, Role: LEARNER
I20240502 21:27:54.318171 28652 consensus_queue.cc:260] T c9b5be01b5e941b1848df79c7a9db348 P 4f58800ba2ed45eabd81c2f2e691aacd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } attrs { replace: true } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } } peers { permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: NON_VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: true } }
I20240502 21:27:54.328634 28658 tablet_bootstrap.cc:492] T 11cb9a2fbda946fb9f303f2c465bccb6 P a3e500d7077a48acb5edb33f7e178c81: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:27:54.329123 28658 tablet_bootstrap.cc:492] T 11cb9a2fbda946fb9f303f2c465bccb6 P a3e500d7077a48acb5edb33f7e178c81: Bootstrap complete.
I20240502 21:27:54.329492 28658 ts_tablet_manager.cc:1367] T 11cb9a2fbda946fb9f303f2c465bccb6 P a3e500d7077a48acb5edb33f7e178c81: Time spent bootstrapping tablet: real 0.074s user 0.042s sys 0.017s
I20240502 21:27:54.330334 28658 raft_consensus.cc:348] T 11cb9a2fbda946fb9f303f2c465bccb6 P a3e500d7077a48acb5edb33f7e178c81 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } attrs { replace: true } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: true } }
I20240502 21:27:54.330801 28658 raft_consensus.cc:729] T 11cb9a2fbda946fb9f303f2c465bccb6 P a3e500d7077a48acb5edb33f7e178c81 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a3e500d7077a48acb5edb33f7e178c81, State: Initialized, Role: LEARNER
I20240502 21:27:54.331120 28658 consensus_queue.cc:260] T 11cb9a2fbda946fb9f303f2c465bccb6 P a3e500d7077a48acb5edb33f7e178c81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } attrs { replace: true } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: true } }
I20240502 21:27:54.332947 28658 ts_tablet_manager.cc:1398] T 11cb9a2fbda946fb9f303f2c465bccb6 P a3e500d7077a48acb5edb33f7e178c81: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 21:27:54.333969 24338 tablet_copy_service.cc:342] P 038c5d9d1d7a49c981c4c34fbc00d9e6: Request end of tablet copy session a3e500d7077a48acb5edb33f7e178c81-11cb9a2fbda946fb9f303f2c465bccb6 received from {username='slave'} at 127.15.255.136:38895
I20240502 21:27:54.334111 24338 tablet_copy_service.cc:433] P 038c5d9d1d7a49c981c4c34fbc00d9e6: ending tablet copy session a3e500d7077a48acb5edb33f7e178c81-11cb9a2fbda946fb9f303f2c465bccb6 on tablet 11cb9a2fbda946fb9f303f2c465bccb6 with peer a3e500d7077a48acb5edb33f7e178c81
I20240502 21:27:54.341185 28652 ts_tablet_manager.cc:1398] T c9b5be01b5e941b1848df79c7a9db348 P 4f58800ba2ed45eabd81c2f2e691aacd: Time spent starting tablet: real 0.031s user 0.017s sys 0.011s
I20240502 21:27:54.342232 24070 tablet_copy_service.cc:342] P 6c194a6db6c145acafc0f3068b8ce1fd: Request end of tablet copy session 4f58800ba2ed45eabd81c2f2e691aacd-c9b5be01b5e941b1848df79c7a9db348 received from {username='slave'} at 127.15.255.137:60133
I20240502 21:27:54.342376 24070 tablet_copy_service.cc:433] P 6c194a6db6c145acafc0f3068b8ce1fd: ending tablet copy session 4f58800ba2ed45eabd81c2f2e691aacd-c9b5be01b5e941b1848df79c7a9db348 on tablet c9b5be01b5e941b1848df79c7a9db348 with peer 4f58800ba2ed45eabd81c2f2e691aacd
I20240502 21:27:54.349025 28674 tablet_copy_client.cc:523] T 0b3c514213bb40ca98955d6409a3b372 P 4f58800ba2ed45eabd81c2f2e691aacd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:27:54.351811 28674 tablet_bootstrap.cc:492] T 0b3c514213bb40ca98955d6409a3b372 P 4f58800ba2ed45eabd81c2f2e691aacd: Bootstrap starting.
I20240502 21:27:54.354676 28663 tablet_bootstrap.cc:492] T 77fc7226878f44d3884ccb1fc41c44a4 P 4f58800ba2ed45eabd81c2f2e691aacd: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:27:54.355211 28663 tablet_bootstrap.cc:492] T 77fc7226878f44d3884ccb1fc41c44a4 P 4f58800ba2ed45eabd81c2f2e691aacd: Bootstrap complete.
I20240502 21:27:54.355547 28663 ts_tablet_manager.cc:1367] T 77fc7226878f44d3884ccb1fc41c44a4 P 4f58800ba2ed45eabd81c2f2e691aacd: Time spent bootstrapping tablet: real 0.089s user 0.055s sys 0.009s
I20240502 21:27:54.356668 28663 raft_consensus.cc:348] T 77fc7226878f44d3884ccb1fc41c44a4 P 4f58800ba2ed45eabd81c2f2e691aacd [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } attrs { replace: true } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } } peers { permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: NON_VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: true } }
I20240502 21:27:54.357030 28663 raft_consensus.cc:729] T 77fc7226878f44d3884ccb1fc41c44a4 P 4f58800ba2ed45eabd81c2f2e691aacd [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4f58800ba2ed45eabd81c2f2e691aacd, State: Initialized, Role: LEARNER
I20240502 21:27:54.357314 28663 consensus_queue.cc:260] T 77fc7226878f44d3884ccb1fc41c44a4 P 4f58800ba2ed45eabd81c2f2e691aacd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } attrs { replace: true } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } } peers { permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: NON_VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: true } }
I20240502 21:27:54.358074 28663 ts_tablet_manager.cc:1398] T 77fc7226878f44d3884ccb1fc41c44a4 P 4f58800ba2ed45eabd81c2f2e691aacd: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:27:54.358886 24070 tablet_copy_service.cc:342] P 6c194a6db6c145acafc0f3068b8ce1fd: Request end of tablet copy session 4f58800ba2ed45eabd81c2f2e691aacd-77fc7226878f44d3884ccb1fc41c44a4 received from {username='slave'} at 127.15.255.137:60133
I20240502 21:27:54.359035 24070 tablet_copy_service.cc:433] P 6c194a6db6c145acafc0f3068b8ce1fd: ending tablet copy session 4f58800ba2ed45eabd81c2f2e691aacd-77fc7226878f44d3884ccb1fc41c44a4 on tablet 77fc7226878f44d3884ccb1fc41c44a4 with peer 4f58800ba2ed45eabd81c2f2e691aacd
I20240502 21:27:54.377583 28684 tablet_copy_client.cc:523] T a274546743354e79bffa26ae7000e895 P 4f58800ba2ed45eabd81c2f2e691aacd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:27:54.381078 28684 tablet_bootstrap.cc:492] T a274546743354e79bffa26ae7000e895 P 4f58800ba2ed45eabd81c2f2e691aacd: Bootstrap starting.
I20240502 21:27:54.391839 28668 tablet_bootstrap.cc:492] T f84bbe304597450c978ae28e0c4e7a3e P fb6004b8640a4906aa0cfe63e383d40e: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:27:54.392263 28668 tablet_bootstrap.cc:492] T f84bbe304597450c978ae28e0c4e7a3e P fb6004b8640a4906aa0cfe63e383d40e: Bootstrap complete.
I20240502 21:27:54.392594 28668 ts_tablet_manager.cc:1367] T f84bbe304597450c978ae28e0c4e7a3e P fb6004b8640a4906aa0cfe63e383d40e: Time spent bootstrapping tablet: real 0.100s user 0.044s sys 0.012s
I20240502 21:27:54.393491 28668 raft_consensus.cc:348] T f84bbe304597450c978ae28e0c4e7a3e P fb6004b8640a4906aa0cfe63e383d40e [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } attrs { replace: true } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: NON_VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: true } }
I20240502 21:27:54.393792 28668 raft_consensus.cc:729] T f84bbe304597450c978ae28e0c4e7a3e P fb6004b8640a4906aa0cfe63e383d40e [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: fb6004b8640a4906aa0cfe63e383d40e, State: Initialized, Role: LEARNER
I20240502 21:27:54.394006 28668 consensus_queue.cc:260] T f84bbe304597450c978ae28e0c4e7a3e P fb6004b8640a4906aa0cfe63e383d40e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } attrs { replace: true } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: NON_VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: true } }
I20240502 21:27:54.395443 28668 ts_tablet_manager.cc:1398] T f84bbe304597450c978ae28e0c4e7a3e P fb6004b8640a4906aa0cfe63e383d40e: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240502 21:27:54.400728 26300 tablet_copy_service.cc:342] P a3e500d7077a48acb5edb33f7e178c81: Request end of tablet copy session fb6004b8640a4906aa0cfe63e383d40e-f84bbe304597450c978ae28e0c4e7a3e received from {username='slave'} at 127.15.255.134:54653
I20240502 21:27:54.400883 26300 tablet_copy_service.cc:433] P a3e500d7077a48acb5edb33f7e178c81: ending tablet copy session fb6004b8640a4906aa0cfe63e383d40e-f84bbe304597450c978ae28e0c4e7a3e on tablet f84bbe304597450c978ae28e0c4e7a3e with peer fb6004b8640a4906aa0cfe63e383d40e
I20240502 21:27:54.414225 28674 tablet_bootstrap.cc:492] T 0b3c514213bb40ca98955d6409a3b372 P 4f58800ba2ed45eabd81c2f2e691aacd: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:27:54.414736 28674 tablet_bootstrap.cc:492] T 0b3c514213bb40ca98955d6409a3b372 P 4f58800ba2ed45eabd81c2f2e691aacd: Bootstrap complete.
I20240502 21:27:54.415072 28674 ts_tablet_manager.cc:1367] T 0b3c514213bb40ca98955d6409a3b372 P 4f58800ba2ed45eabd81c2f2e691aacd: Time spent bootstrapping tablet: real 0.063s user 0.050s sys 0.013s
I20240502 21:27:54.415956 28674 raft_consensus.cc:348] T 0b3c514213bb40ca98955d6409a3b372 P 4f58800ba2ed45eabd81c2f2e691aacd [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } attrs { replace: true } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: false } } peers { permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: NON_VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: true } }
I20240502 21:27:54.416311 28674 raft_consensus.cc:729] T 0b3c514213bb40ca98955d6409a3b372 P 4f58800ba2ed45eabd81c2f2e691aacd [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4f58800ba2ed45eabd81c2f2e691aacd, State: Initialized, Role: LEARNER
I20240502 21:27:54.416541 28710 tablet_copy_client.cc:523] T 4eebf4508766421eb6c547a0fbbc28b1 P a3e500d7077a48acb5edb33f7e178c81: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:27:54.416579 28674 consensus_queue.cc:260] T 0b3c514213bb40ca98955d6409a3b372 P 4f58800ba2ed45eabd81c2f2e691aacd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } attrs { replace: true } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: false } } peers { permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: NON_VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: true } }
I20240502 21:27:54.417205 28674 ts_tablet_manager.cc:1398] T 0b3c514213bb40ca98955d6409a3b372 P 4f58800ba2ed45eabd81c2f2e691aacd: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 21:27:54.418782 24205 tablet_copy_service.cc:342] P 009d3e7e8a8b4e2497de3a73d01ced49: Request end of tablet copy session 4f58800ba2ed45eabd81c2f2e691aacd-0b3c514213bb40ca98955d6409a3b372 received from {username='slave'} at 127.15.255.137:51277
I20240502 21:27:54.418902 24205 tablet_copy_service.cc:433] P 009d3e7e8a8b4e2497de3a73d01ced49: ending tablet copy session 4f58800ba2ed45eabd81c2f2e691aacd-0b3c514213bb40ca98955d6409a3b372 on tablet 0b3c514213bb40ca98955d6409a3b372 with peer 4f58800ba2ed45eabd81c2f2e691aacd
I20240502 21:27:54.419859 28710 tablet_bootstrap.cc:492] T 4eebf4508766421eb6c547a0fbbc28b1 P a3e500d7077a48acb5edb33f7e178c81: Bootstrap starting.
I20240502 21:27:54.430817 28724 tablet_copy_client.cc:523] T 30236241b670443489240f191d9b9d50 P a3e500d7077a48acb5edb33f7e178c81: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:27:54.433670 28724 tablet_bootstrap.cc:492] T 30236241b670443489240f191d9b9d50 P a3e500d7077a48acb5edb33f7e178c81: Bootstrap starting.
I20240502 21:27:54.443593 28734 tablet_copy_client.cc:523] T fe044374ad7b4999afc72ab812466749 P fb6004b8640a4906aa0cfe63e383d40e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:27:54.447232 28734 tablet_bootstrap.cc:492] T fe044374ad7b4999afc72ab812466749 P fb6004b8640a4906aa0cfe63e383d40e: Bootstrap starting.
I20240502 21:27:54.458540 28684 tablet_bootstrap.cc:492] T a274546743354e79bffa26ae7000e895 P 4f58800ba2ed45eabd81c2f2e691aacd: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:27:54.459060 28684 tablet_bootstrap.cc:492] T a274546743354e79bffa26ae7000e895 P 4f58800ba2ed45eabd81c2f2e691aacd: Bootstrap complete.
I20240502 21:27:54.459390 28684 ts_tablet_manager.cc:1367] T a274546743354e79bffa26ae7000e895 P 4f58800ba2ed45eabd81c2f2e691aacd: Time spent bootstrapping tablet: real 0.078s user 0.042s sys 0.016s
I20240502 21:27:54.460227 28684 raft_consensus.cc:348] T a274546743354e79bffa26ae7000e895 P 4f58800ba2ed45eabd81c2f2e691aacd [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } attrs { replace: true } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } } peers { permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: NON_VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: true } }
I20240502 21:27:54.460644 28684 raft_consensus.cc:729] T a274546743354e79bffa26ae7000e895 P 4f58800ba2ed45eabd81c2f2e691aacd [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4f58800ba2ed45eabd81c2f2e691aacd, State: Initialized, Role: LEARNER
I20240502 21:27:54.460913 28684 consensus_queue.cc:260] T a274546743354e79bffa26ae7000e895 P 4f58800ba2ed45eabd81c2f2e691aacd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } attrs { replace: true } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } } peers { permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: NON_VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: true } }
I20240502 21:27:54.461707 28684 ts_tablet_manager.cc:1398] T a274546743354e79bffa26ae7000e895 P 4f58800ba2ed45eabd81c2f2e691aacd: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:27:54.462590 24070 tablet_copy_service.cc:342] P 6c194a6db6c145acafc0f3068b8ce1fd: Request end of tablet copy session 4f58800ba2ed45eabd81c2f2e691aacd-a274546743354e79bffa26ae7000e895 received from {username='slave'} at 127.15.255.137:60133
I20240502 21:27:54.462746 24070 tablet_copy_service.cc:433] P 6c194a6db6c145acafc0f3068b8ce1fd: ending tablet copy session 4f58800ba2ed45eabd81c2f2e691aacd-a274546743354e79bffa26ae7000e895 on tablet a274546743354e79bffa26ae7000e895 with peer 4f58800ba2ed45eabd81c2f2e691aacd
I20240502 21:27:54.470096 28745 tablet_copy_client.cc:523] T 52f621ea3f174490948eae854c706a92 P 4f58800ba2ed45eabd81c2f2e691aacd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:27:54.473888 28745 tablet_bootstrap.cc:492] T 52f621ea3f174490948eae854c706a92 P 4f58800ba2ed45eabd81c2f2e691aacd: Bootstrap starting.
I20240502 21:27:54.479105 28710 tablet_bootstrap.cc:492] T 4eebf4508766421eb6c547a0fbbc28b1 P a3e500d7077a48acb5edb33f7e178c81: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:27:54.479609 28710 tablet_bootstrap.cc:492] T 4eebf4508766421eb6c547a0fbbc28b1 P a3e500d7077a48acb5edb33f7e178c81: Bootstrap complete.
I20240502 21:27:54.479941 28710 ts_tablet_manager.cc:1367] T 4eebf4508766421eb6c547a0fbbc28b1 P a3e500d7077a48acb5edb33f7e178c81: Time spent bootstrapping tablet: real 0.060s user 0.048s sys 0.005s
I20240502 21:27:54.480893 28710 raft_consensus.cc:348] T 4eebf4508766421eb6c547a0fbbc28b1 P a3e500d7077a48acb5edb33f7e178c81 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } attrs { replace: true } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: true } }
I20240502 21:27:54.481283 28710 raft_consensus.cc:729] T 4eebf4508766421eb6c547a0fbbc28b1 P a3e500d7077a48acb5edb33f7e178c81 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a3e500d7077a48acb5edb33f7e178c81, State: Initialized, Role: LEARNER
I20240502 21:27:54.481585 28710 consensus_queue.cc:260] T 4eebf4508766421eb6c547a0fbbc28b1 P a3e500d7077a48acb5edb33f7e178c81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } attrs { replace: true } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: true } }
I20240502 21:27:54.483258 28710 ts_tablet_manager.cc:1398] T 4eebf4508766421eb6c547a0fbbc28b1 P a3e500d7077a48acb5edb33f7e178c81: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 21:27:54.484105 23935 tablet_copy_service.cc:342] P 6810a866decb4f5089f7d35c870e4204: Request end of tablet copy session a3e500d7077a48acb5edb33f7e178c81-4eebf4508766421eb6c547a0fbbc28b1 received from {username='slave'} at 127.15.255.136:46499
I20240502 21:27:54.484253 23935 tablet_copy_service.cc:433] P 6810a866decb4f5089f7d35c870e4204: ending tablet copy session a3e500d7077a48acb5edb33f7e178c81-4eebf4508766421eb6c547a0fbbc28b1 on tablet 4eebf4508766421eb6c547a0fbbc28b1 with peer a3e500d7077a48acb5edb33f7e178c81
I20240502 21:27:54.488200 28756 tablet_copy_client.cc:523] T 85e77d5faa8346cd8a99dcbcdf75f250 P ee27c3f9e0ec46caa742ac386d6b7c20: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:27:54.491777 28756 tablet_bootstrap.cc:492] T 85e77d5faa8346cd8a99dcbcdf75f250 P ee27c3f9e0ec46caa742ac386d6b7c20: Bootstrap starting.
I20240502 21:27:54.513022 28734 tablet_bootstrap.cc:492] T fe044374ad7b4999afc72ab812466749 P fb6004b8640a4906aa0cfe63e383d40e: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:27:54.513499 28734 tablet_bootstrap.cc:492] T fe044374ad7b4999afc72ab812466749 P fb6004b8640a4906aa0cfe63e383d40e: Bootstrap complete.
I20240502 21:27:54.513806 28734 ts_tablet_manager.cc:1367] T fe044374ad7b4999afc72ab812466749 P fb6004b8640a4906aa0cfe63e383d40e: Time spent bootstrapping tablet: real 0.067s user 0.042s sys 0.009s
I20240502 21:27:54.514636 28734 raft_consensus.cc:348] T fe044374ad7b4999afc72ab812466749 P fb6004b8640a4906aa0cfe63e383d40e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } attrs { replace: true } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: false } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: NON_VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: true } }
I20240502 21:27:54.514940 28734 raft_consensus.cc:729] T fe044374ad7b4999afc72ab812466749 P fb6004b8640a4906aa0cfe63e383d40e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: fb6004b8640a4906aa0cfe63e383d40e, State: Initialized, Role: LEARNER
I20240502 21:27:54.515228 28734 consensus_queue.cc:260] T fe044374ad7b4999afc72ab812466749 P fb6004b8640a4906aa0cfe63e383d40e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } attrs { replace: true } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: false } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: NON_VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: true } }
I20240502 21:27:54.516786 28734 ts_tablet_manager.cc:1398] T fe044374ad7b4999afc72ab812466749 P fb6004b8640a4906aa0cfe63e383d40e: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 21:27:54.517516 24338 tablet_copy_service.cc:342] P 038c5d9d1d7a49c981c4c34fbc00d9e6: Request end of tablet copy session fb6004b8640a4906aa0cfe63e383d40e-fe044374ad7b4999afc72ab812466749 received from {username='slave'} at 127.15.255.134:51733
I20240502 21:27:54.517685 24338 tablet_copy_service.cc:433] P 038c5d9d1d7a49c981c4c34fbc00d9e6: ending tablet copy session fb6004b8640a4906aa0cfe63e383d40e-fe044374ad7b4999afc72ab812466749 on tablet fe044374ad7b4999afc72ab812466749 with peer fb6004b8640a4906aa0cfe63e383d40e
I20240502 21:27:54.533804 28780 tablet_copy_client.cc:523] T cca45c89b9fe40e7800d95c1598a9e52 P a3e500d7077a48acb5edb33f7e178c81: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:27:54.536252 28724 tablet_bootstrap.cc:492] T 30236241b670443489240f191d9b9d50 P a3e500d7077a48acb5edb33f7e178c81: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:27:54.536705 28724 tablet_bootstrap.cc:492] T 30236241b670443489240f191d9b9d50 P a3e500d7077a48acb5edb33f7e178c81: Bootstrap complete.
I20240502 21:27:54.537005 28724 ts_tablet_manager.cc:1367] T 30236241b670443489240f191d9b9d50 P a3e500d7077a48acb5edb33f7e178c81: Time spent bootstrapping tablet: real 0.103s user 0.049s sys 0.020s
I20240502 21:27:54.537672 28780 tablet_bootstrap.cc:492] T cca45c89b9fe40e7800d95c1598a9e52 P a3e500d7077a48acb5edb33f7e178c81: Bootstrap starting.
I20240502 21:27:54.537775 28724 raft_consensus.cc:348] T 30236241b670443489240f191d9b9d50 P a3e500d7077a48acb5edb33f7e178c81 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } attrs { replace: true } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: false } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: true } }
I20240502 21:27:54.538028 28724 raft_consensus.cc:729] T 30236241b670443489240f191d9b9d50 P a3e500d7077a48acb5edb33f7e178c81 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a3e500d7077a48acb5edb33f7e178c81, State: Initialized, Role: LEARNER
I20240502 21:27:54.538200 28724 consensus_queue.cc:260] T 30236241b670443489240f191d9b9d50 P a3e500d7077a48acb5edb33f7e178c81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } attrs { replace: true } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: false } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: true } }
I20240502 21:27:54.538743 28724 ts_tablet_manager.cc:1398] T 30236241b670443489240f191d9b9d50 P a3e500d7077a48acb5edb33f7e178c81: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 21:27:54.541167 23935 tablet_copy_service.cc:342] P 6810a866decb4f5089f7d35c870e4204: Request end of tablet copy session a3e500d7077a48acb5edb33f7e178c81-30236241b670443489240f191d9b9d50 received from {username='slave'} at 127.15.255.136:46499
I20240502 21:27:54.541306 23935 tablet_copy_service.cc:433] P 6810a866decb4f5089f7d35c870e4204: ending tablet copy session a3e500d7077a48acb5edb33f7e178c81-30236241b670443489240f191d9b9d50 on tablet 30236241b670443489240f191d9b9d50 with peer a3e500d7077a48acb5edb33f7e178c81
I20240502 21:27:54.562850 28756 tablet_bootstrap.cc:492] T 85e77d5faa8346cd8a99dcbcdf75f250 P ee27c3f9e0ec46caa742ac386d6b7c20: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:27:54.563405 28756 tablet_bootstrap.cc:492] T 85e77d5faa8346cd8a99dcbcdf75f250 P ee27c3f9e0ec46caa742ac386d6b7c20: Bootstrap complete.
I20240502 21:27:54.563812 28756 ts_tablet_manager.cc:1367] T 85e77d5faa8346cd8a99dcbcdf75f250 P ee27c3f9e0ec46caa742ac386d6b7c20: Time spent bootstrapping tablet: real 0.072s user 0.056s sys 0.004s
I20240502 21:27:54.564795 28756 raft_consensus.cc:348] T 85e77d5faa8346cd8a99dcbcdf75f250 P ee27c3f9e0ec46caa742ac386d6b7c20 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } attrs { replace: true } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: NON_VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: true } }
I20240502 21:27:54.565188 28756 raft_consensus.cc:729] T 85e77d5faa8346cd8a99dcbcdf75f250 P ee27c3f9e0ec46caa742ac386d6b7c20 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ee27c3f9e0ec46caa742ac386d6b7c20, State: Initialized, Role: LEARNER
I20240502 21:27:54.565485 28756 consensus_queue.cc:260] T 85e77d5faa8346cd8a99dcbcdf75f250 P ee27c3f9e0ec46caa742ac386d6b7c20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } attrs { replace: true } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: NON_VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: true } }
I20240502 21:27:54.566288 28756 ts_tablet_manager.cc:1398] T 85e77d5faa8346cd8a99dcbcdf75f250 P ee27c3f9e0ec46caa742ac386d6b7c20: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:27:54.567095 24205 tablet_copy_service.cc:342] P 009d3e7e8a8b4e2497de3a73d01ced49: Request end of tablet copy session ee27c3f9e0ec46caa742ac386d6b7c20-85e77d5faa8346cd8a99dcbcdf75f250 received from {username='slave'} at 127.15.255.133:44245
I20240502 21:27:54.567216 24205 tablet_copy_service.cc:433] P 009d3e7e8a8b4e2497de3a73d01ced49: ending tablet copy session ee27c3f9e0ec46caa742ac386d6b7c20-85e77d5faa8346cd8a99dcbcdf75f250 on tablet 85e77d5faa8346cd8a99dcbcdf75f250 with peer ee27c3f9e0ec46caa742ac386d6b7c20
I20240502 21:27:54.569900 28745 tablet_bootstrap.cc:492] T 52f621ea3f174490948eae854c706a92 P 4f58800ba2ed45eabd81c2f2e691aacd: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:27:54.570363 28745 tablet_bootstrap.cc:492] T 52f621ea3f174490948eae854c706a92 P 4f58800ba2ed45eabd81c2f2e691aacd: Bootstrap complete.
I20240502 21:27:54.570670 28745 ts_tablet_manager.cc:1367] T 52f621ea3f174490948eae854c706a92 P 4f58800ba2ed45eabd81c2f2e691aacd: Time spent bootstrapping tablet: real 0.097s user 0.054s sys 0.008s
I20240502 21:27:54.571833 28745 raft_consensus.cc:348] T 52f621ea3f174490948eae854c706a92 P 4f58800ba2ed45eabd81c2f2e691aacd [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } attrs { replace: true } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: false } } peers { permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: NON_VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: true } }
I20240502 21:27:54.572141 28745 raft_consensus.cc:729] T 52f621ea3f174490948eae854c706a92 P 4f58800ba2ed45eabd81c2f2e691aacd [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4f58800ba2ed45eabd81c2f2e691aacd, State: Initialized, Role: LEARNER
I20240502 21:27:54.572381 28745 consensus_queue.cc:260] T 52f621ea3f174490948eae854c706a92 P 4f58800ba2ed45eabd81c2f2e691aacd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } attrs { replace: true } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: false } } peers { permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: NON_VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: true } }
I20240502 21:27:54.573076 28745 ts_tablet_manager.cc:1398] T 52f621ea3f174490948eae854c706a92 P 4f58800ba2ed45eabd81c2f2e691aacd: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 21:27:54.574549 24338 tablet_copy_service.cc:342] P 038c5d9d1d7a49c981c4c34fbc00d9e6: Request end of tablet copy session 4f58800ba2ed45eabd81c2f2e691aacd-52f621ea3f174490948eae854c706a92 received from {username='slave'} at 127.15.255.137:48505
I20240502 21:27:54.574719 24338 tablet_copy_service.cc:433] P 038c5d9d1d7a49c981c4c34fbc00d9e6: ending tablet copy session 4f58800ba2ed45eabd81c2f2e691aacd-52f621ea3f174490948eae854c706a92 on tablet 52f621ea3f174490948eae854c706a92 with peer 4f58800ba2ed45eabd81c2f2e691aacd
I20240502 21:27:54.592842 28780 tablet_bootstrap.cc:492] T cca45c89b9fe40e7800d95c1598a9e52 P a3e500d7077a48acb5edb33f7e178c81: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:27:54.593276 28780 tablet_bootstrap.cc:492] T cca45c89b9fe40e7800d95c1598a9e52 P a3e500d7077a48acb5edb33f7e178c81: Bootstrap complete.
I20240502 21:27:54.593565 28780 ts_tablet_manager.cc:1367] T cca45c89b9fe40e7800d95c1598a9e52 P a3e500d7077a48acb5edb33f7e178c81: Time spent bootstrapping tablet: real 0.056s user 0.034s sys 0.008s
I20240502 21:27:54.594282 28780 raft_consensus.cc:348] T cca45c89b9fe40e7800d95c1598a9e52 P a3e500d7077a48acb5edb33f7e178c81 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } attrs { replace: true } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: false } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: true } }
I20240502 21:27:54.594569 28780 raft_consensus.cc:729] T cca45c89b9fe40e7800d95c1598a9e52 P a3e500d7077a48acb5edb33f7e178c81 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a3e500d7077a48acb5edb33f7e178c81, State: Initialized, Role: LEARNER
I20240502 21:27:54.594831 28780 consensus_queue.cc:260] T cca45c89b9fe40e7800d95c1598a9e52 P a3e500d7077a48acb5edb33f7e178c81 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } attrs { replace: true } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: false } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: true } }
I20240502 21:27:54.595417 28780 ts_tablet_manager.cc:1398] T cca45c89b9fe40e7800d95c1598a9e52 P a3e500d7077a48acb5edb33f7e178c81: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:27:54.596203 24205 tablet_copy_service.cc:342] P 009d3e7e8a8b4e2497de3a73d01ced49: Request end of tablet copy session a3e500d7077a48acb5edb33f7e178c81-cca45c89b9fe40e7800d95c1598a9e52 received from {username='slave'} at 127.15.255.136:51951
I20240502 21:27:54.596325 24205 tablet_copy_service.cc:433] P 009d3e7e8a8b4e2497de3a73d01ced49: ending tablet copy session a3e500d7077a48acb5edb33f7e178c81-cca45c89b9fe40e7800d95c1598a9e52 on tablet cca45c89b9fe40e7800d95c1598a9e52 with peer a3e500d7077a48acb5edb33f7e178c81
I20240502 21:27:54.705727 27127 raft_consensus.cc:1203] T 77fc7226878f44d3884ccb1fc41c44a4 P 4f58800ba2ed45eabd81c2f2e691aacd [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240502 21:27:54.730800 25878 raft_consensus.cc:1203] T 549666d19e724a96801db183a07809d8 P ee27c3f9e0ec46caa742ac386d6b7c20 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240502 21:27:54.748956 27127 raft_consensus.cc:1203] T c9b5be01b5e941b1848df79c7a9db348 P 4f58800ba2ed45eabd81c2f2e691aacd [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240502 21:27:54.759701 27127 raft_consensus.cc:1203] T 0b3c514213bb40ca98955d6409a3b372 P 4f58800ba2ed45eabd81c2f2e691aacd [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240502 21:27:54.761785 29095 raft_consensus.cc:1050] T adf80b9d71314010a7a4cf052fd8a511 P 6810a866decb4f5089f7d35c870e4204: attempting to promote NON_VOTER fb6004b8640a4906aa0cfe63e383d40e to VOTER
I20240502 21:27:54.762620 29095 consensus_queue.cc:237] T adf80b9d71314010a7a4cf052fd8a511 P 6810a866decb4f5089f7d35c870e4204 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } attrs { replace: true } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: false } }
I20240502 21:27:54.764788 26012 raft_consensus.cc:1261] T adf80b9d71314010a7a4cf052fd8a511 P fb6004b8640a4906aa0cfe63e383d40e [term 1 LEARNER]: Refusing update from remote peer 6810a866decb4f5089f7d35c870e4204: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:27:54.764843 26279 raft_consensus.cc:1261] T adf80b9d71314010a7a4cf052fd8a511 P a3e500d7077a48acb5edb33f7e178c81 [term 1 FOLLOWER]: Refusing update from remote peer 6810a866decb4f5089f7d35c870e4204: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:27:54.764868 24317 raft_consensus.cc:1261] T adf80b9d71314010a7a4cf052fd8a511 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 1 FOLLOWER]: Refusing update from remote peer 6810a866decb4f5089f7d35c870e4204: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:27:54.765380 29097 consensus_queue.cc:1035] T adf80b9d71314010a7a4cf052fd8a511 P 6810a866decb4f5089f7d35c870e4204 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:27:54.765751 28790 consensus_queue.cc:1035] T adf80b9d71314010a7a4cf052fd8a511 P 6810a866decb4f5089f7d35c870e4204 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:27:54.766036 29095 consensus_queue.cc:1035] T adf80b9d71314010a7a4cf052fd8a511 P 6810a866decb4f5089f7d35c870e4204 [LEADER]: Connected to new peer: Peer: permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:27:54.773494 28790 raft_consensus.cc:2932] T adf80b9d71314010a7a4cf052fd8a511 P 6810a866decb4f5089f7d35c870e4204 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, fb6004b8640a4906aa0cfe63e383d40e (127.15.255.134) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } attrs { replace: true } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: false } } }
I20240502 21:27:54.774327 26012 raft_consensus.cc:2932] T adf80b9d71314010a7a4cf052fd8a511 P fb6004b8640a4906aa0cfe63e383d40e [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, fb6004b8640a4906aa0cfe63e383d40e (127.15.255.134) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } attrs { replace: true } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: false } } }
I20240502 21:27:54.776505 24317 raft_consensus.cc:2932] T adf80b9d71314010a7a4cf052fd8a511 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, fb6004b8640a4906aa0cfe63e383d40e (127.15.255.134) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } attrs { replace: true } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: false } } }
I20240502 21:27:54.777906 26279 raft_consensus.cc:2932] T adf80b9d71314010a7a4cf052fd8a511 P a3e500d7077a48acb5edb33f7e178c81 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, fb6004b8640a4906aa0cfe63e383d40e (127.15.255.134) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } attrs { replace: true } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: false } } }
I20240502 21:27:54.781401 23770 catalog_manager.cc:5484] T adf80b9d71314010a7a4cf052fd8a511 P fb6004b8640a4906aa0cfe63e383d40e reported cstate change: config changed from index 7 to 8, fb6004b8640a4906aa0cfe63e383d40e (127.15.255.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "6810a866decb4f5089f7d35c870e4204" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } attrs { replace: true } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: false } } }
I20240502 21:27:54.787653 23915 consensus_queue.cc:237] T adf80b9d71314010a7a4cf052fd8a511 P 6810a866decb4f5089f7d35c870e4204 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: false } }
I20240502 21:27:54.789144 26279 raft_consensus.cc:1203] T 30236241b670443489240f191d9b9d50 P a3e500d7077a48acb5edb33f7e178c81 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240502 21:27:54.789275 26280 raft_consensus.cc:1261] T adf80b9d71314010a7a4cf052fd8a511 P a3e500d7077a48acb5edb33f7e178c81 [term 1 FOLLOWER]: Refusing update from remote peer 6810a866decb4f5089f7d35c870e4204: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:27:54.789556 26012 raft_consensus.cc:1261] T adf80b9d71314010a7a4cf052fd8a511 P fb6004b8640a4906aa0cfe63e383d40e [term 1 FOLLOWER]: Refusing update from remote peer 6810a866decb4f5089f7d35c870e4204: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:27:54.789988 28790 consensus_queue.cc:1035] T adf80b9d71314010a7a4cf052fd8a511 P 6810a866decb4f5089f7d35c870e4204 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:27:54.790218 29095 consensus_queue.cc:1035] T adf80b9d71314010a7a4cf052fd8a511 P 6810a866decb4f5089f7d35c870e4204 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:27:54.792340 29095 raft_consensus.cc:2932] T adf80b9d71314010a7a4cf052fd8a511 P 6810a866decb4f5089f7d35c870e4204 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 038c5d9d1d7a49c981c4c34fbc00d9e6 (127.15.255.132) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: false } } }
I20240502 21:27:54.792946 26012 raft_consensus.cc:2932] T adf80b9d71314010a7a4cf052fd8a511 P fb6004b8640a4906aa0cfe63e383d40e [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 038c5d9d1d7a49c981c4c34fbc00d9e6 (127.15.255.132) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: false } } }
I20240502 21:27:54.793835 26279 raft_consensus.cc:2932] T adf80b9d71314010a7a4cf052fd8a511 P a3e500d7077a48acb5edb33f7e178c81 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 038c5d9d1d7a49c981c4c34fbc00d9e6 (127.15.255.132) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: false } } }
I20240502 21:27:54.795228 23757 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet adf80b9d71314010a7a4cf052fd8a511 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:27:54.798035 23771 catalog_manager.cc:5484] T adf80b9d71314010a7a4cf052fd8a511 P fb6004b8640a4906aa0cfe63e383d40e reported cstate change: config changed from index 8 to 9, VOTER 038c5d9d1d7a49c981c4c34fbc00d9e6 (127.15.255.132) evicted. New cstate: current_term: 1 leader_uuid: "6810a866decb4f5089f7d35c870e4204" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: false } } }
I20240502 21:27:54.802148 24297 tablet_service.cc:1514] Processing DeleteTablet for tablet adf80b9d71314010a7a4cf052fd8a511 with delete_type TABLET_DATA_TOMBSTONED (TS 038c5d9d1d7a49c981c4c34fbc00d9e6 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:58268
I20240502 21:27:54.803122 29106 tablet_replica.cc:330] T adf80b9d71314010a7a4cf052fd8a511 P 038c5d9d1d7a49c981c4c34fbc00d9e6: stopping tablet replica
I20240502 21:27:54.803340 29106 raft_consensus.cc:2229] T adf80b9d71314010a7a4cf052fd8a511 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:54.803692 29106 raft_consensus.cc:2258] T adf80b9d71314010a7a4cf052fd8a511 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:54.804361 29106 ts_tablet_manager.cc:1875] T adf80b9d71314010a7a4cf052fd8a511 P 038c5d9d1d7a49c981c4c34fbc00d9e6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:27:54.808542 29106 ts_tablet_manager.cc:1888] T adf80b9d71314010a7a4cf052fd8a511 P 038c5d9d1d7a49c981c4c34fbc00d9e6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240502 21:27:54.808667 29106 log.cc:1196] T adf80b9d71314010a7a4cf052fd8a511 P 038c5d9d1d7a49c981c4c34fbc00d9e6: Deleting WAL directory at /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-3/wal/wals/adf80b9d71314010a7a4cf052fd8a511
I20240502 21:27:54.809365 23755 catalog_manager.cc:4830] TS 038c5d9d1d7a49c981c4c34fbc00d9e6 (127.15.255.132:46813): tablet adf80b9d71314010a7a4cf052fd8a511 (table rebalance_test_table_0 [id=1e673d0be4694336bbf65d9c114c614f]) successfully deleted
I20240502 21:27:54.811815 27127 raft_consensus.cc:1203] T a274546743354e79bffa26ae7000e895 P 4f58800ba2ed45eabd81c2f2e691aacd [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240502 21:27:54.844847 26279 raft_consensus.cc:1203] T 11cb9a2fbda946fb9f303f2c465bccb6 P a3e500d7077a48acb5edb33f7e178c81 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240502 21:27:54.864848 26012 raft_consensus.cc:1203] T f84bbe304597450c978ae28e0c4e7a3e P fb6004b8640a4906aa0cfe63e383d40e [term 2 LEARNER]: Deduplicated request from leader. Original: 2.7->[2.8-2.8] Dedup: 2.8->[]
I20240502 21:27:54.898049 26012 raft_consensus.cc:1203] T fe044374ad7b4999afc72ab812466749 P fb6004b8640a4906aa0cfe63e383d40e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240502 21:27:54.942760 26279 raft_consensus.cc:1203] T 4eebf4508766421eb6c547a0fbbc28b1 P a3e500d7077a48acb5edb33f7e178c81 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240502 21:27:54.979948 26279 raft_consensus.cc:1203] T cca45c89b9fe40e7800d95c1598a9e52 P a3e500d7077a48acb5edb33f7e178c81 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240502 21:27:55.020118 25878 raft_consensus.cc:1203] T 85e77d5faa8346cd8a99dcbcdf75f250 P ee27c3f9e0ec46caa742ac386d6b7c20 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 21:27:55.056879 27127 raft_consensus.cc:1203] T 52f621ea3f174490948eae854c706a92 P 4f58800ba2ed45eabd81c2f2e691aacd [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240502 21:27:55.086659 29237 raft_consensus.cc:1050] T 77fc7226878f44d3884ccb1fc41c44a4 P 6c194a6db6c145acafc0f3068b8ce1fd: attempting to promote NON_VOTER 4f58800ba2ed45eabd81c2f2e691aacd to VOTER
I20240502 21:27:55.087677 29237 consensus_queue.cc:237] T 77fc7226878f44d3884ccb1fc41c44a4 P 6c194a6db6c145acafc0f3068b8ce1fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } attrs { replace: true } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } } peers { permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: false } }
I20240502 21:27:55.090390 27127 raft_consensus.cc:1261] T 77fc7226878f44d3884ccb1fc41c44a4 P 4f58800ba2ed45eabd81c2f2e691aacd [term 1 LEARNER]: Refusing update from remote peer 6c194a6db6c145acafc0f3068b8ce1fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:27:55.091303 26145 raft_consensus.cc:1261] T 77fc7226878f44d3884ccb1fc41c44a4 P e117afbf8819424e8a98476453dfde2a [term 1 FOLLOWER]: Refusing update from remote peer 6c194a6db6c145acafc0f3068b8ce1fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:27:55.091966 23915 raft_consensus.cc:1261] T 77fc7226878f44d3884ccb1fc41c44a4 P 6810a866decb4f5089f7d35c870e4204 [term 1 FOLLOWER]: Refusing update from remote peer 6c194a6db6c145acafc0f3068b8ce1fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:27:55.091881 29237 consensus_queue.cc:1035] T 77fc7226878f44d3884ccb1fc41c44a4 P 6c194a6db6c145acafc0f3068b8ce1fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:27:55.093329 28941 consensus_queue.cc:1035] T 77fc7226878f44d3884ccb1fc41c44a4 P 6c194a6db6c145acafc0f3068b8ce1fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:27:55.096680 29236 consensus_queue.cc:1035] T 77fc7226878f44d3884ccb1fc41c44a4 P 6c194a6db6c145acafc0f3068b8ce1fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:27:55.104452 29236 raft_consensus.cc:2932] T 77fc7226878f44d3884ccb1fc41c44a4 P 6c194a6db6c145acafc0f3068b8ce1fd [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 4f58800ba2ed45eabd81c2f2e691aacd (127.15.255.137) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } attrs { replace: true } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } } peers { permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: false } } }
I20240502 21:27:55.106026 26145 raft_consensus.cc:2932] T 77fc7226878f44d3884ccb1fc41c44a4 P e117afbf8819424e8a98476453dfde2a [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 4f58800ba2ed45eabd81c2f2e691aacd (127.15.255.137) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } attrs { replace: true } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } } peers { permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: false } } }
I20240502 21:27:55.105881 27127 raft_consensus.cc:2932] T 77fc7226878f44d3884ccb1fc41c44a4 P 4f58800ba2ed45eabd81c2f2e691aacd [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 4f58800ba2ed45eabd81c2f2e691aacd (127.15.255.137) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } attrs { replace: true } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } } peers { permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: false } } }
I20240502 21:27:55.111179 23916 raft_consensus.cc:2932] T 77fc7226878f44d3884ccb1fc41c44a4 P 6810a866decb4f5089f7d35c870e4204 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 4f58800ba2ed45eabd81c2f2e691aacd (127.15.255.137) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } attrs { replace: true } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } } peers { permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: false } } }
I20240502 21:27:55.112515 23765 catalog_manager.cc:5484] T 77fc7226878f44d3884ccb1fc41c44a4 P 4f58800ba2ed45eabd81c2f2e691aacd reported cstate change: config changed from index 7 to 8, 4f58800ba2ed45eabd81c2f2e691aacd (127.15.255.137) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } attrs { replace: true } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } } peers { permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: false } } }
I20240502 21:27:55.119114 24047 consensus_queue.cc:237] T 77fc7226878f44d3884ccb1fc41c44a4 P 6c194a6db6c145acafc0f3068b8ce1fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } } peers { permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: false } }
I20240502 21:27:55.120893 27127 raft_consensus.cc:1261] T 77fc7226878f44d3884ccb1fc41c44a4 P 4f58800ba2ed45eabd81c2f2e691aacd [term 1 FOLLOWER]: Refusing update from remote peer 6c194a6db6c145acafc0f3068b8ce1fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:27:55.121464 28941 consensus_queue.cc:1035] T 77fc7226878f44d3884ccb1fc41c44a4 P 6c194a6db6c145acafc0f3068b8ce1fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:27:55.122516 26145 raft_consensus.cc:1261] T 77fc7226878f44d3884ccb1fc41c44a4 P e117afbf8819424e8a98476453dfde2a [term 1 FOLLOWER]: Refusing update from remote peer 6c194a6db6c145acafc0f3068b8ce1fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:27:55.123577 29236 raft_consensus.cc:2932] T 77fc7226878f44d3884ccb1fc41c44a4 P 6c194a6db6c145acafc0f3068b8ce1fd [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 6810a866decb4f5089f7d35c870e4204 (127.15.255.129) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } } peers { permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: false } } }
I20240502 21:27:55.124785 27126 raft_consensus.cc:2932] T 77fc7226878f44d3884ccb1fc41c44a4 P 4f58800ba2ed45eabd81c2f2e691aacd [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 6810a866decb4f5089f7d35c870e4204 (127.15.255.129) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } } peers { permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: false } } }
I20240502 21:27:55.125100 29237 consensus_queue.cc:1035] T 77fc7226878f44d3884ccb1fc41c44a4 P 6c194a6db6c145acafc0f3068b8ce1fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:27:55.127255 23757 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 77fc7226878f44d3884ccb1fc41c44a4 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:27:55.127197 26147 raft_consensus.cc:2932] T 77fc7226878f44d3884ccb1fc41c44a4 P e117afbf8819424e8a98476453dfde2a [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 6810a866decb4f5089f7d35c870e4204 (127.15.255.129) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } } peers { permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: false } } }
I20240502 21:27:55.130474 23765 catalog_manager.cc:5484] T 77fc7226878f44d3884ccb1fc41c44a4 P 6c194a6db6c145acafc0f3068b8ce1fd reported cstate change: config changed from index 8 to 9, VOTER 6810a866decb4f5089f7d35c870e4204 (127.15.255.129) evicted. New cstate: current_term: 1 leader_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:27:55.135058 23894 tablet_service.cc:1514] Processing DeleteTablet for tablet 77fc7226878f44d3884ccb1fc41c44a4 with delete_type TABLET_DATA_TOMBSTONED (TS 6810a866decb4f5089f7d35c870e4204 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:33512
I20240502 21:27:55.136946 29251 tablet_replica.cc:330] T 77fc7226878f44d3884ccb1fc41c44a4 P 6810a866decb4f5089f7d35c870e4204: stopping tablet replica
I20240502 21:27:55.137204 29251 raft_consensus.cc:2229] T 77fc7226878f44d3884ccb1fc41c44a4 P 6810a866decb4f5089f7d35c870e4204 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:55.137595 29251 raft_consensus.cc:2258] T 77fc7226878f44d3884ccb1fc41c44a4 P 6810a866decb4f5089f7d35c870e4204 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:55.138916 29251 ts_tablet_manager.cc:1875] T 77fc7226878f44d3884ccb1fc41c44a4 P 6810a866decb4f5089f7d35c870e4204: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:27:55.143316 29251 ts_tablet_manager.cc:1888] T 77fc7226878f44d3884ccb1fc41c44a4 P 6810a866decb4f5089f7d35c870e4204: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240502 21:27:55.143467 29251 log.cc:1196] T 77fc7226878f44d3884ccb1fc41c44a4 P 6810a866decb4f5089f7d35c870e4204: Deleting WAL directory at /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-0/wal/wals/77fc7226878f44d3884ccb1fc41c44a4
I20240502 21:27:55.144755 23757 catalog_manager.cc:4830] TS 6810a866decb4f5089f7d35c870e4204 (127.15.255.129:40435): tablet 77fc7226878f44d3884ccb1fc41c44a4 (table rebalance_test_table_7 [id=754dbb3bce6a4875a0b642faf5315709]) successfully deleted
I20240502 21:27:55.172000 27234 raft_consensus.cc:1050] T 549666d19e724a96801db183a07809d8 P 6810a866decb4f5089f7d35c870e4204: attempting to promote NON_VOTER ee27c3f9e0ec46caa742ac386d6b7c20 to VOTER
I20240502 21:27:55.172892 27234 consensus_queue.cc:237] T 549666d19e724a96801db183a07809d8 P 6810a866decb4f5089f7d35c870e4204 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } attrs { replace: true } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: false } }
I20240502 21:27:55.174898 24047 raft_consensus.cc:1261] T 549666d19e724a96801db183a07809d8 P 6c194a6db6c145acafc0f3068b8ce1fd [term 1 FOLLOWER]: Refusing update from remote peer 6810a866decb4f5089f7d35c870e4204: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:27:55.174947 25878 raft_consensus.cc:1261] T 549666d19e724a96801db183a07809d8 P ee27c3f9e0ec46caa742ac386d6b7c20 [term 1 LEARNER]: Refusing update from remote peer 6810a866decb4f5089f7d35c870e4204: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:27:55.175135 26279 raft_consensus.cc:1261] T 549666d19e724a96801db183a07809d8 P a3e500d7077a48acb5edb33f7e178c81 [term 1 FOLLOWER]: Refusing update from remote peer 6810a866decb4f5089f7d35c870e4204: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:27:55.175413 27234 consensus_queue.cc:1035] T 549666d19e724a96801db183a07809d8 P 6810a866decb4f5089f7d35c870e4204 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:27:55.175724 28790 consensus_queue.cc:1035] T 549666d19e724a96801db183a07809d8 P 6810a866decb4f5089f7d35c870e4204 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:27:55.176040 29095 consensus_queue.cc:1035] T 549666d19e724a96801db183a07809d8 P 6810a866decb4f5089f7d35c870e4204 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:27:55.179527 27234 raft_consensus.cc:2932] T 549666d19e724a96801db183a07809d8 P 6810a866decb4f5089f7d35c870e4204 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, ee27c3f9e0ec46caa742ac386d6b7c20 (127.15.255.133) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } attrs { replace: true } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: false } } }
I20240502 21:27:55.180168 25878 raft_consensus.cc:2932] T 549666d19e724a96801db183a07809d8 P ee27c3f9e0ec46caa742ac386d6b7c20 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, ee27c3f9e0ec46caa742ac386d6b7c20 (127.15.255.133) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } attrs { replace: true } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: false } } }
I20240502 21:27:55.182945 24047 raft_consensus.cc:2932] T 549666d19e724a96801db183a07809d8 P 6c194a6db6c145acafc0f3068b8ce1fd [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, ee27c3f9e0ec46caa742ac386d6b7c20 (127.15.255.133) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } attrs { replace: true } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: false } } }
I20240502 21:27:55.184008 23766 catalog_manager.cc:5484] T 549666d19e724a96801db183a07809d8 P 6810a866decb4f5089f7d35c870e4204 reported cstate change: config changed from index 7 to 8, ee27c3f9e0ec46caa742ac386d6b7c20 (127.15.255.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "6810a866decb4f5089f7d35c870e4204" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:27:55.192943 26279 raft_consensus.cc:2932] T 549666d19e724a96801db183a07809d8 P a3e500d7077a48acb5edb33f7e178c81 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, ee27c3f9e0ec46caa742ac386d6b7c20 (127.15.255.133) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } attrs { replace: true } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: false } } }
I20240502 21:27:55.199404 23916 consensus_queue.cc:237] T 549666d19e724a96801db183a07809d8 P 6810a866decb4f5089f7d35c870e4204 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: false } }
I20240502 21:27:55.200896 26279 raft_consensus.cc:1261] T 549666d19e724a96801db183a07809d8 P a3e500d7077a48acb5edb33f7e178c81 [term 1 FOLLOWER]: Refusing update from remote peer 6810a866decb4f5089f7d35c870e4204: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:27:55.200997 25878 raft_consensus.cc:1261] T 549666d19e724a96801db183a07809d8 P ee27c3f9e0ec46caa742ac386d6b7c20 [term 1 FOLLOWER]: Refusing update from remote peer 6810a866decb4f5089f7d35c870e4204: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:27:55.201406 29095 consensus_queue.cc:1035] T 549666d19e724a96801db183a07809d8 P 6810a866decb4f5089f7d35c870e4204 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:27:55.201876 29095 consensus_queue.cc:1035] T 549666d19e724a96801db183a07809d8 P 6810a866decb4f5089f7d35c870e4204 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:27:55.203785 27234 raft_consensus.cc:2932] T 549666d19e724a96801db183a07809d8 P 6810a866decb4f5089f7d35c870e4204 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 6c194a6db6c145acafc0f3068b8ce1fd (127.15.255.130) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: false } } }
I20240502 21:27:55.206802 23757 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 549666d19e724a96801db183a07809d8 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:27:55.207670 26279 raft_consensus.cc:2932] T 549666d19e724a96801db183a07809d8 P a3e500d7077a48acb5edb33f7e178c81 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 6c194a6db6c145acafc0f3068b8ce1fd (127.15.255.130) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: false } } }
I20240502 21:27:55.207996 25878 raft_consensus.cc:2932] T 549666d19e724a96801db183a07809d8 P ee27c3f9e0ec46caa742ac386d6b7c20 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 6c194a6db6c145acafc0f3068b8ce1fd (127.15.255.130) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: false } } }
I20240502 21:27:55.210505 23766 catalog_manager.cc:5484] T 549666d19e724a96801db183a07809d8 P 6810a866decb4f5089f7d35c870e4204 reported cstate change: config changed from index 8 to 9, VOTER 6c194a6db6c145acafc0f3068b8ce1fd (127.15.255.130) evicted. New cstate: current_term: 1 leader_uuid: "6810a866decb4f5089f7d35c870e4204" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:27:55.216892 24027 tablet_service.cc:1514] Processing DeleteTablet for tablet 549666d19e724a96801db183a07809d8 with delete_type TABLET_DATA_TOMBSTONED (TS 6c194a6db6c145acafc0f3068b8ce1fd not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:47386
I20240502 21:27:55.217636 29256 tablet_replica.cc:330] T 549666d19e724a96801db183a07809d8 P 6c194a6db6c145acafc0f3068b8ce1fd: stopping tablet replica
I20240502 21:27:55.217839 29256 raft_consensus.cc:2229] T 549666d19e724a96801db183a07809d8 P 6c194a6db6c145acafc0f3068b8ce1fd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:55.218072 29256 raft_consensus.cc:2258] T 549666d19e724a96801db183a07809d8 P 6c194a6db6c145acafc0f3068b8ce1fd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:55.218910 29256 ts_tablet_manager.cc:1875] T 549666d19e724a96801db183a07809d8 P 6c194a6db6c145acafc0f3068b8ce1fd: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:27:55.223261 29256 ts_tablet_manager.cc:1888] T 549666d19e724a96801db183a07809d8 P 6c194a6db6c145acafc0f3068b8ce1fd: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240502 21:27:55.223357 29256 log.cc:1196] T 549666d19e724a96801db183a07809d8 P 6c194a6db6c145acafc0f3068b8ce1fd: Deleting WAL directory at /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-1/wal/wals/549666d19e724a96801db183a07809d8
I20240502 21:27:55.223986 23757 catalog_manager.cc:4830] TS 6c194a6db6c145acafc0f3068b8ce1fd (127.15.255.130:41235): tablet 549666d19e724a96801db183a07809d8 (table rebalance_test_table_3 [id=933ce90acb9f4b91ab6bd41e87a8fe99]) successfully deleted
I20240502 21:27:55.249538 29257 raft_consensus.cc:1050] T 0b3c514213bb40ca98955d6409a3b372 P 009d3e7e8a8b4e2497de3a73d01ced49: attempting to promote NON_VOTER 4f58800ba2ed45eabd81c2f2e691aacd to VOTER
I20240502 21:27:55.250347 29257 consensus_queue.cc:237] T 0b3c514213bb40ca98955d6409a3b372 P 009d3e7e8a8b4e2497de3a73d01ced49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } attrs { replace: true } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: false } } peers { permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: false } }
I20240502 21:27:55.253206 26012 raft_consensus.cc:1261] T 0b3c514213bb40ca98955d6409a3b372 P fb6004b8640a4906aa0cfe63e383d40e [term 1 FOLLOWER]: Refusing update from remote peer 009d3e7e8a8b4e2497de3a73d01ced49: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:27:55.253193 24317 raft_consensus.cc:1261] T 0b3c514213bb40ca98955d6409a3b372 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 1 FOLLOWER]: Refusing update from remote peer 009d3e7e8a8b4e2497de3a73d01ced49: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:27:55.253453 27126 raft_consensus.cc:1261] T 0b3c514213bb40ca98955d6409a3b372 P 4f58800ba2ed45eabd81c2f2e691aacd [term 1 LEARNER]: Refusing update from remote peer 009d3e7e8a8b4e2497de3a73d01ced49: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:27:55.253746 29259 consensus_queue.cc:1035] T 0b3c514213bb40ca98955d6409a3b372 P 009d3e7e8a8b4e2497de3a73d01ced49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:27:55.254160 29257 consensus_queue.cc:1035] T 0b3c514213bb40ca98955d6409a3b372 P 009d3e7e8a8b4e2497de3a73d01ced49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:27:55.254411 27842 consensus_queue.cc:1035] T 0b3c514213bb40ca98955d6409a3b372 P 009d3e7e8a8b4e2497de3a73d01ced49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:27:55.267473 29101 raft_consensus.cc:1050] T 11cb9a2fbda946fb9f303f2c465bccb6 P 038c5d9d1d7a49c981c4c34fbc00d9e6: attempting to promote NON_VOTER a3e500d7077a48acb5edb33f7e178c81 to VOTER
I20240502 21:27:55.268082 29101 consensus_queue.cc:237] T 11cb9a2fbda946fb9f303f2c465bccb6 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } attrs { replace: true } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } }
I20240502 21:27:55.272315 27842 raft_consensus.cc:2932] T 0b3c514213bb40ca98955d6409a3b372 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 4f58800ba2ed45eabd81c2f2e691aacd (127.15.255.137) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } attrs { replace: true } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: false } } peers { permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: false } } }
I20240502 21:27:55.272856 26145 raft_consensus.cc:1261] T 11cb9a2fbda946fb9f303f2c465bccb6 P e117afbf8819424e8a98476453dfde2a [term 1 FOLLOWER]: Refusing update from remote peer 038c5d9d1d7a49c981c4c34fbc00d9e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:27:55.273980 26279 raft_consensus.cc:1261] T 11cb9a2fbda946fb9f303f2c465bccb6 P a3e500d7077a48acb5edb33f7e178c81 [term 1 LEARNER]: Refusing update from remote peer 038c5d9d1d7a49c981c4c34fbc00d9e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:27:55.274835 24317 raft_consensus.cc:2932] T 0b3c514213bb40ca98955d6409a3b372 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 4f58800ba2ed45eabd81c2f2e691aacd (127.15.255.137) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } attrs { replace: true } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: false } } peers { permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: false } } }
I20240502 21:27:55.275499 28460 consensus_queue.cc:1035] T 11cb9a2fbda946fb9f303f2c465bccb6 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:27:55.276926 27126 raft_consensus.cc:2932] T 0b3c514213bb40ca98955d6409a3b372 P 4f58800ba2ed45eabd81c2f2e691aacd [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 4f58800ba2ed45eabd81c2f2e691aacd (127.15.255.137) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } attrs { replace: true } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: false } } peers { permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: false } } }
I20240502 21:27:55.278333 23771 catalog_manager.cc:5484] T 0b3c514213bb40ca98955d6409a3b372 P 009d3e7e8a8b4e2497de3a73d01ced49 reported cstate change: config changed from index 7 to 8, 4f58800ba2ed45eabd81c2f2e691aacd (127.15.255.137) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:27:55.281479 26012 raft_consensus.cc:2932] T 0b3c514213bb40ca98955d6409a3b372 P fb6004b8640a4906aa0cfe63e383d40e [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 4f58800ba2ed45eabd81c2f2e691aacd (127.15.255.137) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } attrs { replace: true } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: false } } peers { permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: false } } }
I20240502 21:27:55.288790 29265 consensus_queue.cc:1035] T 11cb9a2fbda946fb9f303f2c465bccb6 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:27:55.293315 24183 raft_consensus.cc:1261] T 11cb9a2fbda946fb9f303f2c465bccb6 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 1 FOLLOWER]: Refusing update from remote peer 038c5d9d1d7a49c981c4c34fbc00d9e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:27:55.294698 29237 raft_consensus.cc:1050] T a274546743354e79bffa26ae7000e895 P 6c194a6db6c145acafc0f3068b8ce1fd: attempting to promote NON_VOTER 4f58800ba2ed45eabd81c2f2e691aacd to VOTER
I20240502 21:27:55.295502 29237 consensus_queue.cc:237] T a274546743354e79bffa26ae7000e895 P 6c194a6db6c145acafc0f3068b8ce1fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } attrs { replace: true } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } } peers { permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: false } }
I20240502 21:27:55.300707 29265 consensus_queue.cc:1035] T 11cb9a2fbda946fb9f303f2c465bccb6 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:27:55.305653 27126 raft_consensus.cc:1261] T a274546743354e79bffa26ae7000e895 P 4f58800ba2ed45eabd81c2f2e691aacd [term 1 LEARNER]: Refusing update from remote peer 6c194a6db6c145acafc0f3068b8ce1fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:27:55.305774 24184 raft_consensus.cc:1261] T a274546743354e79bffa26ae7000e895 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 1 FOLLOWER]: Refusing update from remote peer 6c194a6db6c145acafc0f3068b8ce1fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:27:55.306360 26147 raft_consensus.cc:1261] T a274546743354e79bffa26ae7000e895 P e117afbf8819424e8a98476453dfde2a [term 1 FOLLOWER]: Refusing update from remote peer 6c194a6db6c145acafc0f3068b8ce1fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:27:55.306389 29237 consensus_queue.cc:1035] T a274546743354e79bffa26ae7000e895 P 6c194a6db6c145acafc0f3068b8ce1fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:27:55.307013 29236 consensus_queue.cc:1035] T a274546743354e79bffa26ae7000e895 P 6c194a6db6c145acafc0f3068b8ce1fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:27:55.307334 29266 raft_consensus.cc:2932] T 11cb9a2fbda946fb9f303f2c465bccb6 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, a3e500d7077a48acb5edb33f7e178c81 (127.15.255.136) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } attrs { replace: true } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } } }
I20240502 21:27:55.307802 24184 raft_consensus.cc:2932] T 11cb9a2fbda946fb9f303f2c465bccb6 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, a3e500d7077a48acb5edb33f7e178c81 (127.15.255.136) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } attrs { replace: true } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } } }
I20240502 21:27:55.309206 29237 consensus_queue.cc:1035] T a274546743354e79bffa26ae7000e895 P 6c194a6db6c145acafc0f3068b8ce1fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:27:55.315047 23766 catalog_manager.cc:5484] T 11cb9a2fbda946fb9f303f2c465bccb6 P 009d3e7e8a8b4e2497de3a73d01ced49 reported cstate change: config changed from index 7 to 8, a3e500d7077a48acb5edb33f7e178c81 (127.15.255.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } attrs { replace: true } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } } }
I20240502 21:27:55.318841 29241 raft_consensus.cc:1050] T c9b5be01b5e941b1848df79c7a9db348 P 6c194a6db6c145acafc0f3068b8ce1fd: attempting to promote NON_VOTER 4f58800ba2ed45eabd81c2f2e691aacd to VOTER
I20240502 21:27:55.319640 29241 consensus_queue.cc:237] T c9b5be01b5e941b1848df79c7a9db348 P 6c194a6db6c145acafc0f3068b8ce1fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } attrs { replace: true } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } } peers { permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: false } }
I20240502 21:27:55.324892 26279 raft_consensus.cc:2932] T 11cb9a2fbda946fb9f303f2c465bccb6 P a3e500d7077a48acb5edb33f7e178c81 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, a3e500d7077a48acb5edb33f7e178c81 (127.15.255.136) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } attrs { replace: true } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } } }
I20240502 21:27:55.325690 29236 raft_consensus.cc:2932] T a274546743354e79bffa26ae7000e895 P 6c194a6db6c145acafc0f3068b8ce1fd [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 4f58800ba2ed45eabd81c2f2e691aacd (127.15.255.137) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } attrs { replace: true } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } } peers { permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: false } } }
I20240502 21:27:55.328163 26145 raft_consensus.cc:2932] T 11cb9a2fbda946fb9f303f2c465bccb6 P e117afbf8819424e8a98476453dfde2a [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, a3e500d7077a48acb5edb33f7e178c81 (127.15.255.136) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } attrs { replace: true } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } } }
I20240502 21:27:55.329789 27126 raft_consensus.cc:2932] T a274546743354e79bffa26ae7000e895 P 4f58800ba2ed45eabd81c2f2e691aacd [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 4f58800ba2ed45eabd81c2f2e691aacd (127.15.255.137) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } attrs { replace: true } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } } peers { permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: false } } }
I20240502 21:27:55.332289 24317 consensus_queue.cc:237] T 11cb9a2fbda946fb9f303f2c465bccb6 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } }
I20240502 21:27:55.332080 24184 raft_consensus.cc:2932] T a274546743354e79bffa26ae7000e895 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 4f58800ba2ed45eabd81c2f2e691aacd (127.15.255.137) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } attrs { replace: true } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } } peers { permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: false } } }
I20240502 21:27:55.334383 26279 raft_consensus.cc:1261] T 11cb9a2fbda946fb9f303f2c465bccb6 P a3e500d7077a48acb5edb33f7e178c81 [term 1 FOLLOWER]: Refusing update from remote peer 038c5d9d1d7a49c981c4c34fbc00d9e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:27:55.335709 23766 catalog_manager.cc:5484] T a274546743354e79bffa26ae7000e895 P 4f58800ba2ed45eabd81c2f2e691aacd reported cstate change: config changed from index 7 to 8, 4f58800ba2ed45eabd81c2f2e691aacd (127.15.255.137) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } attrs { replace: true } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } } peers { permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: false } } }
I20240502 21:27:55.336355 29266 consensus_queue.cc:1035] T 11cb9a2fbda946fb9f303f2c465bccb6 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:27:55.337815 26147 raft_consensus.cc:1261] T 11cb9a2fbda946fb9f303f2c465bccb6 P e117afbf8819424e8a98476453dfde2a [term 1 FOLLOWER]: Refusing update from remote peer 038c5d9d1d7a49c981c4c34fbc00d9e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:27:55.338428 29266 consensus_queue.cc:1035] T 11cb9a2fbda946fb9f303f2c465bccb6 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:27:55.340234 26145 raft_consensus.cc:2932] T a274546743354e79bffa26ae7000e895 P e117afbf8819424e8a98476453dfde2a [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 4f58800ba2ed45eabd81c2f2e691aacd (127.15.255.137) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } attrs { replace: true } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } } peers { permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: false } } }
I20240502 21:27:55.340715 29265 raft_consensus.cc:2932] T 11cb9a2fbda946fb9f303f2c465bccb6 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 009d3e7e8a8b4e2497de3a73d01ced49 (127.15.255.131) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } } }
I20240502 21:27:55.344929 27234 raft_consensus.cc:1050] T 30236241b670443489240f191d9b9d50 P 6810a866decb4f5089f7d35c870e4204: attempting to promote NON_VOTER a3e500d7077a48acb5edb33f7e178c81 to VOTER
I20240502 21:27:55.345741 27234 consensus_queue.cc:237] T 30236241b670443489240f191d9b9d50 P 6810a866decb4f5089f7d35c870e4204 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } attrs { replace: true } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: false } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } }
I20240502 21:27:55.346707 26280 raft_consensus.cc:2932] T 11cb9a2fbda946fb9f303f2c465bccb6 P a3e500d7077a48acb5edb33f7e178c81 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 009d3e7e8a8b4e2497de3a73d01ced49 (127.15.255.131) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } } }
I20240502 21:27:55.347980 27126 raft_consensus.cc:1261] T c9b5be01b5e941b1848df79c7a9db348 P 4f58800ba2ed45eabd81c2f2e691aacd [term 1 LEARNER]: Refusing update from remote peer 6c194a6db6c145acafc0f3068b8ce1fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:27:55.348140 23755 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 11cb9a2fbda946fb9f303f2c465bccb6 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:27:55.349069 29236 consensus_queue.cc:1035] T c9b5be01b5e941b1848df79c7a9db348 P 6c194a6db6c145acafc0f3068b8ce1fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:27:55.351505 23916 raft_consensus.cc:1261] T c9b5be01b5e941b1848df79c7a9db348 P 6810a866decb4f5089f7d35c870e4204 [term 1 FOLLOWER]: Refusing update from remote peer 6c194a6db6c145acafc0f3068b8ce1fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:27:55.351617 26146 raft_consensus.cc:1261] T c9b5be01b5e941b1848df79c7a9db348 P e117afbf8819424e8a98476453dfde2a [term 1 FOLLOWER]: Refusing update from remote peer 6c194a6db6c145acafc0f3068b8ce1fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:27:55.352093 29237 consensus_queue.cc:1035] T c9b5be01b5e941b1848df79c7a9db348 P 6c194a6db6c145acafc0f3068b8ce1fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:27:55.352963 23765 catalog_manager.cc:5484] T 11cb9a2fbda946fb9f303f2c465bccb6 P 038c5d9d1d7a49c981c4c34fbc00d9e6 reported cstate change: config changed from index 8 to 9, VOTER 009d3e7e8a8b4e2497de3a73d01ced49 (127.15.255.131) evicted. New cstate: current_term: 1 leader_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:27:55.353539 26146 raft_consensus.cc:2932] T 11cb9a2fbda946fb9f303f2c465bccb6 P e117afbf8819424e8a98476453dfde2a [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 009d3e7e8a8b4e2497de3a73d01ced49 (127.15.255.131) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } } }
I20240502 21:27:55.358049 24047 consensus_queue.cc:237] T a274546743354e79bffa26ae7000e895 P 6c194a6db6c145acafc0f3068b8ce1fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } } peers { permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: false } }
I20240502 21:27:55.360339 24162 tablet_service.cc:1514] Processing DeleteTablet for tablet 11cb9a2fbda946fb9f303f2c465bccb6 with delete_type TABLET_DATA_TOMBSTONED (TS 009d3e7e8a8b4e2497de3a73d01ced49 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:33102
I20240502 21:27:55.362607 29281 tablet_replica.cc:330] T 11cb9a2fbda946fb9f303f2c465bccb6 P 009d3e7e8a8b4e2497de3a73d01ced49: stopping tablet replica
I20240502 21:27:55.362720 29236 consensus_queue.cc:1035] T c9b5be01b5e941b1848df79c7a9db348 P 6c194a6db6c145acafc0f3068b8ce1fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:27:55.362962 29281 raft_consensus.cc:2229] T 11cb9a2fbda946fb9f303f2c465bccb6 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:55.363374 29281 raft_consensus.cc:2258] T 11cb9a2fbda946fb9f303f2c465bccb6 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:55.364053 27126 raft_consensus.cc:1261] T a274546743354e79bffa26ae7000e895 P 4f58800ba2ed45eabd81c2f2e691aacd [term 1 FOLLOWER]: Refusing update from remote peer 6c194a6db6c145acafc0f3068b8ce1fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:27:55.364688 29236 consensus_queue.cc:1035] T a274546743354e79bffa26ae7000e895 P 6c194a6db6c145acafc0f3068b8ce1fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:27:55.367300 28941 raft_consensus.cc:2932] T a274546743354e79bffa26ae7000e895 P 6c194a6db6c145acafc0f3068b8ce1fd [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 009d3e7e8a8b4e2497de3a73d01ced49 (127.15.255.131) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } } peers { permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: false } } }
I20240502 21:27:55.367933 27125 raft_consensus.cc:2932] T a274546743354e79bffa26ae7000e895 P 4f58800ba2ed45eabd81c2f2e691aacd [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 009d3e7e8a8b4e2497de3a73d01ced49 (127.15.255.131) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } } peers { permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: false } } }
I20240502 21:27:55.369074 26145 raft_consensus.cc:1261] T a274546743354e79bffa26ae7000e895 P e117afbf8819424e8a98476453dfde2a [term 1 FOLLOWER]: Refusing update from remote peer 6c194a6db6c145acafc0f3068b8ce1fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:27:55.371640 28941 consensus_queue.cc:1035] T a274546743354e79bffa26ae7000e895 P 6c194a6db6c145acafc0f3068b8ce1fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:27:55.371815 23757 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet a274546743354e79bffa26ae7000e895 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:27:55.373947 26279 raft_consensus.cc:1261] T 30236241b670443489240f191d9b9d50 P a3e500d7077a48acb5edb33f7e178c81 [term 1 LEARNER]: Refusing update from remote peer 6810a866decb4f5089f7d35c870e4204: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:27:55.375154 29095 consensus_queue.cc:1035] T 30236241b670443489240f191d9b9d50 P 6810a866decb4f5089f7d35c870e4204 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:27:55.376399 26012 raft_consensus.cc:1261] T 30236241b670443489240f191d9b9d50 P fb6004b8640a4906aa0cfe63e383d40e [term 1 FOLLOWER]: Refusing update from remote peer 6810a866decb4f5089f7d35c870e4204: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:27:55.378196 29095 consensus_queue.cc:1035] T 30236241b670443489240f191d9b9d50 P 6810a866decb4f5089f7d35c870e4204 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:27:55.378234 27127 raft_consensus.cc:2932] T c9b5be01b5e941b1848df79c7a9db348 P 4f58800ba2ed45eabd81c2f2e691aacd [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 4f58800ba2ed45eabd81c2f2e691aacd (127.15.255.137) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } attrs { replace: true } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } } peers { permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: false } } }
I20240502 21:27:55.381764 29236 raft_consensus.cc:2932] T c9b5be01b5e941b1848df79c7a9db348 P 6c194a6db6c145acafc0f3068b8ce1fd [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 4f58800ba2ed45eabd81c2f2e691aacd (127.15.255.137) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } attrs { replace: true } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } } peers { permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: false } } }
I20240502 21:27:55.386562 26145 raft_consensus.cc:2932] T a274546743354e79bffa26ae7000e895 P e117afbf8819424e8a98476453dfde2a [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 009d3e7e8a8b4e2497de3a73d01ced49 (127.15.255.131) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } } peers { permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: false } } }
I20240502 21:27:55.388074 29281 ts_tablet_manager.cc:1875] T 11cb9a2fbda946fb9f303f2c465bccb6 P 009d3e7e8a8b4e2497de3a73d01ced49: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:27:55.388554 23915 raft_consensus.cc:2932] T c9b5be01b5e941b1848df79c7a9db348 P 6810a866decb4f5089f7d35c870e4204 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 4f58800ba2ed45eabd81c2f2e691aacd (127.15.255.137) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } attrs { replace: true } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } } peers { permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: false } } }
I20240502 21:27:55.390250 26147 raft_consensus.cc:2932] T c9b5be01b5e941b1848df79c7a9db348 P e117afbf8819424e8a98476453dfde2a [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 4f58800ba2ed45eabd81c2f2e691aacd (127.15.255.137) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } attrs { replace: true } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } } peers { permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: false } } }
I20240502 21:27:55.391268 23765 catalog_manager.cc:5484] T a274546743354e79bffa26ae7000e895 P 6c194a6db6c145acafc0f3068b8ce1fd reported cstate change: config changed from index 8 to 9, VOTER 009d3e7e8a8b4e2497de3a73d01ced49 (127.15.255.131) evicted. New cstate: current_term: 1 leader_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:27:55.394094 29281 ts_tablet_manager.cc:1888] T 11cb9a2fbda946fb9f303f2c465bccb6 P 009d3e7e8a8b4e2497de3a73d01ced49: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240502 21:27:55.394208 29281 log.cc:1196] T 11cb9a2fbda946fb9f303f2c465bccb6 P 009d3e7e8a8b4e2497de3a73d01ced49: Deleting WAL directory at /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-2/wal/wals/11cb9a2fbda946fb9f303f2c465bccb6
I20240502 21:27:55.395355 29095 raft_consensus.cc:2932] T 30236241b670443489240f191d9b9d50 P 6810a866decb4f5089f7d35c870e4204 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, a3e500d7077a48acb5edb33f7e178c81 (127.15.255.136) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } attrs { replace: true } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: false } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } } }
I20240502 21:27:55.396308 23771 catalog_manager.cc:5484] T c9b5be01b5e941b1848df79c7a9db348 P e117afbf8819424e8a98476453dfde2a reported cstate change: config changed from index 7 to 8, 4f58800ba2ed45eabd81c2f2e691aacd (127.15.255.137) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } attrs { replace: true } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } } peers { permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: false } } }
I20240502 21:27:55.396674 24317 raft_consensus.cc:1261] T 30236241b670443489240f191d9b9d50 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 1 FOLLOWER]: Refusing update from remote peer 6810a866decb4f5089f7d35c870e4204: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:27:55.398550 28790 consensus_queue.cc:1035] T 30236241b670443489240f191d9b9d50 P 6810a866decb4f5089f7d35c870e4204 [LEADER]: Connected to new peer: Peer: permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:27:55.400027 29263 raft_consensus.cc:1050] T fe044374ad7b4999afc72ab812466749 P 038c5d9d1d7a49c981c4c34fbc00d9e6: attempting to promote NON_VOTER fb6004b8640a4906aa0cfe63e383d40e to VOTER
I20240502 21:27:55.400333 26279 raft_consensus.cc:2932] T 30236241b670443489240f191d9b9d50 P a3e500d7077a48acb5edb33f7e178c81 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, a3e500d7077a48acb5edb33f7e178c81 (127.15.255.136) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } attrs { replace: true } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: false } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } } }
I20240502 21:27:55.400859 29263 consensus_queue.cc:237] T fe044374ad7b4999afc72ab812466749 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } attrs { replace: true } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: false } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: false } }
I20240502 21:27:55.404367 23915 raft_consensus.cc:1261] T fe044374ad7b4999afc72ab812466749 P 6810a866decb4f5089f7d35c870e4204 [term 1 FOLLOWER]: Refusing update from remote peer 038c5d9d1d7a49c981c4c34fbc00d9e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:27:55.404677 24317 raft_consensus.cc:2932] T 30236241b670443489240f191d9b9d50 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, a3e500d7077a48acb5edb33f7e178c81 (127.15.255.136) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } attrs { replace: true } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: false } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } } }
I20240502 21:27:55.405114 29266 consensus_queue.cc:1035] T fe044374ad7b4999afc72ab812466749 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:27:55.406800 24162 tablet_service.cc:1514] Processing DeleteTablet for tablet a274546743354e79bffa26ae7000e895 with delete_type TABLET_DATA_TOMBSTONED (TS 009d3e7e8a8b4e2497de3a73d01ced49 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:33102
I20240502 21:27:55.407670 26012 raft_consensus.cc:1261] T fe044374ad7b4999afc72ab812466749 P fb6004b8640a4906aa0cfe63e383d40e [term 1 LEARNER]: Refusing update from remote peer 038c5d9d1d7a49c981c4c34fbc00d9e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:27:55.407737 26011 raft_consensus.cc:2932] T 30236241b670443489240f191d9b9d50 P fb6004b8640a4906aa0cfe63e383d40e [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, a3e500d7077a48acb5edb33f7e178c81 (127.15.255.136) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } attrs { replace: true } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: false } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } } }
I20240502 21:27:55.408532 25878 raft_consensus.cc:1261] T fe044374ad7b4999afc72ab812466749 P ee27c3f9e0ec46caa742ac386d6b7c20 [term 1 FOLLOWER]: Refusing update from remote peer 038c5d9d1d7a49c981c4c34fbc00d9e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:27:55.409659 29265 consensus_queue.cc:1035] T fe044374ad7b4999afc72ab812466749 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:27:55.410161 29265 consensus_queue.cc:1035] T fe044374ad7b4999afc72ab812466749 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:27:55.413607 23768 catalog_manager.cc:5484] T 30236241b670443489240f191d9b9d50 P fb6004b8640a4906aa0cfe63e383d40e reported cstate change: config changed from index 7 to 8, a3e500d7077a48acb5edb33f7e178c81 (127.15.255.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "6810a866decb4f5089f7d35c870e4204" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } attrs { replace: true } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: false } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } } }
I20240502 21:27:55.420218 29281 tablet_replica.cc:330] T a274546743354e79bffa26ae7000e895 P 009d3e7e8a8b4e2497de3a73d01ced49: stopping tablet replica
I20240502 21:27:55.420486 29281 raft_consensus.cc:2229] T a274546743354e79bffa26ae7000e895 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:55.421721 29281 raft_consensus.cc:2258] T a274546743354e79bffa26ae7000e895 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:55.421861 23756 catalog_manager.cc:4830] TS 009d3e7e8a8b4e2497de3a73d01ced49 (127.15.255.131:37895): tablet 11cb9a2fbda946fb9f303f2c465bccb6 (table rebalance_test_table_5 [id=e10e5576c0ea45a09c0eab75c2023b78]) successfully deleted
I20240502 21:27:55.426189 24047 consensus_queue.cc:237] T c9b5be01b5e941b1848df79c7a9db348 P 6c194a6db6c145acafc0f3068b8ce1fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } } peers { permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: false } }
I20240502 21:27:55.428671 26145 raft_consensus.cc:1261] T c9b5be01b5e941b1848df79c7a9db348 P e117afbf8819424e8a98476453dfde2a [term 1 FOLLOWER]: Refusing update from remote peer 6c194a6db6c145acafc0f3068b8ce1fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:27:55.428897 27127 raft_consensus.cc:1261] T c9b5be01b5e941b1848df79c7a9db348 P 4f58800ba2ed45eabd81c2f2e691aacd [term 1 FOLLOWER]: Refusing update from remote peer 6c194a6db6c145acafc0f3068b8ce1fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:27:55.429365 29236 consensus_queue.cc:1035] T c9b5be01b5e941b1848df79c7a9db348 P 6c194a6db6c145acafc0f3068b8ce1fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:27:55.429121 29265 raft_consensus.cc:2932] T fe044374ad7b4999afc72ab812466749 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, fb6004b8640a4906aa0cfe63e383d40e (127.15.255.134) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } attrs { replace: true } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: false } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: false } } }
I20240502 21:27:55.429694 29241 consensus_queue.cc:1035] T c9b5be01b5e941b1848df79c7a9db348 P 6c194a6db6c145acafc0f3068b8ce1fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:27:55.430804 29281 ts_tablet_manager.cc:1875] T a274546743354e79bffa26ae7000e895 P 009d3e7e8a8b4e2497de3a73d01ced49: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:27:55.433743 29236 raft_consensus.cc:2932] T c9b5be01b5e941b1848df79c7a9db348 P 6c194a6db6c145acafc0f3068b8ce1fd [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 6810a866decb4f5089f7d35c870e4204 (127.15.255.129) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } } peers { permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: false } } }
I20240502 21:27:55.437081 26147 raft_consensus.cc:2932] T c9b5be01b5e941b1848df79c7a9db348 P e117afbf8819424e8a98476453dfde2a [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 6810a866decb4f5089f7d35c870e4204 (127.15.255.129) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } } peers { permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: false } } }
I20240502 21:27:55.439081 29281 ts_tablet_manager.cc:1888] T a274546743354e79bffa26ae7000e895 P 009d3e7e8a8b4e2497de3a73d01ced49: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240502 21:27:55.439249 29281 log.cc:1196] T a274546743354e79bffa26ae7000e895 P 009d3e7e8a8b4e2497de3a73d01ced49: Deleting WAL directory at /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-2/wal/wals/a274546743354e79bffa26ae7000e895
I20240502 21:27:55.439968 23756 catalog_manager.cc:4830] TS 009d3e7e8a8b4e2497de3a73d01ced49 (127.15.255.131:37895): tablet a274546743354e79bffa26ae7000e895 (table rebalance_test_table_3 [id=933ce90acb9f4b91ab6bd41e87a8fe99]) successfully deleted
I20240502 21:27:55.441779 26012 raft_consensus.cc:2932] T fe044374ad7b4999afc72ab812466749 P fb6004b8640a4906aa0cfe63e383d40e [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, fb6004b8640a4906aa0cfe63e383d40e (127.15.255.134) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } attrs { replace: true } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: false } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: false } } }
I20240502 21:27:55.442147 23770 catalog_manager.cc:5484] T c9b5be01b5e941b1848df79c7a9db348 P e117afbf8819424e8a98476453dfde2a reported cstate change: config changed from index 8 to 9, VOTER 6810a866decb4f5089f7d35c870e4204 (127.15.255.129) evicted. New cstate: current_term: 1 leader_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } } peers { permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: false } } }
I20240502 21:27:55.442467 23915 raft_consensus.cc:2932] T fe044374ad7b4999afc72ab812466749 P 6810a866decb4f5089f7d35c870e4204 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, fb6004b8640a4906aa0cfe63e383d40e (127.15.255.134) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } attrs { replace: true } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: false } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: false } } }
I20240502 21:27:55.446403 23765 catalog_manager.cc:5484] T fe044374ad7b4999afc72ab812466749 P 038c5d9d1d7a49c981c4c34fbc00d9e6 reported cstate change: config changed from index 7 to 8, fb6004b8640a4906aa0cfe63e383d40e (127.15.255.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:27:55.446846 27125 raft_consensus.cc:2932] T c9b5be01b5e941b1848df79c7a9db348 P 4f58800ba2ed45eabd81c2f2e691aacd [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 6810a866decb4f5089f7d35c870e4204 (127.15.255.129) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } } peers { permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: false } } }
I20240502 21:27:55.447463 23894 tablet_service.cc:1514] Processing DeleteTablet for tablet c9b5be01b5e941b1848df79c7a9db348 with delete_type TABLET_DATA_TOMBSTONED (TS 6810a866decb4f5089f7d35c870e4204 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:33512
I20240502 21:27:55.448458 29251 tablet_replica.cc:330] T c9b5be01b5e941b1848df79c7a9db348 P 6810a866decb4f5089f7d35c870e4204: stopping tablet replica
I20240502 21:27:55.448649 29251 raft_consensus.cc:2229] T c9b5be01b5e941b1848df79c7a9db348 P 6810a866decb4f5089f7d35c870e4204 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:55.448872 29251 raft_consensus.cc:2258] T c9b5be01b5e941b1848df79c7a9db348 P 6810a866decb4f5089f7d35c870e4204 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:55.449841 29251 ts_tablet_manager.cc:1875] T c9b5be01b5e941b1848df79c7a9db348 P 6810a866decb4f5089f7d35c870e4204: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:27:55.451530 23757 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c9b5be01b5e941b1848df79c7a9db348 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:27:55.453754 24316 consensus_queue.cc:237] T fe044374ad7b4999afc72ab812466749 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: false } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: false } }
I20240502 21:27:55.456501 25879 raft_consensus.cc:1261] T fe044374ad7b4999afc72ab812466749 P ee27c3f9e0ec46caa742ac386d6b7c20 [term 1 FOLLOWER]: Refusing update from remote peer 038c5d9d1d7a49c981c4c34fbc00d9e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:27:55.457176 29101 consensus_queue.cc:1035] T fe044374ad7b4999afc72ab812466749 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:27:55.457437 26012 raft_consensus.cc:1261] T fe044374ad7b4999afc72ab812466749 P fb6004b8640a4906aa0cfe63e383d40e [term 1 FOLLOWER]: Refusing update from remote peer 038c5d9d1d7a49c981c4c34fbc00d9e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:27:55.457912 29101 consensus_queue.cc:1035] T fe044374ad7b4999afc72ab812466749 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:27:55.458230 25879 raft_consensus.cc:2932] T fe044374ad7b4999afc72ab812466749 P ee27c3f9e0ec46caa742ac386d6b7c20 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, fb6004b8640a4906aa0cfe63e383d40e (127.15.255.134) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } attrs { replace: true } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: false } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: false } } }
I20240502 21:27:55.460297 29101 raft_consensus.cc:2932] T fe044374ad7b4999afc72ab812466749 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 6810a866decb4f5089f7d35c870e4204 (127.15.255.129) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: false } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: false } } }
I20240502 21:27:55.465560 29251 ts_tablet_manager.cc:1888] T c9b5be01b5e941b1848df79c7a9db348 P 6810a866decb4f5089f7d35c870e4204: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240502 21:27:55.465746 29251 log.cc:1196] T c9b5be01b5e941b1848df79c7a9db348 P 6810a866decb4f5089f7d35c870e4204: Deleting WAL directory at /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-0/wal/wals/c9b5be01b5e941b1848df79c7a9db348
I20240502 21:27:55.466004 26011 raft_consensus.cc:2932] T fe044374ad7b4999afc72ab812466749 P fb6004b8640a4906aa0cfe63e383d40e [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 6810a866decb4f5089f7d35c870e4204 (127.15.255.129) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: false } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: false } } }
I20240502 21:27:55.466563 23757 catalog_manager.cc:4830] TS 6810a866decb4f5089f7d35c870e4204 (127.15.255.129:40435): tablet c9b5be01b5e941b1848df79c7a9db348 (table rebalance_test_table_2 [id=85ca987b38d84a7db5119dfc88b4feaf]) successfully deleted
I20240502 21:27:55.470568 23755 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet fe044374ad7b4999afc72ab812466749 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:27:55.471232 23766 catalog_manager.cc:5484] T fe044374ad7b4999afc72ab812466749 P fb6004b8640a4906aa0cfe63e383d40e reported cstate change: config changed from index 8 to 9, VOTER 6810a866decb4f5089f7d35c870e4204 (127.15.255.129) evicted. New cstate: current_term: 1 leader_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: false } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: false } } }
I20240502 21:27:55.473521 25878 raft_consensus.cc:2932] T fe044374ad7b4999afc72ab812466749 P ee27c3f9e0ec46caa742ac386d6b7c20 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 6810a866decb4f5089f7d35c870e4204 (127.15.255.129) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: false } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: false } } }
I20240502 21:27:55.474319 23894 tablet_service.cc:1514] Processing DeleteTablet for tablet fe044374ad7b4999afc72ab812466749 with delete_type TABLET_DATA_TOMBSTONED (TS 6810a866decb4f5089f7d35c870e4204 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:33512
I20240502 21:27:55.475129 29251 tablet_replica.cc:330] T fe044374ad7b4999afc72ab812466749 P 6810a866decb4f5089f7d35c870e4204: stopping tablet replica
I20240502 21:27:55.475406 29251 raft_consensus.cc:2229] T fe044374ad7b4999afc72ab812466749 P 6810a866decb4f5089f7d35c870e4204 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:55.475705 29251 raft_consensus.cc:2258] T fe044374ad7b4999afc72ab812466749 P 6810a866decb4f5089f7d35c870e4204 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:55.476917 29251 ts_tablet_manager.cc:1875] T fe044374ad7b4999afc72ab812466749 P 6810a866decb4f5089f7d35c870e4204: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:27:55.480100 29257 raft_consensus.cc:1050] T 85e77d5faa8346cd8a99dcbcdf75f250 P 009d3e7e8a8b4e2497de3a73d01ced49: attempting to promote NON_VOTER ee27c3f9e0ec46caa742ac386d6b7c20 to VOTER
I20240502 21:27:55.480101 28493 raft_consensus.cc:1050] T cca45c89b9fe40e7800d95c1598a9e52 P 009d3e7e8a8b4e2497de3a73d01ced49: attempting to promote NON_VOTER a3e500d7077a48acb5edb33f7e178c81 to VOTER
I20240502 21:27:55.481040 29257 consensus_queue.cc:237] T 85e77d5faa8346cd8a99dcbcdf75f250 P 009d3e7e8a8b4e2497de3a73d01ced49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } attrs { replace: true } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: false } }
I20240502 21:27:55.481384 28493 consensus_queue.cc:237] T cca45c89b9fe40e7800d95c1598a9e52 P 009d3e7e8a8b4e2497de3a73d01ced49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } attrs { replace: true } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: false } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } }
I20240502 21:27:55.483754 26279 raft_consensus.cc:1261] T cca45c89b9fe40e7800d95c1598a9e52 P a3e500d7077a48acb5edb33f7e178c81 [term 1 LEARNER]: Refusing update from remote peer 009d3e7e8a8b4e2497de3a73d01ced49: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:27:55.484460 24047 raft_consensus.cc:1261] T cca45c89b9fe40e7800d95c1598a9e52 P 6c194a6db6c145acafc0f3068b8ce1fd [term 1 FOLLOWER]: Refusing update from remote peer 009d3e7e8a8b4e2497de3a73d01ced49: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:27:55.484400 29259 consensus_queue.cc:1035] T cca45c89b9fe40e7800d95c1598a9e52 P 009d3e7e8a8b4e2497de3a73d01ced49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:27:55.485522 25879 raft_consensus.cc:1261] T cca45c89b9fe40e7800d95c1598a9e52 P ee27c3f9e0ec46caa742ac386d6b7c20 [term 1 FOLLOWER]: Refusing update from remote peer 009d3e7e8a8b4e2497de3a73d01ced49: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:27:55.486028 27842 consensus_queue.cc:1035] T cca45c89b9fe40e7800d95c1598a9e52 P 009d3e7e8a8b4e2497de3a73d01ced49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:27:55.486308 29259 consensus_queue.cc:1035] T cca45c89b9fe40e7800d95c1598a9e52 P 009d3e7e8a8b4e2497de3a73d01ced49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:27:55.507063 29251 ts_tablet_manager.cc:1888] T fe044374ad7b4999afc72ab812466749 P 6810a866decb4f5089f7d35c870e4204: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240502 21:27:55.507208 29251 log.cc:1196] T fe044374ad7b4999afc72ab812466749 P 6810a866decb4f5089f7d35c870e4204: Deleting WAL directory at /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-0/wal/wals/fe044374ad7b4999afc72ab812466749
I20240502 21:27:55.507875 23757 catalog_manager.cc:4830] TS 6810a866decb4f5089f7d35c870e4204 (127.15.255.129:40435): tablet fe044374ad7b4999afc72ab812466749 (table rebalance_test_table_6 [id=9aaf56e5a31f42a79b45624cc2c43779]) successfully deleted
I20240502 21:27:55.509267 29299 raft_consensus.cc:1050] T f84bbe304597450c978ae28e0c4e7a3e P a3e500d7077a48acb5edb33f7e178c81: attempting to promote NON_VOTER fb6004b8640a4906aa0cfe63e383d40e to VOTER
I20240502 21:27:55.510174 29299 consensus_queue.cc:237] T f84bbe304597450c978ae28e0c4e7a3e P a3e500d7077a48acb5edb33f7e178c81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } attrs { replace: true } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: false } }
I20240502 21:27:55.514029 26012 raft_consensus.cc:1261] T f84bbe304597450c978ae28e0c4e7a3e P fb6004b8640a4906aa0cfe63e383d40e [term 2 LEARNER]: Refusing update from remote peer a3e500d7077a48acb5edb33f7e178c81: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 21:27:55.514684 29292 consensus_queue.cc:1035] T f84bbe304597450c978ae28e0c4e7a3e P a3e500d7077a48acb5edb33f7e178c81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:27:55.516544 24047 raft_consensus.cc:1261] T f84bbe304597450c978ae28e0c4e7a3e P 6c194a6db6c145acafc0f3068b8ce1fd [term 2 FOLLOWER]: Refusing update from remote peer a3e500d7077a48acb5edb33f7e178c81: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 21:27:55.517177 28935 consensus_queue.cc:1035] T f84bbe304597450c978ae28e0c4e7a3e P a3e500d7077a48acb5edb33f7e178c81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:27:55.517781 29266 raft_consensus.cc:1050] T 52f621ea3f174490948eae854c706a92 P 038c5d9d1d7a49c981c4c34fbc00d9e6: attempting to promote NON_VOTER 4f58800ba2ed45eabd81c2f2e691aacd to VOTER
I20240502 21:27:55.518095 27842 raft_consensus.cc:2932] T cca45c89b9fe40e7800d95c1598a9e52 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, a3e500d7077a48acb5edb33f7e178c81 (127.15.255.136) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } attrs { replace: true } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: false } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } } }
I20240502 21:27:55.518679 29266 consensus_queue.cc:237] T 52f621ea3f174490948eae854c706a92 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } attrs { replace: true } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: false } } peers { permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: false } }
I20240502 21:27:55.525313 24046 raft_consensus.cc:2932] T cca45c89b9fe40e7800d95c1598a9e52 P 6c194a6db6c145acafc0f3068b8ce1fd [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, a3e500d7077a48acb5edb33f7e178c81 (127.15.255.136) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } attrs { replace: true } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: false } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } } }
I20240502 21:27:55.525961 26280 raft_consensus.cc:2932] T cca45c89b9fe40e7800d95c1598a9e52 P a3e500d7077a48acb5edb33f7e178c81 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, a3e500d7077a48acb5edb33f7e178c81 (127.15.255.136) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } attrs { replace: true } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: false } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } } }
I20240502 21:27:55.530138 24316 raft_consensus.cc:1261] T f84bbe304597450c978ae28e0c4e7a3e P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 2 FOLLOWER]: Refusing update from remote peer a3e500d7077a48acb5edb33f7e178c81: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 21:27:55.534217 29292 consensus_queue.cc:1035] T f84bbe304597450c978ae28e0c4e7a3e P a3e500d7077a48acb5edb33f7e178c81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:27:55.535153 23766 catalog_manager.cc:5484] T cca45c89b9fe40e7800d95c1598a9e52 P 6c194a6db6c145acafc0f3068b8ce1fd reported cstate change: config changed from index 7 to 8, a3e500d7077a48acb5edb33f7e178c81 (127.15.255.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } attrs { replace: true } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: false } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } } }
I20240502 21:27:55.542711 25878 raft_consensus.cc:2932] T cca45c89b9fe40e7800d95c1598a9e52 P ee27c3f9e0ec46caa742ac386d6b7c20 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, a3e500d7077a48acb5edb33f7e178c81 (127.15.255.136) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } attrs { replace: true } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: false } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } } }
I20240502 21:27:55.550341 25878 raft_consensus.cc:1261] T 85e77d5faa8346cd8a99dcbcdf75f250 P ee27c3f9e0ec46caa742ac386d6b7c20 [term 1 LEARNER]: Refusing update from remote peer 009d3e7e8a8b4e2497de3a73d01ced49: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:27:55.551193 29259 consensus_queue.cc:1035] T 85e77d5faa8346cd8a99dcbcdf75f250 P 009d3e7e8a8b4e2497de3a73d01ced49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:27:55.552544 29292 raft_consensus.cc:2932] T f84bbe304597450c978ae28e0c4e7a3e P a3e500d7077a48acb5edb33f7e178c81 [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index 8 to 9, fb6004b8640a4906aa0cfe63e383d40e (127.15.255.134) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } attrs { replace: true } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: false } } }
I20240502 21:27:55.556737 28790 raft_consensus.cc:1050] T 4eebf4508766421eb6c547a0fbbc28b1 P 6810a866decb4f5089f7d35c870e4204: attempting to promote NON_VOTER a3e500d7077a48acb5edb33f7e178c81 to VOTER
I20240502 21:27:55.557106 27125 raft_consensus.cc:1261] T 52f621ea3f174490948eae854c706a92 P 4f58800ba2ed45eabd81c2f2e691aacd [term 1 LEARNER]: Refusing update from remote peer 038c5d9d1d7a49c981c4c34fbc00d9e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:27:55.557621 28790 consensus_queue.cc:237] T 4eebf4508766421eb6c547a0fbbc28b1 P 6810a866decb4f5089f7d35c870e4204 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } attrs { replace: true } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } }
I20240502 21:27:55.557716 29263 consensus_queue.cc:1035] T 52f621ea3f174490948eae854c706a92 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:27:55.559888 26011 raft_consensus.cc:2932] T f84bbe304597450c978ae28e0c4e7a3e P fb6004b8640a4906aa0cfe63e383d40e [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, fb6004b8640a4906aa0cfe63e383d40e (127.15.255.134) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } attrs { replace: true } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: false } } }
I20240502 21:27:55.567166 24183 consensus_queue.cc:237] T cca45c89b9fe40e7800d95c1598a9e52 P 009d3e7e8a8b4e2497de3a73d01ced49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: false } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } }
I20240502 21:27:55.567935 25877 raft_consensus.cc:1261] T 52f621ea3f174490948eae854c706a92 P ee27c3f9e0ec46caa742ac386d6b7c20 [term 1 FOLLOWER]: Refusing update from remote peer 038c5d9d1d7a49c981c4c34fbc00d9e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:27:55.568101 24047 raft_consensus.cc:2932] T f84bbe304597450c978ae28e0c4e7a3e P 6c194a6db6c145acafc0f3068b8ce1fd [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, fb6004b8640a4906aa0cfe63e383d40e (127.15.255.134) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } attrs { replace: true } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: false } } }
I20240502 21:27:55.568550 29263 consensus_queue.cc:1035] T 52f621ea3f174490948eae854c706a92 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:27:55.570636 25876 raft_consensus.cc:1261] T cca45c89b9fe40e7800d95c1598a9e52 P ee27c3f9e0ec46caa742ac386d6b7c20 [term 1 FOLLOWER]: Refusing update from remote peer 009d3e7e8a8b4e2497de3a73d01ced49: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:27:55.571187 29308 consensus_queue.cc:1035] T cca45c89b9fe40e7800d95c1598a9e52 P 009d3e7e8a8b4e2497de3a73d01ced49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:27:55.577849 23770 catalog_manager.cc:5484] T f84bbe304597450c978ae28e0c4e7a3e P 6c194a6db6c145acafc0f3068b8ce1fd reported cstate change: config changed from index 8 to 9, fb6004b8640a4906aa0cfe63e383d40e (127.15.255.134) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "a3e500d7077a48acb5edb33f7e178c81" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } attrs { replace: true } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: false } } }
I20240502 21:27:55.586213 26279 consensus_queue.cc:237] T f84bbe304597450c978ae28e0c4e7a3e P a3e500d7077a48acb5edb33f7e178c81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: false } }
I20240502 21:27:55.588811 23915 raft_consensus.cc:1261] T 52f621ea3f174490948eae854c706a92 P 6810a866decb4f5089f7d35c870e4204 [term 1 FOLLOWER]: Refusing update from remote peer 038c5d9d1d7a49c981c4c34fbc00d9e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:27:55.589497 29266 consensus_queue.cc:1035] T 52f621ea3f174490948eae854c706a92 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:27:55.590754 26012 raft_consensus.cc:1261] T f84bbe304597450c978ae28e0c4e7a3e P fb6004b8640a4906aa0cfe63e383d40e [term 2 FOLLOWER]: Refusing update from remote peer a3e500d7077a48acb5edb33f7e178c81: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 21:27:55.591356 29301 consensus_queue.cc:1035] T f84bbe304597450c978ae28e0c4e7a3e P a3e500d7077a48acb5edb33f7e178c81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 21:27:55.593204 29308 raft_consensus.cc:2932] T cca45c89b9fe40e7800d95c1598a9e52 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 6c194a6db6c145acafc0f3068b8ce1fd (127.15.255.130) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: false } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } } }
I20240502 21:27:55.593981 24317 raft_consensus.cc:1261] T f84bbe304597450c978ae28e0c4e7a3e P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 2 FOLLOWER]: Refusing update from remote peer a3e500d7077a48acb5edb33f7e178c81: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 21:27:55.594738 29292 consensus_queue.cc:1035] T f84bbe304597450c978ae28e0c4e7a3e P a3e500d7077a48acb5edb33f7e178c81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:27:55.595247 29292 raft_consensus.cc:2932] T f84bbe304597450c978ae28e0c4e7a3e P a3e500d7077a48acb5edb33f7e178c81 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 6c194a6db6c145acafc0f3068b8ce1fd (127.15.255.130) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: false } } }
I20240502 21:27:55.595777 24316 raft_consensus.cc:2932] T f84bbe304597450c978ae28e0c4e7a3e P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, fb6004b8640a4906aa0cfe63e383d40e (127.15.255.134) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } attrs { replace: true } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: false } } }
I20240502 21:27:55.598940 23756 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f84bbe304597450c978ae28e0c4e7a3e with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:27:55.599280 25876 raft_consensus.cc:2932] T cca45c89b9fe40e7800d95c1598a9e52 P ee27c3f9e0ec46caa742ac386d6b7c20 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 6c194a6db6c145acafc0f3068b8ce1fd (127.15.255.130) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: false } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } } }
I20240502 21:27:55.600386 26012 raft_consensus.cc:2932] T f84bbe304597450c978ae28e0c4e7a3e P fb6004b8640a4906aa0cfe63e383d40e [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 6c194a6db6c145acafc0f3068b8ce1fd (127.15.255.130) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: false } } }
I20240502 21:27:55.601320 23765 catalog_manager.cc:5484] T f84bbe304597450c978ae28e0c4e7a3e P a3e500d7077a48acb5edb33f7e178c81 reported cstate change: config changed from index 9 to 10, VOTER 6c194a6db6c145acafc0f3068b8ce1fd (127.15.255.130) evicted. New cstate: current_term: 2 leader_uuid: "a3e500d7077a48acb5edb33f7e178c81" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:27:55.607311 23756 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet cca45c89b9fe40e7800d95c1598a9e52 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:27:55.609378 23771 catalog_manager.cc:5484] T cca45c89b9fe40e7800d95c1598a9e52 P 009d3e7e8a8b4e2497de3a73d01ced49 reported cstate change: config changed from index 8 to 9, VOTER 6c194a6db6c145acafc0f3068b8ce1fd (127.15.255.130) evicted. New cstate: current_term: 1 leader_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:55.611989 23916 raft_consensus.cc:1261] T 85e77d5faa8346cd8a99dcbcdf75f250 P 6810a866decb4f5089f7d35c870e4204 [term 1 FOLLOWER]: Refusing update from remote peer 009d3e7e8a8b4e2497de3a73d01ced49: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:27:55.612962 24047 raft_consensus.cc:1261] T 85e77d5faa8346cd8a99dcbcdf75f250 P 6c194a6db6c145acafc0f3068b8ce1fd [term 1 FOLLOWER]: Refusing update from remote peer 009d3e7e8a8b4e2497de3a73d01ced49: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:27:55.615103 26279 raft_consensus.cc:1261] T 4eebf4508766421eb6c547a0fbbc28b1 P a3e500d7077a48acb5edb33f7e178c81 [term 1 LEARNER]: Refusing update from remote peer 6810a866decb4f5089f7d35c870e4204: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:27:55.615684 24027 tablet_service.cc:1514] Processing DeleteTablet for tablet cca45c89b9fe40e7800d95c1598a9e52 with delete_type TABLET_DATA_TOMBSTONED (TS 6c194a6db6c145acafc0f3068b8ce1fd not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:47386
I20240502 21:27:55.616024 29256 tablet_replica.cc:330] T cca45c89b9fe40e7800d95c1598a9e52 P 6c194a6db6c145acafc0f3068b8ce1fd: stopping tablet replica
I20240502 21:27:55.616206 29256 raft_consensus.cc:2229] T cca45c89b9fe40e7800d95c1598a9e52 P 6c194a6db6c145acafc0f3068b8ce1fd [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:55.616653 29256 raft_consensus.cc:2258] T cca45c89b9fe40e7800d95c1598a9e52 P 6c194a6db6c145acafc0f3068b8ce1fd [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:55.618093 29256 ts_tablet_manager.cc:1875] T cca45c89b9fe40e7800d95c1598a9e52 P 6c194a6db6c145acafc0f3068b8ce1fd: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:27:55.616289 29296 consensus_queue.cc:1035] T 85e77d5faa8346cd8a99dcbcdf75f250 P 009d3e7e8a8b4e2497de3a73d01ced49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.002s
I20240502 21:27:55.620388 29308 consensus_queue.cc:1035] T 85e77d5faa8346cd8a99dcbcdf75f250 P 009d3e7e8a8b4e2497de3a73d01ced49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:27:55.626592 29256 ts_tablet_manager.cc:1888] T cca45c89b9fe40e7800d95c1598a9e52 P 6c194a6db6c145acafc0f3068b8ce1fd: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240502 21:27:55.626798 29256 log.cc:1196] T cca45c89b9fe40e7800d95c1598a9e52 P 6c194a6db6c145acafc0f3068b8ce1fd: Deleting WAL directory at /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-1/wal/wals/cca45c89b9fe40e7800d95c1598a9e52
I20240502 21:27:55.628291 23757 catalog_manager.cc:4830] TS 6c194a6db6c145acafc0f3068b8ce1fd (127.15.255.130:41235): tablet cca45c89b9fe40e7800d95c1598a9e52 (table rebalance_test_table_2 [id=85ca987b38d84a7db5119dfc88b4feaf]) successfully deleted
I20240502 21:27:55.628440 24027 tablet_service.cc:1514] Processing DeleteTablet for tablet f84bbe304597450c978ae28e0c4e7a3e with delete_type TABLET_DATA_TOMBSTONED (TS 6c194a6db6c145acafc0f3068b8ce1fd not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:47386
I20240502 21:27:55.628901 29256 tablet_replica.cc:330] T f84bbe304597450c978ae28e0c4e7a3e P 6c194a6db6c145acafc0f3068b8ce1fd: stopping tablet replica
I20240502 21:27:55.629139 29256 raft_consensus.cc:2229] T f84bbe304597450c978ae28e0c4e7a3e P 6c194a6db6c145acafc0f3068b8ce1fd [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:55.629417 29256 raft_consensus.cc:2258] T f84bbe304597450c978ae28e0c4e7a3e P 6c194a6db6c145acafc0f3068b8ce1fd [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:55.631050 29259 raft_consensus.cc:2932] T 85e77d5faa8346cd8a99dcbcdf75f250 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ee27c3f9e0ec46caa742ac386d6b7c20 (127.15.255.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } attrs { replace: true } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: false } } }
I20240502 21:27:55.632020 25876 raft_consensus.cc:2932] T 85e77d5faa8346cd8a99dcbcdf75f250 P ee27c3f9e0ec46caa742ac386d6b7c20 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ee27c3f9e0ec46caa742ac386d6b7c20 (127.15.255.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } attrs { replace: true } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: false } } }
I20240502 21:27:55.636205 23916 raft_consensus.cc:2932] T 85e77d5faa8346cd8a99dcbcdf75f250 P 6810a866decb4f5089f7d35c870e4204 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ee27c3f9e0ec46caa742ac386d6b7c20 (127.15.255.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } attrs { replace: true } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: false } } }
I20240502 21:27:55.639313 28790 consensus_queue.cc:1035] T 4eebf4508766421eb6c547a0fbbc28b1 P 6810a866decb4f5089f7d35c870e4204 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:27:55.639827 29256 ts_tablet_manager.cc:1875] T f84bbe304597450c978ae28e0c4e7a3e P 6c194a6db6c145acafc0f3068b8ce1fd: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:27:55.641146 24315 raft_consensus.cc:1261] T 4eebf4508766421eb6c547a0fbbc28b1 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 1 FOLLOWER]: Refusing update from remote peer 6810a866decb4f5089f7d35c870e4204: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:27:55.641975 29095 consensus_queue.cc:1035] T 4eebf4508766421eb6c547a0fbbc28b1 P 6810a866decb4f5089f7d35c870e4204 [LEADER]: Connected to new peer: Peer: permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:27:55.642457 29266 raft_consensus.cc:2932] T 52f621ea3f174490948eae854c706a92 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 4f58800ba2ed45eabd81c2f2e691aacd (127.15.255.137) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } attrs { replace: true } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: false } } peers { permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: false } } }
I20240502 21:27:55.643298 23770 catalog_manager.cc:5484] T 85e77d5faa8346cd8a99dcbcdf75f250 P ee27c3f9e0ec46caa742ac386d6b7c20 reported cstate change: config changed from index 4 to 5, ee27c3f9e0ec46caa742ac386d6b7c20 (127.15.255.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } attrs { replace: true } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: false } } }
I20240502 21:27:55.644850 26147 raft_consensus.cc:1261] T 4eebf4508766421eb6c547a0fbbc28b1 P e117afbf8819424e8a98476453dfde2a [term 1 FOLLOWER]: Refusing update from remote peer 6810a866decb4f5089f7d35c870e4204: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:27:55.645906 28790 consensus_queue.cc:1035] T 4eebf4508766421eb6c547a0fbbc28b1 P 6810a866decb4f5089f7d35c870e4204 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:27:55.650161 27127 raft_consensus.cc:2932] T 52f621ea3f174490948eae854c706a92 P 4f58800ba2ed45eabd81c2f2e691aacd [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 4f58800ba2ed45eabd81c2f2e691aacd (127.15.255.137) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } attrs { replace: true } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: false } } peers { permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: false } } }
I20240502 21:27:55.654615 23771 catalog_manager.cc:5484] T 52f621ea3f174490948eae854c706a92 P 038c5d9d1d7a49c981c4c34fbc00d9e6 reported cstate change: config changed from index 7 to 8, 4f58800ba2ed45eabd81c2f2e691aacd (127.15.255.137) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:27:55.662379 26280 raft_consensus.cc:1261] T cca45c89b9fe40e7800d95c1598a9e52 P a3e500d7077a48acb5edb33f7e178c81 [term 1 FOLLOWER]: Refusing update from remote peer 009d3e7e8a8b4e2497de3a73d01ced49: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:27:55.662940 29296 consensus_queue.cc:1035] T cca45c89b9fe40e7800d95c1598a9e52 P 009d3e7e8a8b4e2497de3a73d01ced49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:27:55.665784 26280 raft_consensus.cc:2932] T cca45c89b9fe40e7800d95c1598a9e52 P a3e500d7077a48acb5edb33f7e178c81 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 6c194a6db6c145acafc0f3068b8ce1fd (127.15.255.130) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: false } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } } }
I20240502 21:27:55.670658 24317 raft_consensus.cc:2932] T f84bbe304597450c978ae28e0c4e7a3e P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 6c194a6db6c145acafc0f3068b8ce1fd (127.15.255.130) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: false } } }
I20240502 21:27:55.672785 25878 raft_consensus.cc:2932] T 52f621ea3f174490948eae854c706a92 P ee27c3f9e0ec46caa742ac386d6b7c20 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 4f58800ba2ed45eabd81c2f2e691aacd (127.15.255.137) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } attrs { replace: true } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: false } } peers { permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: false } } }
I20240502 21:27:55.689873 28790 raft_consensus.cc:2932] T 4eebf4508766421eb6c547a0fbbc28b1 P 6810a866decb4f5089f7d35c870e4204 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, a3e500d7077a48acb5edb33f7e178c81 (127.15.255.136) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } attrs { replace: true } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } } }
I20240502 21:27:55.697770 29256 ts_tablet_manager.cc:1888] T f84bbe304597450c978ae28e0c4e7a3e P 6c194a6db6c145acafc0f3068b8ce1fd: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.9
I20240502 21:27:55.697960 29256 log.cc:1196] T f84bbe304597450c978ae28e0c4e7a3e P 6c194a6db6c145acafc0f3068b8ce1fd: Deleting WAL directory at /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-1/wal/wals/f84bbe304597450c978ae28e0c4e7a3e
I20240502 21:27:55.697681 24316 raft_consensus.cc:2932] T 4eebf4508766421eb6c547a0fbbc28b1 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, a3e500d7077a48acb5edb33f7e178c81 (127.15.255.136) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } attrs { replace: true } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } } }
I20240502 21:27:55.697681 24047 raft_consensus.cc:2932] T 85e77d5faa8346cd8a99dcbcdf75f250 P 6c194a6db6c145acafc0f3068b8ce1fd [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, ee27c3f9e0ec46caa742ac386d6b7c20 (127.15.255.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } attrs { replace: true } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: false } } }
I20240502 21:27:55.698871 23757 catalog_manager.cc:4830] TS 6c194a6db6c145acafc0f3068b8ce1fd (127.15.255.130:41235): tablet f84bbe304597450c978ae28e0c4e7a3e (table rebalance_test_table_7 [id=754dbb3bce6a4875a0b642faf5315709]) successfully deleted
I20240502 21:27:55.699252 26280 raft_consensus.cc:2932] T 4eebf4508766421eb6c547a0fbbc28b1 P a3e500d7077a48acb5edb33f7e178c81 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, a3e500d7077a48acb5edb33f7e178c81 (127.15.255.136) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } attrs { replace: true } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } } }
I20240502 21:27:55.702365 24183 consensus_queue.cc:237] T 85e77d5faa8346cd8a99dcbcdf75f250 P 009d3e7e8a8b4e2497de3a73d01ced49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: false } }
I20240502 21:27:55.703121 23771 catalog_manager.cc:5484] T 4eebf4508766421eb6c547a0fbbc28b1 P 038c5d9d1d7a49c981c4c34fbc00d9e6 reported cstate change: config changed from index 7 to 8, a3e500d7077a48acb5edb33f7e178c81 (127.15.255.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "6810a866decb4f5089f7d35c870e4204" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } attrs { replace: true } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } } }
I20240502 21:27:55.704905 26147 raft_consensus.cc:2932] T 4eebf4508766421eb6c547a0fbbc28b1 P e117afbf8819424e8a98476453dfde2a [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, a3e500d7077a48acb5edb33f7e178c81 (127.15.255.136) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } attrs { replace: true } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } } }
I20240502 21:27:55.706996 25877 raft_consensus.cc:1261] T 85e77d5faa8346cd8a99dcbcdf75f250 P ee27c3f9e0ec46caa742ac386d6b7c20 [term 1 FOLLOWER]: Refusing update from remote peer 009d3e7e8a8b4e2497de3a73d01ced49: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:27:55.707685 28493 consensus_queue.cc:1035] T 85e77d5faa8346cd8a99dcbcdf75f250 P 009d3e7e8a8b4e2497de3a73d01ced49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:27:55.710445 23916 consensus_queue.cc:237] T 4eebf4508766421eb6c547a0fbbc28b1 P 6810a866decb4f5089f7d35c870e4204 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } }
I20240502 21:27:55.712488 26147 raft_consensus.cc:1261] T 4eebf4508766421eb6c547a0fbbc28b1 P e117afbf8819424e8a98476453dfde2a [term 1 FOLLOWER]: Refusing update from remote peer 6810a866decb4f5089f7d35c870e4204: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:27:55.714468 24047 raft_consensus.cc:1261] T 85e77d5faa8346cd8a99dcbcdf75f250 P 6c194a6db6c145acafc0f3068b8ce1fd [term 1 FOLLOWER]: Refusing update from remote peer 009d3e7e8a8b4e2497de3a73d01ced49: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:27:55.715083 28493 consensus_queue.cc:1035] T 85e77d5faa8346cd8a99dcbcdf75f250 P 009d3e7e8a8b4e2497de3a73d01ced49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:27:55.715018 29259 raft_consensus.cc:2932] T 85e77d5faa8346cd8a99dcbcdf75f250 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 6810a866decb4f5089f7d35c870e4204 (127.15.255.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: false } } }
I20240502 21:27:55.716109 25877 raft_consensus.cc:2932] T 85e77d5faa8346cd8a99dcbcdf75f250 P ee27c3f9e0ec46caa742ac386d6b7c20 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 6810a866decb4f5089f7d35c870e4204 (127.15.255.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: false } } }
I20240502 21:27:55.719897 23756 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 85e77d5faa8346cd8a99dcbcdf75f250 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:27:55.720918 24047 raft_consensus.cc:2932] T 85e77d5faa8346cd8a99dcbcdf75f250 P 6c194a6db6c145acafc0f3068b8ce1fd [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 6810a866decb4f5089f7d35c870e4204 (127.15.255.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: false } } }
I20240502 21:27:55.722187 23766 catalog_manager.cc:5484] T 85e77d5faa8346cd8a99dcbcdf75f250 P 009d3e7e8a8b4e2497de3a73d01ced49 reported cstate change: config changed from index 5 to 6, VOTER 6810a866decb4f5089f7d35c870e4204 (127.15.255.129) evicted. New cstate: current_term: 1 leader_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:27:55.729377 23894 tablet_service.cc:1514] Processing DeleteTablet for tablet 85e77d5faa8346cd8a99dcbcdf75f250 with delete_type TABLET_DATA_TOMBSTONED (TS 6810a866decb4f5089f7d35c870e4204 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:33512
I20240502 21:27:55.729751 29251 tablet_replica.cc:330] T 85e77d5faa8346cd8a99dcbcdf75f250 P 6810a866decb4f5089f7d35c870e4204: stopping tablet replica
I20240502 21:27:55.729967 29251 raft_consensus.cc:2229] T 85e77d5faa8346cd8a99dcbcdf75f250 P 6810a866decb4f5089f7d35c870e4204 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:55.730314 29251 raft_consensus.cc:2258] T 85e77d5faa8346cd8a99dcbcdf75f250 P 6810a866decb4f5089f7d35c870e4204 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:55.731362 29251 ts_tablet_manager.cc:1875] T 85e77d5faa8346cd8a99dcbcdf75f250 P 6810a866decb4f5089f7d35c870e4204: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:27:55.733037 28790 consensus_queue.cc:1035] T 4eebf4508766421eb6c547a0fbbc28b1 P 6810a866decb4f5089f7d35c870e4204 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:27:55.737428 28790 raft_consensus.cc:2932] T 4eebf4508766421eb6c547a0fbbc28b1 P 6810a866decb4f5089f7d35c870e4204 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 038c5d9d1d7a49c981c4c34fbc00d9e6 (127.15.255.132) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } } }
I20240502 21:27:55.738615 29251 ts_tablet_manager.cc:1888] T 85e77d5faa8346cd8a99dcbcdf75f250 P 6810a866decb4f5089f7d35c870e4204: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 21:27:55.738726 29251 log.cc:1196] T 85e77d5faa8346cd8a99dcbcdf75f250 P 6810a866decb4f5089f7d35c870e4204: Deleting WAL directory at /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-0/wal/wals/85e77d5faa8346cd8a99dcbcdf75f250
I20240502 21:27:55.739548 26279 raft_consensus.cc:1261] T 4eebf4508766421eb6c547a0fbbc28b1 P a3e500d7077a48acb5edb33f7e178c81 [term 1 FOLLOWER]: Refusing update from remote peer 6810a866decb4f5089f7d35c870e4204: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:27:55.711122 23915 raft_consensus.cc:2932] T 52f621ea3f174490948eae854c706a92 P 6810a866decb4f5089f7d35c870e4204 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 4f58800ba2ed45eabd81c2f2e691aacd (127.15.255.137) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } attrs { replace: true } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: false } } peers { permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: false } } }
I20240502 21:27:55.741338 23757 catalog_manager.cc:4830] TS 6810a866decb4f5089f7d35c870e4204 (127.15.255.129:40435): tablet 85e77d5faa8346cd8a99dcbcdf75f250 (table rebalance_test_table_1 [id=e962de13fbc44d1986b1e1ee191d22e2]) successfully deleted
I20240502 21:27:55.741168 26147 raft_consensus.cc:2932] T 4eebf4508766421eb6c547a0fbbc28b1 P e117afbf8819424e8a98476453dfde2a [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 038c5d9d1d7a49c981c4c34fbc00d9e6 (127.15.255.132) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } } }
I20240502 21:27:55.741693 29095 consensus_queue.cc:1035] T 4eebf4508766421eb6c547a0fbbc28b1 P 6810a866decb4f5089f7d35c870e4204 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:27:55.743767 23757 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 4eebf4508766421eb6c547a0fbbc28b1 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:27:55.744756 26280 raft_consensus.cc:2932] T 4eebf4508766421eb6c547a0fbbc28b1 P a3e500d7077a48acb5edb33f7e178c81 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 038c5d9d1d7a49c981c4c34fbc00d9e6 (127.15.255.132) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } } }
I20240502 21:27:55.746781 23771 catalog_manager.cc:5484] T 4eebf4508766421eb6c547a0fbbc28b1 P 6810a866decb4f5089f7d35c870e4204 reported cstate change: config changed from index 8 to 9, VOTER 038c5d9d1d7a49c981c4c34fbc00d9e6 (127.15.255.132) evicted. New cstate: current_term: 1 leader_uuid: "6810a866decb4f5089f7d35c870e4204" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:55.750857 24297 tablet_service.cc:1514] Processing DeleteTablet for tablet 4eebf4508766421eb6c547a0fbbc28b1 with delete_type TABLET_DATA_TOMBSTONED (TS 038c5d9d1d7a49c981c4c34fbc00d9e6 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:58268
I20240502 21:27:55.764211 29323 tablet_replica.cc:330] T 4eebf4508766421eb6c547a0fbbc28b1 P 038c5d9d1d7a49c981c4c34fbc00d9e6: stopping tablet replica
I20240502 21:27:55.764492 29323 raft_consensus.cc:2229] T 4eebf4508766421eb6c547a0fbbc28b1 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:55.764761 29323 raft_consensus.cc:2258] T 4eebf4508766421eb6c547a0fbbc28b1 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:55.765661 29323 ts_tablet_manager.cc:1875] T 4eebf4508766421eb6c547a0fbbc28b1 P 038c5d9d1d7a49c981c4c34fbc00d9e6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:27:55.771039 29323 ts_tablet_manager.cc:1888] T 4eebf4508766421eb6c547a0fbbc28b1 P 038c5d9d1d7a49c981c4c34fbc00d9e6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240502 21:27:55.771260 29323 log.cc:1196] T 4eebf4508766421eb6c547a0fbbc28b1 P 038c5d9d1d7a49c981c4c34fbc00d9e6: Deleting WAL directory at /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-3/wal/wals/4eebf4508766421eb6c547a0fbbc28b1
I20240502 21:27:55.772018 23755 catalog_manager.cc:4830] TS 038c5d9d1d7a49c981c4c34fbc00d9e6 (127.15.255.132:46813): tablet 4eebf4508766421eb6c547a0fbbc28b1 (table rebalance_test_table_1 [id=e962de13fbc44d1986b1e1ee191d22e2]) successfully deleted
I20240502 21:27:55.772377 24315 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "52f621ea3f174490948eae854c706a92"
dest_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:38228
I20240502 21:27:55.772570 24315 raft_consensus.cc:595] T 52f621ea3f174490948eae854c706a92 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 1 LEADER]: Received request to transfer leadership
I20240502 21:27:55.833321 24183 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "0b3c514213bb40ca98955d6409a3b372"
dest_uuid: "009d3e7e8a8b4e2497de3a73d01ced49"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:58964
I20240502 21:27:55.833534 24183 raft_consensus.cc:595] T 0b3c514213bb40ca98955d6409a3b372 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 1 LEADER]: Received request to transfer leadership
I20240502 21:27:55.883440 23915 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "30236241b670443489240f191d9b9d50"
dest_uuid: "6810a866decb4f5089f7d35c870e4204"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:46448
I20240502 21:27:55.883666 23915 raft_consensus.cc:595] T 30236241b670443489240f191d9b9d50 P 6810a866decb4f5089f7d35c870e4204 [term 1 LEADER]: Received request to transfer leadership
I20240502 21:27:55.903502 24315 consensus_queue.cc:237] T d5697d8b06094211b261f144de9d46f8 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } attrs { replace: true } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } } peers { permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: NON_VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: true } }
I20240502 21:27:55.905537 26280 raft_consensus.cc:1261] T d5697d8b06094211b261f144de9d46f8 P a3e500d7077a48acb5edb33f7e178c81 [term 1 FOLLOWER]: Refusing update from remote peer 038c5d9d1d7a49c981c4c34fbc00d9e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:27:55.905701 24183 raft_consensus.cc:1261] T d5697d8b06094211b261f144de9d46f8 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 1 FOLLOWER]: Refusing update from remote peer 038c5d9d1d7a49c981c4c34fbc00d9e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:27:55.906157 29265 consensus_queue.cc:1035] T d5697d8b06094211b261f144de9d46f8 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240502 21:27:55.906515 24251 consensus_peers.cc:485] T d5697d8b06094211b261f144de9d46f8 P 038c5d9d1d7a49c981c4c34fbc00d9e6 -> Peer 4f58800ba2ed45eabd81c2f2e691aacd (127.15.255.137:40161): Couldn't send request to peer 4f58800ba2ed45eabd81c2f2e691aacd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d5697d8b06094211b261f144de9d46f8. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:55.906536 29266 consensus_queue.cc:1035] T d5697d8b06094211b261f144de9d46f8 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:27:55.909332 29101 raft_consensus.cc:2932] T d5697d8b06094211b261f144de9d46f8 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 4f58800ba2ed45eabd81c2f2e691aacd (127.15.255.137) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } attrs { replace: true } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } } peers { permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: NON_VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: true } } }
I20240502 21:27:55.909917 24183 raft_consensus.cc:2932] T d5697d8b06094211b261f144de9d46f8 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 4f58800ba2ed45eabd81c2f2e691aacd (127.15.255.137) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } attrs { replace: true } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } } peers { permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: NON_VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: true } } }
I20240502 21:27:55.909998 26280 raft_consensus.cc:2932] T d5697d8b06094211b261f144de9d46f8 P a3e500d7077a48acb5edb33f7e178c81 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 4f58800ba2ed45eabd81c2f2e691aacd (127.15.255.137) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } attrs { replace: true } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } } peers { permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: NON_VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: true } } }
I20240502 21:27:55.915454 23770 catalog_manager.cc:5484] T d5697d8b06094211b261f144de9d46f8 P 009d3e7e8a8b4e2497de3a73d01ced49 reported cstate change: config changed from index 6 to 7, NON_VOTER 4f58800ba2ed45eabd81c2f2e691aacd (127.15.255.137) added. New cstate: current_term: 1 leader_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } attrs { replace: true } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } } peers { permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: NON_VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: true } } }
I20240502 21:27:55.928265 24315 consensus_queue.cc:237] T fef55863beb6435dbb25bfb6d71e3cd7 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } attrs { replace: true } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: NON_VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: true } }
I20240502 21:27:55.930259 26280 raft_consensus.cc:1261] T fef55863beb6435dbb25bfb6d71e3cd7 P a3e500d7077a48acb5edb33f7e178c81 [term 1 FOLLOWER]: Refusing update from remote peer 038c5d9d1d7a49c981c4c34fbc00d9e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:27:55.930292 23915 raft_consensus.cc:1261] T fef55863beb6435dbb25bfb6d71e3cd7 P 6810a866decb4f5089f7d35c870e4204 [term 1 FOLLOWER]: Refusing update from remote peer 038c5d9d1d7a49c981c4c34fbc00d9e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240502 21:27:55.930454 24250 consensus_peers.cc:485] T fef55863beb6435dbb25bfb6d71e3cd7 P 038c5d9d1d7a49c981c4c34fbc00d9e6 -> Peer fb6004b8640a4906aa0cfe63e383d40e (127.15.255.134:32825): Couldn't send request to peer fb6004b8640a4906aa0cfe63e383d40e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fef55863beb6435dbb25bfb6d71e3cd7. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:55.930934 29265 consensus_queue.cc:1035] T fef55863beb6435dbb25bfb6d71e3cd7 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:27:55.931260 29263 consensus_queue.cc:1035] T fef55863beb6435dbb25bfb6d71e3cd7 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:27:55.934257 29101 raft_consensus.cc:2932] T fef55863beb6435dbb25bfb6d71e3cd7 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER fb6004b8640a4906aa0cfe63e383d40e (127.15.255.134) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } attrs { replace: true } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: NON_VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: true } } }
I20240502 21:27:55.934916 23915 raft_consensus.cc:2932] T fef55863beb6435dbb25bfb6d71e3cd7 P 6810a866decb4f5089f7d35c870e4204 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER fb6004b8640a4906aa0cfe63e383d40e (127.15.255.134) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } attrs { replace: true } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: NON_VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: true } } }
I20240502 21:27:55.935020 26280 raft_consensus.cc:2932] T fef55863beb6435dbb25bfb6d71e3cd7 P a3e500d7077a48acb5edb33f7e178c81 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER fb6004b8640a4906aa0cfe63e383d40e (127.15.255.134) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } attrs { replace: true } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: NON_VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: true } } }
I20240502 21:27:55.939502 23766 catalog_manager.cc:5484] T fef55863beb6435dbb25bfb6d71e3cd7 P 038c5d9d1d7a49c981c4c34fbc00d9e6 reported cstate change: config changed from index 6 to 7, NON_VOTER fb6004b8640a4906aa0cfe63e383d40e (127.15.255.134) added. New cstate: current_term: 1 leader_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: NON_VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:55.945396 29095 raft_consensus.cc:979] T 30236241b670443489240f191d9b9d50 P 6810a866decb4f5089f7d35c870e4204: : Instructing follower a3e500d7077a48acb5edb33f7e178c81 to start an election
I20240502 21:27:55.945554 29095 raft_consensus.cc:1067] T 30236241b670443489240f191d9b9d50 P 6810a866decb4f5089f7d35c870e4204 [term 1 LEADER]: Signalling peer a3e500d7077a48acb5edb33f7e178c81 to start an election
I20240502 21:27:55.946192 26280 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "30236241b670443489240f191d9b9d50"
dest_uuid: "a3e500d7077a48acb5edb33f7e178c81"
from {username='slave'} at 127.15.255.129:44953
I20240502 21:27:55.946385 26280 raft_consensus.cc:482] T 30236241b670443489240f191d9b9d50 P a3e500d7077a48acb5edb33f7e178c81 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:55.946481 26280 raft_consensus.cc:3037] T 30236241b670443489240f191d9b9d50 P a3e500d7077a48acb5edb33f7e178c81 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:55.949061 26280 raft_consensus.cc:504] T 30236241b670443489240f191d9b9d50 P a3e500d7077a48acb5edb33f7e178c81 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } attrs { replace: true } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: false } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } }
I20240502 21:27:55.950188 26280 leader_election.cc:288] T 30236241b670443489240f191d9b9d50 P a3e500d7077a48acb5edb33f7e178c81 [CANDIDATE]: Term 2 election: Requested vote from peers 6810a866decb4f5089f7d35c870e4204 (127.15.255.129:40435), 038c5d9d1d7a49c981c4c34fbc00d9e6 (127.15.255.132:46813), fb6004b8640a4906aa0cfe63e383d40e (127.15.255.134:32825)
I20240502 21:27:55.950368 23915 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30236241b670443489240f191d9b9d50" candidate_uuid: "a3e500d7077a48acb5edb33f7e178c81" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "6810a866decb4f5089f7d35c870e4204"
I20240502 21:27:55.950569 23915 raft_consensus.cc:3032] T 30236241b670443489240f191d9b9d50 P 6810a866decb4f5089f7d35c870e4204 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 21:27:55.950577 24315 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30236241b670443489240f191d9b9d50" candidate_uuid: "a3e500d7077a48acb5edb33f7e178c81" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6"
I20240502 21:27:55.950663 23915 raft_consensus.cc:729] T 30236241b670443489240f191d9b9d50 P 6810a866decb4f5089f7d35c870e4204 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 6810a866decb4f5089f7d35c870e4204, State: Running, Role: LEADER
I20240502 21:27:55.950788 24315 raft_consensus.cc:3037] T 30236241b670443489240f191d9b9d50 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:55.950812 26012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30236241b670443489240f191d9b9d50" candidate_uuid: "a3e500d7077a48acb5edb33f7e178c81" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "fb6004b8640a4906aa0cfe63e383d40e"
I20240502 21:27:55.951001 26012 raft_consensus.cc:3037] T 30236241b670443489240f191d9b9d50 P fb6004b8640a4906aa0cfe63e383d40e [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:55.950947 23915 consensus_queue.cc:260] T 30236241b670443489240f191d9b9d50 P 6810a866decb4f5089f7d35c870e4204 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } attrs { replace: true } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: false } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } }
I20240502 21:27:55.951380 23915 raft_consensus.cc:3037] T 30236241b670443489240f191d9b9d50 P 6810a866decb4f5089f7d35c870e4204 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:55.952950 26012 raft_consensus.cc:2454] T 30236241b670443489240f191d9b9d50 P fb6004b8640a4906aa0cfe63e383d40e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a3e500d7077a48acb5edb33f7e178c81 in term 2.
I20240502 21:27:55.953438 24315 raft_consensus.cc:2454] T 30236241b670443489240f191d9b9d50 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a3e500d7077a48acb5edb33f7e178c81 in term 2.
I20240502 21:27:55.953846 26212 leader_election.cc:302] T 30236241b670443489240f191d9b9d50 P a3e500d7077a48acb5edb33f7e178c81 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 038c5d9d1d7a49c981c4c34fbc00d9e6, a3e500d7077a48acb5edb33f7e178c81, fb6004b8640a4906aa0cfe63e383d40e; no voters:
I20240502 21:27:55.954015 23915 raft_consensus.cc:2454] T 30236241b670443489240f191d9b9d50 P 6810a866decb4f5089f7d35c870e4204 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a3e500d7077a48acb5edb33f7e178c81 in term 2.
I20240502 21:27:55.954128 29292 raft_consensus.cc:2781] T 30236241b670443489240f191d9b9d50 P a3e500d7077a48acb5edb33f7e178c81 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 21:27:55.954308 29292 raft_consensus.cc:686] T 30236241b670443489240f191d9b9d50 P a3e500d7077a48acb5edb33f7e178c81 [term 2 LEADER]: Becoming Leader. State: Replica: a3e500d7077a48acb5edb33f7e178c81, State: Running, Role: LEADER
I20240502 21:27:55.954568 29292 consensus_queue.cc:237] T 30236241b670443489240f191d9b9d50 P a3e500d7077a48acb5edb33f7e178c81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } attrs { replace: true } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: false } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } }
I20240502 21:27:55.957149 23771 catalog_manager.cc:5484] T 30236241b670443489240f191d9b9d50 P a3e500d7077a48acb5edb33f7e178c81 reported cstate change: term changed from 1 to 2, leader changed from 6810a866decb4f5089f7d35c870e4204 (127.15.255.129) to a3e500d7077a48acb5edb33f7e178c81 (127.15.255.136). New cstate: current_term: 2 leader_uuid: "a3e500d7077a48acb5edb33f7e178c81" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:27:55.958114 23916 consensus_queue.cc:237] T 8f14db150fda449e962be2466f1d2a0e P 6810a866decb4f5089f7d35c870e4204 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } attrs { replace: true } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: false } } peers { permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: NON_VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: true } }
I20240502 21:27:55.960768 24183 raft_consensus.cc:1261] T 8f14db150fda449e962be2466f1d2a0e P 009d3e7e8a8b4e2497de3a73d01ced49 [term 1 FOLLOWER]: Refusing update from remote peer 6810a866decb4f5089f7d35c870e4204: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:27:55.961251 25878 raft_consensus.cc:1261] T 8f14db150fda449e962be2466f1d2a0e P ee27c3f9e0ec46caa742ac386d6b7c20 [term 1 FOLLOWER]: Refusing update from remote peer 6810a866decb4f5089f7d35c870e4204: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:27:55.961424 29095 consensus_queue.cc:1035] T 8f14db150fda449e962be2466f1d2a0e P 6810a866decb4f5089f7d35c870e4204 [LEADER]: Connected to new peer: Peer: permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:27:55.962561 29095 consensus_queue.cc:1035] T 8f14db150fda449e962be2466f1d2a0e P 6810a866decb4f5089f7d35c870e4204 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:27:55.965938 28790 raft_consensus.cc:2932] T 8f14db150fda449e962be2466f1d2a0e P 6810a866decb4f5089f7d35c870e4204 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 4f58800ba2ed45eabd81c2f2e691aacd (127.15.255.137) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } attrs { replace: true } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: false } } peers { permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: NON_VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: true } } }
I20240502 21:27:55.966624 25878 raft_consensus.cc:2932] T 8f14db150fda449e962be2466f1d2a0e P ee27c3f9e0ec46caa742ac386d6b7c20 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 4f58800ba2ed45eabd81c2f2e691aacd (127.15.255.137) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } attrs { replace: true } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: false } } peers { permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: NON_VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: true } } }
I20240502 21:27:55.971753 23771 catalog_manager.cc:5484] T 8f14db150fda449e962be2466f1d2a0e P 6810a866decb4f5089f7d35c870e4204 reported cstate change: config changed from index 6 to 7, NON_VOTER 4f58800ba2ed45eabd81c2f2e691aacd (127.15.255.137) added. New cstate: current_term: 1 leader_uuid: "6810a866decb4f5089f7d35c870e4204" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: NON_VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:55.979529 24183 raft_consensus.cc:2932] T 8f14db150fda449e962be2466f1d2a0e P 009d3e7e8a8b4e2497de3a73d01ced49 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 4f58800ba2ed45eabd81c2f2e691aacd (127.15.255.137) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } attrs { replace: true } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: false } } peers { permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: NON_VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: true } } }
W20240502 21:27:55.995952 23849 consensus_peers.cc:485] T 8f14db150fda449e962be2466f1d2a0e P 6810a866decb4f5089f7d35c870e4204 -> Peer 4f58800ba2ed45eabd81c2f2e691aacd (127.15.255.137:40161): Couldn't send request to peer 4f58800ba2ed45eabd81c2f2e691aacd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8f14db150fda449e962be2466f1d2a0e. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:55.999107 23916 consensus_queue.cc:237] T 83b95f3ae7564b81b31a2f5673ee22bb P 6810a866decb4f5089f7d35c870e4204 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } attrs { replace: true } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } } peers { permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: NON_VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: true } }
I20240502 21:27:56.001412 26147 raft_consensus.cc:1261] T 83b95f3ae7564b81b31a2f5673ee22bb P e117afbf8819424e8a98476453dfde2a [term 1 FOLLOWER]: Refusing update from remote peer 6810a866decb4f5089f7d35c870e4204: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240502 21:27:56.001617 23849 consensus_peers.cc:485] T 83b95f3ae7564b81b31a2f5673ee22bb P 6810a866decb4f5089f7d35c870e4204 -> Peer 4f58800ba2ed45eabd81c2f2e691aacd (127.15.255.137:40161): Couldn't send request to peer 4f58800ba2ed45eabd81c2f2e691aacd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 83b95f3ae7564b81b31a2f5673ee22bb. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:56.001641 24315 raft_consensus.cc:1261] T 83b95f3ae7564b81b31a2f5673ee22bb P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 1 FOLLOWER]: Refusing update from remote peer 6810a866decb4f5089f7d35c870e4204: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:27:56.002027 27234 consensus_queue.cc:1035] T 83b95f3ae7564b81b31a2f5673ee22bb P 6810a866decb4f5089f7d35c870e4204 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:27:56.002465 29095 consensus_queue.cc:1035] T 83b95f3ae7564b81b31a2f5673ee22bb P 6810a866decb4f5089f7d35c870e4204 [LEADER]: Connected to new peer: Peer: permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:27:56.005359 29095 raft_consensus.cc:2932] T 83b95f3ae7564b81b31a2f5673ee22bb P 6810a866decb4f5089f7d35c870e4204 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 4f58800ba2ed45eabd81c2f2e691aacd (127.15.255.137) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } attrs { replace: true } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } } peers { permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: NON_VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: true } } }
I20240502 21:27:56.006132 26147 raft_consensus.cc:2932] T 83b95f3ae7564b81b31a2f5673ee22bb P e117afbf8819424e8a98476453dfde2a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 4f58800ba2ed45eabd81c2f2e691aacd (127.15.255.137) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } attrs { replace: true } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } } peers { permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: NON_VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: true } } }
I20240502 21:27:56.009714 24315 raft_consensus.cc:2932] T 83b95f3ae7564b81b31a2f5673ee22bb P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 4f58800ba2ed45eabd81c2f2e691aacd (127.15.255.137) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } attrs { replace: true } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } } peers { permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: NON_VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: true } } }
I20240502 21:27:56.010315 23766 catalog_manager.cc:5484] T 83b95f3ae7564b81b31a2f5673ee22bb P e117afbf8819424e8a98476453dfde2a reported cstate change: config changed from index 6 to 7, NON_VOTER 4f58800ba2ed45eabd81c2f2e691aacd (127.15.255.137) added. New cstate: current_term: 1 leader_uuid: "6810a866decb4f5089f7d35c870e4204" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } attrs { replace: true } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } } peers { permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: NON_VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: true } } }
I20240502 21:27:56.070917 24315 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "52f621ea3f174490948eae854c706a92"
dest_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:38274
I20240502 21:27:56.071141 24315 raft_consensus.cc:595] T 52f621ea3f174490948eae854c706a92 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 1 LEADER]: Received request to transfer leadership
I20240502 21:27:56.092233 24183 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "0b3c514213bb40ca98955d6409a3b372"
dest_uuid: "009d3e7e8a8b4e2497de3a73d01ced49"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:58984
I20240502 21:27:56.092566 24183 raft_consensus.cc:595] T 0b3c514213bb40ca98955d6409a3b372 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 1 LEADER]: Received request to transfer leadership
I20240502 21:27:56.154099 24315 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "52f621ea3f174490948eae854c706a92"
dest_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:38306
I20240502 21:27:56.154302 24315 raft_consensus.cc:595] T 52f621ea3f174490948eae854c706a92 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 1 LEADER]: Received request to transfer leadership
I20240502 21:27:56.169389 24183 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "0b3c514213bb40ca98955d6409a3b372"
dest_uuid: "009d3e7e8a8b4e2497de3a73d01ced49"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:59014
I20240502 21:27:56.169579 24183 raft_consensus.cc:595] T 0b3c514213bb40ca98955d6409a3b372 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 1 LEADER]: Received request to transfer leadership
I20240502 21:27:56.190703 29265 raft_consensus.cc:979] T 52f621ea3f174490948eae854c706a92 P 038c5d9d1d7a49c981c4c34fbc00d9e6: : Instructing follower ee27c3f9e0ec46caa742ac386d6b7c20 to start an election
I20240502 21:27:56.190873 29265 raft_consensus.cc:1067] T 52f621ea3f174490948eae854c706a92 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 1 LEADER]: Signalling peer ee27c3f9e0ec46caa742ac386d6b7c20 to start an election
I20240502 21:27:56.191576 25878 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "52f621ea3f174490948eae854c706a92"
dest_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20"
from {username='slave'} at 127.15.255.132:42901
I20240502 21:27:56.191841 25878 raft_consensus.cc:482] T 52f621ea3f174490948eae854c706a92 P ee27c3f9e0ec46caa742ac386d6b7c20 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:56.192029 25878 raft_consensus.cc:3037] T 52f621ea3f174490948eae854c706a92 P ee27c3f9e0ec46caa742ac386d6b7c20 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:56.194284 25878 raft_consensus.cc:504] T 52f621ea3f174490948eae854c706a92 P ee27c3f9e0ec46caa742ac386d6b7c20 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } attrs { replace: true } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: false } } peers { permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: false } }
I20240502 21:27:56.195905 25878 leader_election.cc:288] T 52f621ea3f174490948eae854c706a92 P ee27c3f9e0ec46caa742ac386d6b7c20 [CANDIDATE]: Term 2 election: Requested vote from peers 6810a866decb4f5089f7d35c870e4204 (127.15.255.129:40435), 038c5d9d1d7a49c981c4c34fbc00d9e6 (127.15.255.132:46813), 4f58800ba2ed45eabd81c2f2e691aacd (127.15.255.137:40161)
I20240502 21:27:56.196262 23916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52f621ea3f174490948eae854c706a92" candidate_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "6810a866decb4f5089f7d35c870e4204"
I20240502 21:27:56.196552 23916 raft_consensus.cc:3037] T 52f621ea3f174490948eae854c706a92 P 6810a866decb4f5089f7d35c870e4204 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:56.196487 24315 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52f621ea3f174490948eae854c706a92" candidate_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6"
I20240502 21:27:56.196722 24315 raft_consensus.cc:3032] T 52f621ea3f174490948eae854c706a92 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 21:27:56.196801 24315 raft_consensus.cc:729] T 52f621ea3f174490948eae854c706a92 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 038c5d9d1d7a49c981c4c34fbc00d9e6, State: Running, Role: LEADER
I20240502 21:27:56.197117 24315 consensus_queue.cc:260] T 52f621ea3f174490948eae854c706a92 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } attrs { replace: true } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: false } } peers { permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: false } }
I20240502 21:27:56.197592 24315 raft_consensus.cc:3037] T 52f621ea3f174490948eae854c706a92 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:56.198573 23916 raft_consensus.cc:2454] T 52f621ea3f174490948eae854c706a92 P 6810a866decb4f5089f7d35c870e4204 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee27c3f9e0ec46caa742ac386d6b7c20 in term 2.
I20240502 21:27:56.200387 24315 raft_consensus.cc:2454] T 52f621ea3f174490948eae854c706a92 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee27c3f9e0ec46caa742ac386d6b7c20 in term 2.
I20240502 21:27:56.201135 25812 leader_election.cc:302] T 52f621ea3f174490948eae854c706a92 P ee27c3f9e0ec46caa742ac386d6b7c20 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 038c5d9d1d7a49c981c4c34fbc00d9e6, 6810a866decb4f5089f7d35c870e4204, ee27c3f9e0ec46caa742ac386d6b7c20; no voters:
I20240502 21:27:56.201510 29015 raft_consensus.cc:2781] T 52f621ea3f174490948eae854c706a92 P ee27c3f9e0ec46caa742ac386d6b7c20 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 21:27:56.202015 29015 raft_consensus.cc:686] T 52f621ea3f174490948eae854c706a92 P ee27c3f9e0ec46caa742ac386d6b7c20 [term 2 LEADER]: Becoming Leader. State: Replica: ee27c3f9e0ec46caa742ac386d6b7c20, State: Running, Role: LEADER
I20240502 21:27:56.202644 29015 consensus_queue.cc:237] T 52f621ea3f174490948eae854c706a92 P ee27c3f9e0ec46caa742ac386d6b7c20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } attrs { replace: true } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: false } } peers { permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: false } }
I20240502 21:27:56.206797 23766 catalog_manager.cc:5484] T 52f621ea3f174490948eae854c706a92 P ee27c3f9e0ec46caa742ac386d6b7c20 reported cstate change: term changed from 1 to 2, leader changed from 038c5d9d1d7a49c981c4c34fbc00d9e6 (127.15.255.132) to ee27c3f9e0ec46caa742ac386d6b7c20 (127.15.255.133). New cstate: current_term: 2 leader_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:56.209249 27125 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52f621ea3f174490948eae854c706a92" candidate_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "4f58800ba2ed45eabd81c2f2e691aacd"
I20240502 21:27:56.209604 27125 raft_consensus.cc:3037] T 52f621ea3f174490948eae854c706a92 P 4f58800ba2ed45eabd81c2f2e691aacd [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:56.212275 27125 raft_consensus.cc:2454] T 52f621ea3f174490948eae854c706a92 P 4f58800ba2ed45eabd81c2f2e691aacd [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee27c3f9e0ec46caa742ac386d6b7c20 in term 2.
I20240502 21:27:56.263217 27842 raft_consensus.cc:979] T 0b3c514213bb40ca98955d6409a3b372 P 009d3e7e8a8b4e2497de3a73d01ced49: : Instructing follower 4f58800ba2ed45eabd81c2f2e691aacd to start an election
I20240502 21:27:56.263396 27842 raft_consensus.cc:1067] T 0b3c514213bb40ca98955d6409a3b372 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 1 LEADER]: Signalling peer 4f58800ba2ed45eabd81c2f2e691aacd to start an election
I20240502 21:27:56.264143 27125 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "0b3c514213bb40ca98955d6409a3b372"
dest_uuid: "4f58800ba2ed45eabd81c2f2e691aacd"
from {username='slave'} at 127.15.255.131:58067
I20240502 21:27:56.264506 27125 raft_consensus.cc:482] T 0b3c514213bb40ca98955d6409a3b372 P 4f58800ba2ed45eabd81c2f2e691aacd [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:56.264665 27125 raft_consensus.cc:3037] T 0b3c514213bb40ca98955d6409a3b372 P 4f58800ba2ed45eabd81c2f2e691aacd [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:56.267601 27125 raft_consensus.cc:504] T 0b3c514213bb40ca98955d6409a3b372 P 4f58800ba2ed45eabd81c2f2e691aacd [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } attrs { replace: true } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: false } } peers { permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: false } }
I20240502 21:27:56.276523 27125 leader_election.cc:288] T 0b3c514213bb40ca98955d6409a3b372 P 4f58800ba2ed45eabd81c2f2e691aacd [CANDIDATE]: Term 2 election: Requested vote from peers 009d3e7e8a8b4e2497de3a73d01ced49 (127.15.255.131:37895), 038c5d9d1d7a49c981c4c34fbc00d9e6 (127.15.255.132:46813), fb6004b8640a4906aa0cfe63e383d40e (127.15.255.134:32825)
I20240502 21:27:56.284372 24315 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b3c514213bb40ca98955d6409a3b372" candidate_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6"
I20240502 21:27:56.284657 24315 raft_consensus.cc:3037] T 0b3c514213bb40ca98955d6409a3b372 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:56.285391 26012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b3c514213bb40ca98955d6409a3b372" candidate_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "fb6004b8640a4906aa0cfe63e383d40e"
I20240502 21:27:56.285634 26012 raft_consensus.cc:3037] T 0b3c514213bb40ca98955d6409a3b372 P fb6004b8640a4906aa0cfe63e383d40e [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:56.287230 24315 raft_consensus.cc:2454] T 0b3c514213bb40ca98955d6409a3b372 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4f58800ba2ed45eabd81c2f2e691aacd in term 2.
I20240502 21:27:56.287910 24183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b3c514213bb40ca98955d6409a3b372" candidate_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "009d3e7e8a8b4e2497de3a73d01ced49"
I20240502 21:27:56.288153 24183 raft_consensus.cc:3032] T 0b3c514213bb40ca98955d6409a3b372 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 21:27:56.288241 24183 raft_consensus.cc:729] T 0b3c514213bb40ca98955d6409a3b372 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 009d3e7e8a8b4e2497de3a73d01ced49, State: Running, Role: LEADER
I20240502 21:27:56.288379 26012 raft_consensus.cc:2454] T 0b3c514213bb40ca98955d6409a3b372 P fb6004b8640a4906aa0cfe63e383d40e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4f58800ba2ed45eabd81c2f2e691aacd in term 2.
I20240502 21:27:56.288619 24183 consensus_queue.cc:260] T 0b3c514213bb40ca98955d6409a3b372 P 009d3e7e8a8b4e2497de3a73d01ced49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } attrs { replace: true } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: false } } peers { permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: false } }
I20240502 21:27:56.288856 27024 leader_election.cc:302] T 0b3c514213bb40ca98955d6409a3b372 P 4f58800ba2ed45eabd81c2f2e691aacd [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 038c5d9d1d7a49c981c4c34fbc00d9e6, 4f58800ba2ed45eabd81c2f2e691aacd, fb6004b8640a4906aa0cfe63e383d40e; no voters:
I20240502 21:27:56.289059 24183 raft_consensus.cc:3037] T 0b3c514213bb40ca98955d6409a3b372 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:56.289497 29536 raft_consensus.cc:2781] T 0b3c514213bb40ca98955d6409a3b372 P 4f58800ba2ed45eabd81c2f2e691aacd [term 2 FOLLOWER]: Leader election won for term 2
I20240502 21:27:56.289856 29536 raft_consensus.cc:686] T 0b3c514213bb40ca98955d6409a3b372 P 4f58800ba2ed45eabd81c2f2e691aacd [term 2 LEADER]: Becoming Leader. State: Replica: 4f58800ba2ed45eabd81c2f2e691aacd, State: Running, Role: LEADER
I20240502 21:27:56.290340 29536 consensus_queue.cc:237] T 0b3c514213bb40ca98955d6409a3b372 P 4f58800ba2ed45eabd81c2f2e691aacd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } attrs { replace: true } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: false } } peers { permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: false } }
I20240502 21:27:56.291690 24183 raft_consensus.cc:2454] T 0b3c514213bb40ca98955d6409a3b372 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4f58800ba2ed45eabd81c2f2e691aacd in term 2.
I20240502 21:27:56.296722 23766 catalog_manager.cc:5484] T 0b3c514213bb40ca98955d6409a3b372 P 4f58800ba2ed45eabd81c2f2e691aacd reported cstate change: term changed from 1 to 2, leader changed from 009d3e7e8a8b4e2497de3a73d01ced49 (127.15.255.131) to 4f58800ba2ed45eabd81c2f2e691aacd (127.15.255.137). New cstate: current_term: 2 leader_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:27:56.380646 24315 raft_consensus.cc:1261] T 30236241b670443489240f191d9b9d50 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 2 FOLLOWER]: Refusing update from remote peer a3e500d7077a48acb5edb33f7e178c81: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 21:27:56.381245 29292 consensus_queue.cc:1035] T 30236241b670443489240f191d9b9d50 P a3e500d7077a48acb5edb33f7e178c81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:27:56.418931 29559 ts_tablet_manager.cc:902] T fef55863beb6435dbb25bfb6d71e3cd7 P fb6004b8640a4906aa0cfe63e383d40e: Initiating tablet copy from peer 038c5d9d1d7a49c981c4c34fbc00d9e6 (127.15.255.132:46813)
I20240502 21:27:56.431742 29559 tablet_copy_client.cc:308] T fef55863beb6435dbb25bfb6d71e3cd7 P fb6004b8640a4906aa0cfe63e383d40e: tablet copy: Beginning tablet copy session from remote peer at address 127.15.255.132:46813
I20240502 21:27:56.432499 24338 tablet_copy_service.cc:140] P 038c5d9d1d7a49c981c4c34fbc00d9e6: Received BeginTabletCopySession request for tablet fef55863beb6435dbb25bfb6d71e3cd7 from peer fb6004b8640a4906aa0cfe63e383d40e ({username='slave'} at 127.15.255.134:51733)
I20240502 21:27:56.432670 24338 tablet_copy_service.cc:161] P 038c5d9d1d7a49c981c4c34fbc00d9e6: Beginning new tablet copy session on tablet fef55863beb6435dbb25bfb6d71e3cd7 from peer fb6004b8640a4906aa0cfe63e383d40e at {username='slave'} at 127.15.255.134:51733: session id = fb6004b8640a4906aa0cfe63e383d40e-fef55863beb6435dbb25bfb6d71e3cd7
I20240502 21:27:56.434959 24338 tablet_copy_source_session.cc:205] T fef55863beb6435dbb25bfb6d71e3cd7 P 038c5d9d1d7a49c981c4c34fbc00d9e6: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:27:56.436231 29559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fef55863beb6435dbb25bfb6d71e3cd7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:56.441448 29559 tablet_copy_client.cc:783] T fef55863beb6435dbb25bfb6d71e3cd7 P fb6004b8640a4906aa0cfe63e383d40e: tablet copy: Starting download of 0 data blocks...
I20240502 21:27:56.441658 29559 tablet_copy_client.cc:647] T fef55863beb6435dbb25bfb6d71e3cd7 P fb6004b8640a4906aa0cfe63e383d40e: tablet copy: Starting download of 1 WAL segments...
I20240502 21:27:56.443092 29572 ts_tablet_manager.cc:902] T d5697d8b06094211b261f144de9d46f8 P 4f58800ba2ed45eabd81c2f2e691aacd: Initiating tablet copy from peer 038c5d9d1d7a49c981c4c34fbc00d9e6 (127.15.255.132:46813)
I20240502 21:27:56.443222 29569 tablet_copy_client.cc:1002] T fef55863beb6435dbb25bfb6d71e3cd7 P fb6004b8640a4906aa0cfe63e383d40e: tablet copy: Injecting latency into file download: 1500
I20240502 21:27:56.444751 29572 tablet_copy_client.cc:308] T d5697d8b06094211b261f144de9d46f8 P 4f58800ba2ed45eabd81c2f2e691aacd: tablet copy: Beginning tablet copy session from remote peer at address 127.15.255.132:46813
I20240502 21:27:56.445574 24338 tablet_copy_service.cc:140] P 038c5d9d1d7a49c981c4c34fbc00d9e6: Received BeginTabletCopySession request for tablet d5697d8b06094211b261f144de9d46f8 from peer 4f58800ba2ed45eabd81c2f2e691aacd ({username='slave'} at 127.15.255.137:48505)
I20240502 21:27:56.445725 24338 tablet_copy_service.cc:161] P 038c5d9d1d7a49c981c4c34fbc00d9e6: Beginning new tablet copy session on tablet d5697d8b06094211b261f144de9d46f8 from peer 4f58800ba2ed45eabd81c2f2e691aacd at {username='slave'} at 127.15.255.137:48505: session id = 4f58800ba2ed45eabd81c2f2e691aacd-d5697d8b06094211b261f144de9d46f8
I20240502 21:27:56.447906 24338 tablet_copy_source_session.cc:205] T d5697d8b06094211b261f144de9d46f8 P 038c5d9d1d7a49c981c4c34fbc00d9e6: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:27:56.449270 29572 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d5697d8b06094211b261f144de9d46f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:56.454613 29572 tablet_copy_client.cc:783] T d5697d8b06094211b261f144de9d46f8 P 4f58800ba2ed45eabd81c2f2e691aacd: tablet copy: Starting download of 0 data blocks...
I20240502 21:27:56.454838 29572 tablet_copy_client.cc:647] T d5697d8b06094211b261f144de9d46f8 P 4f58800ba2ed45eabd81c2f2e691aacd: tablet copy: Starting download of 1 WAL segments...
I20240502 21:27:56.456602 29585 ts_tablet_manager.cc:902] T 8f14db150fda449e962be2466f1d2a0e P 4f58800ba2ed45eabd81c2f2e691aacd: Initiating tablet copy from peer 6810a866decb4f5089f7d35c870e4204 (127.15.255.129:40435)
I20240502 21:27:56.456939 29578 tablet_copy_client.cc:1002] T d5697d8b06094211b261f144de9d46f8 P 4f58800ba2ed45eabd81c2f2e691aacd: tablet copy: Injecting latency into file download: 1500
I20240502 21:27:56.459183 29585 tablet_copy_client.cc:308] T 8f14db150fda449e962be2466f1d2a0e P 4f58800ba2ed45eabd81c2f2e691aacd: tablet copy: Beginning tablet copy session from remote peer at address 127.15.255.129:40435
I20240502 21:27:56.465257 23935 tablet_copy_service.cc:140] P 6810a866decb4f5089f7d35c870e4204: Received BeginTabletCopySession request for tablet 8f14db150fda449e962be2466f1d2a0e from peer 4f58800ba2ed45eabd81c2f2e691aacd ({username='slave'} at 127.15.255.137:35119)
I20240502 21:27:56.465396 23935 tablet_copy_service.cc:161] P 6810a866decb4f5089f7d35c870e4204: Beginning new tablet copy session on tablet 8f14db150fda449e962be2466f1d2a0e from peer 4f58800ba2ed45eabd81c2f2e691aacd at {username='slave'} at 127.15.255.137:35119: session id = 4f58800ba2ed45eabd81c2f2e691aacd-8f14db150fda449e962be2466f1d2a0e
I20240502 21:27:56.467518 23935 tablet_copy_source_session.cc:205] T 8f14db150fda449e962be2466f1d2a0e P 6810a866decb4f5089f7d35c870e4204: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:27:56.468753 29585 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f14db150fda449e962be2466f1d2a0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:56.473064 29585 tablet_copy_client.cc:783] T 8f14db150fda449e962be2466f1d2a0e P 4f58800ba2ed45eabd81c2f2e691aacd: tablet copy: Starting download of 0 data blocks...
I20240502 21:27:56.473289 29585 tablet_copy_client.cc:647] T 8f14db150fda449e962be2466f1d2a0e P 4f58800ba2ed45eabd81c2f2e691aacd: tablet copy: Starting download of 1 WAL segments...
I20240502 21:27:56.474833 29589 tablet_copy_client.cc:1002] T 8f14db150fda449e962be2466f1d2a0e P 4f58800ba2ed45eabd81c2f2e691aacd: tablet copy: Injecting latency into file download: 1500
I20240502 21:27:56.518580 23916 raft_consensus.cc:1261] T 30236241b670443489240f191d9b9d50 P 6810a866decb4f5089f7d35c870e4204 [term 2 FOLLOWER]: Refusing update from remote peer a3e500d7077a48acb5edb33f7e178c81: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 21:27:56.519052 26012 raft_consensus.cc:1261] T 30236241b670443489240f191d9b9d50 P fb6004b8640a4906aa0cfe63e383d40e [term 2 FOLLOWER]: Refusing update from remote peer a3e500d7077a48acb5edb33f7e178c81: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 21:27:56.519176 29292 consensus_queue.cc:1035] T 30236241b670443489240f191d9b9d50 P a3e500d7077a48acb5edb33f7e178c81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:27:56.519809 29292 consensus_queue.cc:1035] T 30236241b670443489240f191d9b9d50 P a3e500d7077a48acb5edb33f7e178c81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:27:56.534068 26280 consensus_queue.cc:237] T 30236241b670443489240f191d9b9d50 P a3e500d7077a48acb5edb33f7e178c81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 8, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: false } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } }
I20240502 21:27:56.539405 26012 raft_consensus.cc:1261] T 30236241b670443489240f191d9b9d50 P fb6004b8640a4906aa0cfe63e383d40e [term 2 FOLLOWER]: Refusing update from remote peer a3e500d7077a48acb5edb33f7e178c81: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 21:27:56.539916 29301 consensus_queue.cc:1035] T 30236241b670443489240f191d9b9d50 P a3e500d7077a48acb5edb33f7e178c81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 21:27:56.542094 28935 raft_consensus.cc:2932] T 30236241b670443489240f191d9b9d50 P a3e500d7077a48acb5edb33f7e178c81 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 6810a866decb4f5089f7d35c870e4204 (127.15.255.129) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: false } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } } }
I20240502 21:27:56.545751 23756 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 30236241b670443489240f191d9b9d50 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:27:56.545935 26012 raft_consensus.cc:2932] T 30236241b670443489240f191d9b9d50 P fb6004b8640a4906aa0cfe63e383d40e [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 6810a866decb4f5089f7d35c870e4204 (127.15.255.129) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: false } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } } }
I20240502 21:27:56.546869 24316 raft_consensus.cc:1261] T 30236241b670443489240f191d9b9d50 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 2 FOLLOWER]: Refusing update from remote peer a3e500d7077a48acb5edb33f7e178c81: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 21:27:56.546963 23771 catalog_manager.cc:5484] T 30236241b670443489240f191d9b9d50 P a3e500d7077a48acb5edb33f7e178c81 reported cstate change: config changed from index 8 to 10, VOTER 6810a866decb4f5089f7d35c870e4204 (127.15.255.129) evicted. New cstate: current_term: 2 leader_uuid: "a3e500d7077a48acb5edb33f7e178c81" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:27:56.547586 29301 consensus_queue.cc:1035] T 30236241b670443489240f191d9b9d50 P a3e500d7077a48acb5edb33f7e178c81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 21:27:56.549188 24316 raft_consensus.cc:2932] T 30236241b670443489240f191d9b9d50 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 6810a866decb4f5089f7d35c870e4204 (127.15.255.129) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: false } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } } }
I20240502 21:27:56.551110 23894 tablet_service.cc:1514] Processing DeleteTablet for tablet 30236241b670443489240f191d9b9d50 with delete_type TABLET_DATA_TOMBSTONED (TS 6810a866decb4f5089f7d35c870e4204 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:33512
I20240502 21:27:56.551765 29597 tablet_replica.cc:330] T 30236241b670443489240f191d9b9d50 P 6810a866decb4f5089f7d35c870e4204: stopping tablet replica
I20240502 21:27:56.552043 29597 raft_consensus.cc:2229] T 30236241b670443489240f191d9b9d50 P 6810a866decb4f5089f7d35c870e4204 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:56.552327 29597 raft_consensus.cc:2258] T 30236241b670443489240f191d9b9d50 P 6810a866decb4f5089f7d35c870e4204 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:56.554221 29597 ts_tablet_manager.cc:1875] T 30236241b670443489240f191d9b9d50 P 6810a866decb4f5089f7d35c870e4204: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:27:56.559783 29597 ts_tablet_manager.cc:1888] T 30236241b670443489240f191d9b9d50 P 6810a866decb4f5089f7d35c870e4204: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.9
I20240502 21:27:56.559896 29597 log.cc:1196] T 30236241b670443489240f191d9b9d50 P 6810a866decb4f5089f7d35c870e4204: Deleting WAL directory at /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-0/wal/wals/30236241b670443489240f191d9b9d50
I20240502 21:27:56.560616 23757 catalog_manager.cc:4830] TS 6810a866decb4f5089f7d35c870e4204 (127.15.255.129:40435): tablet 30236241b670443489240f191d9b9d50 (table rebalance_test_table_4 [id=9d2b330e886e4e83a437506bf99d77a3]) successfully deleted
I20240502 21:27:56.622678 29598 ts_tablet_manager.cc:902] T 83b95f3ae7564b81b31a2f5673ee22bb P 4f58800ba2ed45eabd81c2f2e691aacd: Initiating tablet copy from peer 6810a866decb4f5089f7d35c870e4204 (127.15.255.129:40435)
I20240502 21:27:56.624577 29598 tablet_copy_client.cc:308] T 83b95f3ae7564b81b31a2f5673ee22bb P 4f58800ba2ed45eabd81c2f2e691aacd: tablet copy: Beginning tablet copy session from remote peer at address 127.15.255.129:40435
I20240502 21:27:56.625218 23935 tablet_copy_service.cc:140] P 6810a866decb4f5089f7d35c870e4204: Received BeginTabletCopySession request for tablet 83b95f3ae7564b81b31a2f5673ee22bb from peer 4f58800ba2ed45eabd81c2f2e691aacd ({username='slave'} at 127.15.255.137:35119)
I20240502 21:27:56.625380 23935 tablet_copy_service.cc:161] P 6810a866decb4f5089f7d35c870e4204: Beginning new tablet copy session on tablet 83b95f3ae7564b81b31a2f5673ee22bb from peer 4f58800ba2ed45eabd81c2f2e691aacd at {username='slave'} at 127.15.255.137:35119: session id = 4f58800ba2ed45eabd81c2f2e691aacd-83b95f3ae7564b81b31a2f5673ee22bb
I20240502 21:27:56.627272 23935 tablet_copy_source_session.cc:205] T 83b95f3ae7564b81b31a2f5673ee22bb P 6810a866decb4f5089f7d35c870e4204: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:27:56.628356 29598 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83b95f3ae7564b81b31a2f5673ee22bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:56.632766 29598 tablet_copy_client.cc:783] T 83b95f3ae7564b81b31a2f5673ee22bb P 4f58800ba2ed45eabd81c2f2e691aacd: tablet copy: Starting download of 0 data blocks...
I20240502 21:27:56.632979 29598 tablet_copy_client.cc:647] T 83b95f3ae7564b81b31a2f5673ee22bb P 4f58800ba2ed45eabd81c2f2e691aacd: tablet copy: Starting download of 1 WAL segments...
I20240502 21:27:56.634290 29602 tablet_copy_client.cc:1002] T 83b95f3ae7564b81b31a2f5673ee22bb P 4f58800ba2ed45eabd81c2f2e691aacd: tablet copy: Injecting latency into file download: 1500
I20240502 21:27:56.662505 24315 raft_consensus.cc:1261] T 52f621ea3f174490948eae854c706a92 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 2 FOLLOWER]: Refusing update from remote peer ee27c3f9e0ec46caa742ac386d6b7c20: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 21:27:56.663143 29015 consensus_queue.cc:1035] T 52f621ea3f174490948eae854c706a92 P ee27c3f9e0ec46caa742ac386d6b7c20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:27:56.679237 23916 raft_consensus.cc:1261] T 52f621ea3f174490948eae854c706a92 P 6810a866decb4f5089f7d35c870e4204 [term 2 FOLLOWER]: Refusing update from remote peer ee27c3f9e0ec46caa742ac386d6b7c20: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 21:27:56.679776 29015 consensus_queue.cc:1035] T 52f621ea3f174490948eae854c706a92 P ee27c3f9e0ec46caa742ac386d6b7c20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:27:56.685010 27125 raft_consensus.cc:1261] T 52f621ea3f174490948eae854c706a92 P 4f58800ba2ed45eabd81c2f2e691aacd [term 2 FOLLOWER]: Refusing update from remote peer ee27c3f9e0ec46caa742ac386d6b7c20: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 21:27:56.685611 29615 consensus_queue.cc:1035] T 52f621ea3f174490948eae854c706a92 P ee27c3f9e0ec46caa742ac386d6b7c20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:27:56.695881 24183 raft_consensus.cc:1261] T 0b3c514213bb40ca98955d6409a3b372 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 2 FOLLOWER]: Refusing update from remote peer 4f58800ba2ed45eabd81c2f2e691aacd: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 21:27:56.696488 29536 consensus_queue.cc:1035] T 0b3c514213bb40ca98955d6409a3b372 P 4f58800ba2ed45eabd81c2f2e691aacd [LEADER]: Connected to new peer: Peer: permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:27:56.720561 26012 raft_consensus.cc:1261] T 0b3c514213bb40ca98955d6409a3b372 P fb6004b8640a4906aa0cfe63e383d40e [term 2 FOLLOWER]: Refusing update from remote peer 4f58800ba2ed45eabd81c2f2e691aacd: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 21:27:56.721503 29536 consensus_queue.cc:1035] T 0b3c514213bb40ca98955d6409a3b372 P 4f58800ba2ed45eabd81c2f2e691aacd [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:27:56.742313 24316 raft_consensus.cc:1261] T 0b3c514213bb40ca98955d6409a3b372 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 2 FOLLOWER]: Refusing update from remote peer 4f58800ba2ed45eabd81c2f2e691aacd: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 21:27:56.744933 29536 consensus_queue.cc:1035] T 0b3c514213bb40ca98955d6409a3b372 P 4f58800ba2ed45eabd81c2f2e691aacd [LEADER]: Connected to new peer: Peer: permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:27:57.203233 25878 consensus_queue.cc:237] T 52f621ea3f174490948eae854c706a92 P ee27c3f9e0ec46caa742ac386d6b7c20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 8, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: false } } peers { permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: false } }
I20240502 21:27:57.205176 23915 raft_consensus.cc:1261] T 52f621ea3f174490948eae854c706a92 P 6810a866decb4f5089f7d35c870e4204 [term 2 FOLLOWER]: Refusing update from remote peer ee27c3f9e0ec46caa742ac386d6b7c20: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 21:27:57.205752 27125 raft_consensus.cc:1261] T 52f621ea3f174490948eae854c706a92 P 4f58800ba2ed45eabd81c2f2e691aacd [term 2 FOLLOWER]: Refusing update from remote peer ee27c3f9e0ec46caa742ac386d6b7c20: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 21:27:57.206133 29716 consensus_queue.cc:1035] T 52f621ea3f174490948eae854c706a92 P ee27c3f9e0ec46caa742ac386d6b7c20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 21:27:57.206496 29717 consensus_queue.cc:1035] T 52f621ea3f174490948eae854c706a92 P ee27c3f9e0ec46caa742ac386d6b7c20 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 21:27:57.208957 29615 raft_consensus.cc:2932] T 52f621ea3f174490948eae854c706a92 P ee27c3f9e0ec46caa742ac386d6b7c20 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 038c5d9d1d7a49c981c4c34fbc00d9e6 (127.15.255.132) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: false } } peers { permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: false } } }
I20240502 21:27:57.209430 27125 raft_consensus.cc:2932] T 52f621ea3f174490948eae854c706a92 P 4f58800ba2ed45eabd81c2f2e691aacd [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 038c5d9d1d7a49c981c4c34fbc00d9e6 (127.15.255.132) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: false } } peers { permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: false } } }
I20240502 21:27:57.209858 23915 raft_consensus.cc:2932] T 52f621ea3f174490948eae854c706a92 P 6810a866decb4f5089f7d35c870e4204 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 038c5d9d1d7a49c981c4c34fbc00d9e6 (127.15.255.132) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: false } } peers { permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: false } } }
I20240502 21:27:57.213059 23754 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 52f621ea3f174490948eae854c706a92 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:27:57.214605 23770 catalog_manager.cc:5484] T 52f621ea3f174490948eae854c706a92 P 4f58800ba2ed45eabd81c2f2e691aacd reported cstate change: config changed from index 8 to 10, VOTER 038c5d9d1d7a49c981c4c34fbc00d9e6 (127.15.255.132) evicted. New cstate: current_term: 2 leader_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: false } } peers { permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: false } } }
I20240502 21:27:57.219319 24297 tablet_service.cc:1514] Processing DeleteTablet for tablet 52f621ea3f174490948eae854c706a92 with delete_type TABLET_DATA_TOMBSTONED (TS 038c5d9d1d7a49c981c4c34fbc00d9e6 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:58268
I20240502 21:27:57.224395 29722 tablet_replica.cc:330] T 52f621ea3f174490948eae854c706a92 P 038c5d9d1d7a49c981c4c34fbc00d9e6: stopping tablet replica
I20240502 21:27:57.224606 29722 raft_consensus.cc:2229] T 52f621ea3f174490948eae854c706a92 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:57.224784 29722 raft_consensus.cc:2258] T 52f621ea3f174490948eae854c706a92 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:57.225529 29722 ts_tablet_manager.cc:1875] T 52f621ea3f174490948eae854c706a92 P 038c5d9d1d7a49c981c4c34fbc00d9e6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:27:57.230242 29722 ts_tablet_manager.cc:1888] T 52f621ea3f174490948eae854c706a92 P 038c5d9d1d7a49c981c4c34fbc00d9e6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.9
I20240502 21:27:57.230374 29722 log.cc:1196] T 52f621ea3f174490948eae854c706a92 P 038c5d9d1d7a49c981c4c34fbc00d9e6: Deleting WAL directory at /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-3/wal/wals/52f621ea3f174490948eae854c706a92
I20240502 21:27:57.231106 23755 catalog_manager.cc:4830] TS 038c5d9d1d7a49c981c4c34fbc00d9e6 (127.15.255.132:46813): tablet 52f621ea3f174490948eae854c706a92 (table rebalance_test_table_5 [id=e10e5576c0ea45a09c0eab75c2023b78]) successfully deleted
I20240502 21:27:57.368646 24047 consensus_queue.cc:237] T dc0e070513cc46ca82c1946423ad91c5 P 6c194a6db6c145acafc0f3068b8ce1fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } attrs { replace: true } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } } peers { permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: NON_VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: true } }
W20240502 21:27:57.371176 23981 consensus_peers.cc:485] T dc0e070513cc46ca82c1946423ad91c5 P 6c194a6db6c145acafc0f3068b8ce1fd -> Peer 4f58800ba2ed45eabd81c2f2e691aacd (127.15.255.137:40161): Couldn't send request to peer 4f58800ba2ed45eabd81c2f2e691aacd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: dc0e070513cc46ca82c1946423ad91c5. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:57.371205 26147 raft_consensus.cc:1261] T dc0e070513cc46ca82c1946423ad91c5 P e117afbf8819424e8a98476453dfde2a [term 1 FOLLOWER]: Refusing update from remote peer 6c194a6db6c145acafc0f3068b8ce1fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:27:57.371724 29237 consensus_queue.cc:1035] T dc0e070513cc46ca82c1946423ad91c5 P 6c194a6db6c145acafc0f3068b8ce1fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:27:57.374152 24183 raft_consensus.cc:1261] T dc0e070513cc46ca82c1946423ad91c5 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 1 FOLLOWER]: Refusing update from remote peer 6c194a6db6c145acafc0f3068b8ce1fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:27:57.374698 29237 consensus_queue.cc:1035] T dc0e070513cc46ca82c1946423ad91c5 P 6c194a6db6c145acafc0f3068b8ce1fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:27:57.376137 27125 consensus_queue.cc:237] T 0b3c514213bb40ca98955d6409a3b372 P 4f58800ba2ed45eabd81c2f2e691aacd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 8, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: false } } peers { permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: false } }
I20240502 21:27:57.378262 24183 raft_consensus.cc:2932] T dc0e070513cc46ca82c1946423ad91c5 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 4f58800ba2ed45eabd81c2f2e691aacd (127.15.255.137) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } attrs { replace: true } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } } peers { permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: NON_VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: true } } }
I20240502 21:27:57.378602 26012 raft_consensus.cc:1261] T 0b3c514213bb40ca98955d6409a3b372 P fb6004b8640a4906aa0cfe63e383d40e [term 2 FOLLOWER]: Refusing update from remote peer 4f58800ba2ed45eabd81c2f2e691aacd: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 21:27:57.379065 24316 raft_consensus.cc:1261] T 0b3c514213bb40ca98955d6409a3b372 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 2 FOLLOWER]: Refusing update from remote peer 4f58800ba2ed45eabd81c2f2e691aacd: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 21:27:57.379675 29760 consensus_queue.cc:1035] T 0b3c514213bb40ca98955d6409a3b372 P 4f58800ba2ed45eabd81c2f2e691aacd [LEADER]: Connected to new peer: Peer: permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 21:27:57.382361 29760 consensus_queue.cc:1035] T 0b3c514213bb40ca98955d6409a3b372 P 4f58800ba2ed45eabd81c2f2e691aacd [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 21:27:57.383158 29760 raft_consensus.cc:2932] T 0b3c514213bb40ca98955d6409a3b372 P 4f58800ba2ed45eabd81c2f2e691aacd [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 009d3e7e8a8b4e2497de3a73d01ced49 (127.15.255.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: false } } peers { permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: false } } }
I20240502 21:27:57.383895 23765 catalog_manager.cc:5484] T dc0e070513cc46ca82c1946423ad91c5 P 009d3e7e8a8b4e2497de3a73d01ced49 reported cstate change: config changed from index 6 to 7, NON_VOTER 4f58800ba2ed45eabd81c2f2e691aacd (127.15.255.137) added. New cstate: current_term: 1 leader_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } attrs { replace: true } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } } peers { permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: NON_VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: true } } }
I20240502 21:27:57.377578 29763 raft_consensus.cc:2932] T dc0e070513cc46ca82c1946423ad91c5 P 6c194a6db6c145acafc0f3068b8ce1fd [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 4f58800ba2ed45eabd81c2f2e691aacd (127.15.255.137) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } attrs { replace: true } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } } peers { permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: NON_VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: true } } }
I20240502 21:27:57.389266 26012 raft_consensus.cc:2932] T 0b3c514213bb40ca98955d6409a3b372 P fb6004b8640a4906aa0cfe63e383d40e [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 009d3e7e8a8b4e2497de3a73d01ced49 (127.15.255.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: false } } peers { permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: false } } }
I20240502 21:27:57.389557 24316 raft_consensus.cc:2932] T 0b3c514213bb40ca98955d6409a3b372 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 009d3e7e8a8b4e2497de3a73d01ced49 (127.15.255.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: false } } peers { permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: false } } }
I20240502 21:27:57.395540 23770 catalog_manager.cc:5484] T 0b3c514213bb40ca98955d6409a3b372 P 038c5d9d1d7a49c981c4c34fbc00d9e6 reported cstate change: config changed from index 8 to 10, VOTER 009d3e7e8a8b4e2497de3a73d01ced49 (127.15.255.131) evicted. New cstate: current_term: 2 leader_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: false } } peers { permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: false } } }
I20240502 21:27:57.402038 26147 raft_consensus.cc:2932] T dc0e070513cc46ca82c1946423ad91c5 P e117afbf8819424e8a98476453dfde2a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 4f58800ba2ed45eabd81c2f2e691aacd (127.15.255.137) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } attrs { replace: true } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } } peers { permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: NON_VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: true } } }
I20240502 21:27:57.407146 24162 tablet_service.cc:1514] Processing DeleteTablet for tablet 0b3c514213bb40ca98955d6409a3b372 with delete_type TABLET_DATA_TOMBSTONED (TS 009d3e7e8a8b4e2497de3a73d01ced49 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:33102
I20240502 21:27:57.415450 23755 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 0b3c514213bb40ca98955d6409a3b372 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:27:57.425092 29768 tablet_replica.cc:330] T 0b3c514213bb40ca98955d6409a3b372 P 009d3e7e8a8b4e2497de3a73d01ced49: stopping tablet replica
I20240502 21:27:57.425302 29768 raft_consensus.cc:2229] T 0b3c514213bb40ca98955d6409a3b372 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:57.425496 29768 raft_consensus.cc:2258] T 0b3c514213bb40ca98955d6409a3b372 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:57.426311 29768 ts_tablet_manager.cc:1875] T 0b3c514213bb40ca98955d6409a3b372 P 009d3e7e8a8b4e2497de3a73d01ced49: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:27:57.432919 29768 ts_tablet_manager.cc:1888] T 0b3c514213bb40ca98955d6409a3b372 P 009d3e7e8a8b4e2497de3a73d01ced49: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.9
I20240502 21:27:57.433048 29768 log.cc:1196] T 0b3c514213bb40ca98955d6409a3b372 P 009d3e7e8a8b4e2497de3a73d01ced49: Deleting WAL directory at /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-2/wal/wals/0b3c514213bb40ca98955d6409a3b372
I20240502 21:27:57.433807 23756 catalog_manager.cc:4830] TS 009d3e7e8a8b4e2497de3a73d01ced49 (127.15.255.131:37895): tablet 0b3c514213bb40ca98955d6409a3b372 (table rebalance_test_table_1 [id=e962de13fbc44d1986b1e1ee191d22e2]) successfully deleted
I20240502 21:27:57.490200 23915 consensus_queue.cc:237] T 256ae04a48b849c4a9086b6b1ff5df19 P 6810a866decb4f5089f7d35c870e4204 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } attrs { replace: true } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } } peers { permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: NON_VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: true } }
I20240502 21:27:57.492348 26147 raft_consensus.cc:1261] T 256ae04a48b849c4a9086b6b1ff5df19 P e117afbf8819424e8a98476453dfde2a [term 1 FOLLOWER]: Refusing update from remote peer 6810a866decb4f5089f7d35c870e4204: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240502 21:27:57.492506 23849 consensus_peers.cc:485] T 256ae04a48b849c4a9086b6b1ff5df19 P 6810a866decb4f5089f7d35c870e4204 -> Peer 4f58800ba2ed45eabd81c2f2e691aacd (127.15.255.137:40161): Couldn't send request to peer 4f58800ba2ed45eabd81c2f2e691aacd. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 256ae04a48b849c4a9086b6b1ff5df19. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:27:57.492954 29683 consensus_queue.cc:1035] T 256ae04a48b849c4a9086b6b1ff5df19 P 6810a866decb4f5089f7d35c870e4204 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:27:57.494110 24183 raft_consensus.cc:1261] T 256ae04a48b849c4a9086b6b1ff5df19 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 1 FOLLOWER]: Refusing update from remote peer 6810a866decb4f5089f7d35c870e4204: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:27:57.494664 27234 consensus_queue.cc:1035] T 256ae04a48b849c4a9086b6b1ff5df19 P 6810a866decb4f5089f7d35c870e4204 [LEADER]: Connected to new peer: Peer: permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:27:57.497161 29683 raft_consensus.cc:2932] T 256ae04a48b849c4a9086b6b1ff5df19 P 6810a866decb4f5089f7d35c870e4204 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 4f58800ba2ed45eabd81c2f2e691aacd (127.15.255.137) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } attrs { replace: true } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } } peers { permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: NON_VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: true } } }
I20240502 21:27:57.497831 24183 raft_consensus.cc:2932] T 256ae04a48b849c4a9086b6b1ff5df19 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 4f58800ba2ed45eabd81c2f2e691aacd (127.15.255.137) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } attrs { replace: true } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } } peers { permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: NON_VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: true } } }
I20240502 21:27:57.502852 23770 catalog_manager.cc:5484] T 256ae04a48b849c4a9086b6b1ff5df19 P 6810a866decb4f5089f7d35c870e4204 reported cstate change: config changed from index 6 to 7, NON_VOTER 4f58800ba2ed45eabd81c2f2e691aacd (127.15.255.137) added. New cstate: current_term: 1 leader_uuid: "6810a866decb4f5089f7d35c870e4204" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: NON_VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:57.503183 26147 raft_consensus.cc:2932] T 256ae04a48b849c4a9086b6b1ff5df19 P e117afbf8819424e8a98476453dfde2a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 4f58800ba2ed45eabd81c2f2e691aacd (127.15.255.137) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } attrs { replace: true } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } } peers { permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: NON_VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: true } } }
I20240502 21:27:57.802978 29860 ts_tablet_manager.cc:902] T dc0e070513cc46ca82c1946423ad91c5 P 4f58800ba2ed45eabd81c2f2e691aacd: Initiating tablet copy from peer 6c194a6db6c145acafc0f3068b8ce1fd (127.15.255.130:41235)
I20240502 21:27:57.805454 29860 tablet_copy_client.cc:308] T dc0e070513cc46ca82c1946423ad91c5 P 4f58800ba2ed45eabd81c2f2e691aacd: tablet copy: Beginning tablet copy session from remote peer at address 127.15.255.130:41235
I20240502 21:27:57.806874 24070 tablet_copy_service.cc:140] P 6c194a6db6c145acafc0f3068b8ce1fd: Received BeginTabletCopySession request for tablet dc0e070513cc46ca82c1946423ad91c5 from peer 4f58800ba2ed45eabd81c2f2e691aacd ({username='slave'} at 127.15.255.137:60133)
I20240502 21:27:57.807037 24070 tablet_copy_service.cc:161] P 6c194a6db6c145acafc0f3068b8ce1fd: Beginning new tablet copy session on tablet dc0e070513cc46ca82c1946423ad91c5 from peer 4f58800ba2ed45eabd81c2f2e691aacd at {username='slave'} at 127.15.255.137:60133: session id = 4f58800ba2ed45eabd81c2f2e691aacd-dc0e070513cc46ca82c1946423ad91c5
I20240502 21:27:57.809126 24070 tablet_copy_source_session.cc:205] T dc0e070513cc46ca82c1946423ad91c5 P 6c194a6db6c145acafc0f3068b8ce1fd: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:27:57.810485 29860 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc0e070513cc46ca82c1946423ad91c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:57.815197 29860 tablet_copy_client.cc:783] T dc0e070513cc46ca82c1946423ad91c5 P 4f58800ba2ed45eabd81c2f2e691aacd: tablet copy: Starting download of 0 data blocks...
I20240502 21:27:57.815496 29860 tablet_copy_client.cc:647] T dc0e070513cc46ca82c1946423ad91c5 P 4f58800ba2ed45eabd81c2f2e691aacd: tablet copy: Starting download of 1 WAL segments...
I20240502 21:27:57.817045 29863 tablet_copy_client.cc:1002] T dc0e070513cc46ca82c1946423ad91c5 P 4f58800ba2ed45eabd81c2f2e691aacd: tablet copy: Injecting latency into file download: 1500
I20240502 21:27:57.890162 29875 ts_tablet_manager.cc:902] T 256ae04a48b849c4a9086b6b1ff5df19 P 4f58800ba2ed45eabd81c2f2e691aacd: Initiating tablet copy from peer 6810a866decb4f5089f7d35c870e4204 (127.15.255.129:40435)
I20240502 21:27:57.891464 29875 tablet_copy_client.cc:308] T 256ae04a48b849c4a9086b6b1ff5df19 P 4f58800ba2ed45eabd81c2f2e691aacd: tablet copy: Beginning tablet copy session from remote peer at address 127.15.255.129:40435
I20240502 21:27:57.892153 23935 tablet_copy_service.cc:140] P 6810a866decb4f5089f7d35c870e4204: Received BeginTabletCopySession request for tablet 256ae04a48b849c4a9086b6b1ff5df19 from peer 4f58800ba2ed45eabd81c2f2e691aacd ({username='slave'} at 127.15.255.137:35119)
I20240502 21:27:57.892306 23935 tablet_copy_service.cc:161] P 6810a866decb4f5089f7d35c870e4204: Beginning new tablet copy session on tablet 256ae04a48b849c4a9086b6b1ff5df19 from peer 4f58800ba2ed45eabd81c2f2e691aacd at {username='slave'} at 127.15.255.137:35119: session id = 4f58800ba2ed45eabd81c2f2e691aacd-256ae04a48b849c4a9086b6b1ff5df19
I20240502 21:27:57.894361 23935 tablet_copy_source_session.cc:205] T 256ae04a48b849c4a9086b6b1ff5df19 P 6810a866decb4f5089f7d35c870e4204: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:27:57.895449 29875 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 256ae04a48b849c4a9086b6b1ff5df19. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:27:57.899612 29875 tablet_copy_client.cc:783] T 256ae04a48b849c4a9086b6b1ff5df19 P 4f58800ba2ed45eabd81c2f2e691aacd: tablet copy: Starting download of 0 data blocks...
I20240502 21:27:57.899809 29875 tablet_copy_client.cc:647] T 256ae04a48b849c4a9086b6b1ff5df19 P 4f58800ba2ed45eabd81c2f2e691aacd: tablet copy: Starting download of 1 WAL segments...
I20240502 21:27:57.901226 29879 tablet_copy_client.cc:1002] T 256ae04a48b849c4a9086b6b1ff5df19 P 4f58800ba2ed45eabd81c2f2e691aacd: tablet copy: Injecting latency into file download: 1500
I20240502 21:27:57.943634 29559 tablet_copy_client.cc:523] T fef55863beb6435dbb25bfb6d71e3cd7 P fb6004b8640a4906aa0cfe63e383d40e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:27:57.946296 29559 tablet_bootstrap.cc:492] T fef55863beb6435dbb25bfb6d71e3cd7 P fb6004b8640a4906aa0cfe63e383d40e: Bootstrap starting.
I20240502 21:27:57.957304 29572 tablet_copy_client.cc:523] T d5697d8b06094211b261f144de9d46f8 P 4f58800ba2ed45eabd81c2f2e691aacd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:27:57.959810 29572 tablet_bootstrap.cc:492] T d5697d8b06094211b261f144de9d46f8 P 4f58800ba2ed45eabd81c2f2e691aacd: Bootstrap starting.
I20240502 21:27:57.975186 29585 tablet_copy_client.cc:523] T 8f14db150fda449e962be2466f1d2a0e P 4f58800ba2ed45eabd81c2f2e691aacd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:27:57.978849 29585 tablet_bootstrap.cc:492] T 8f14db150fda449e962be2466f1d2a0e P 4f58800ba2ed45eabd81c2f2e691aacd: Bootstrap starting.
I20240502 21:27:58.005625 29572 tablet_bootstrap.cc:492] T d5697d8b06094211b261f144de9d46f8 P 4f58800ba2ed45eabd81c2f2e691aacd: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:27:58.006110 29572 tablet_bootstrap.cc:492] T d5697d8b06094211b261f144de9d46f8 P 4f58800ba2ed45eabd81c2f2e691aacd: Bootstrap complete.
I20240502 21:27:58.006412 29572 ts_tablet_manager.cc:1367] T d5697d8b06094211b261f144de9d46f8 P 4f58800ba2ed45eabd81c2f2e691aacd: Time spent bootstrapping tablet: real 0.047s user 0.043s sys 0.000s
I20240502 21:27:58.007323 29572 raft_consensus.cc:348] T d5697d8b06094211b261f144de9d46f8 P 4f58800ba2ed45eabd81c2f2e691aacd [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } attrs { replace: true } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } } peers { permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: NON_VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: true } }
I20240502 21:27:58.007616 29572 raft_consensus.cc:729] T d5697d8b06094211b261f144de9d46f8 P 4f58800ba2ed45eabd81c2f2e691aacd [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4f58800ba2ed45eabd81c2f2e691aacd, State: Initialized, Role: LEARNER
I20240502 21:27:58.007823 29572 consensus_queue.cc:260] T d5697d8b06094211b261f144de9d46f8 P 4f58800ba2ed45eabd81c2f2e691aacd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } attrs { replace: true } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } } peers { permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: NON_VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: true } }
I20240502 21:27:58.008553 29572 ts_tablet_manager.cc:1398] T d5697d8b06094211b261f144de9d46f8 P 4f58800ba2ed45eabd81c2f2e691aacd: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240502 21:27:58.009439 24338 tablet_copy_service.cc:342] P 038c5d9d1d7a49c981c4c34fbc00d9e6: Request end of tablet copy session 4f58800ba2ed45eabd81c2f2e691aacd-d5697d8b06094211b261f144de9d46f8 received from {username='slave'} at 127.15.255.137:48505
I20240502 21:27:58.009588 24338 tablet_copy_service.cc:433] P 038c5d9d1d7a49c981c4c34fbc00d9e6: ending tablet copy session 4f58800ba2ed45eabd81c2f2e691aacd-d5697d8b06094211b261f144de9d46f8 on tablet d5697d8b06094211b261f144de9d46f8 with peer 4f58800ba2ed45eabd81c2f2e691aacd
I20240502 21:27:58.023600 29559 tablet_bootstrap.cc:492] T fef55863beb6435dbb25bfb6d71e3cd7 P fb6004b8640a4906aa0cfe63e383d40e: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:27:58.023972 29559 tablet_bootstrap.cc:492] T fef55863beb6435dbb25bfb6d71e3cd7 P fb6004b8640a4906aa0cfe63e383d40e: Bootstrap complete.
I20240502 21:27:58.024223 29559 ts_tablet_manager.cc:1367] T fef55863beb6435dbb25bfb6d71e3cd7 P fb6004b8640a4906aa0cfe63e383d40e: Time spent bootstrapping tablet: real 0.078s user 0.039s sys 0.003s
I20240502 21:27:58.024907 29559 raft_consensus.cc:348] T fef55863beb6435dbb25bfb6d71e3cd7 P fb6004b8640a4906aa0cfe63e383d40e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } attrs { replace: true } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: NON_VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: true } }
I20240502 21:27:58.025141 29559 raft_consensus.cc:729] T fef55863beb6435dbb25bfb6d71e3cd7 P fb6004b8640a4906aa0cfe63e383d40e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: fb6004b8640a4906aa0cfe63e383d40e, State: Initialized, Role: LEARNER
I20240502 21:27:58.025336 29559 consensus_queue.cc:260] T fef55863beb6435dbb25bfb6d71e3cd7 P fb6004b8640a4906aa0cfe63e383d40e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } attrs { replace: true } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: NON_VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: true } }
I20240502 21:27:58.026324 29559 ts_tablet_manager.cc:1398] T fef55863beb6435dbb25bfb6d71e3cd7 P fb6004b8640a4906aa0cfe63e383d40e: Time spent starting tablet: real 0.002s user 0.001s sys 0.004s
I20240502 21:27:58.027370 24338 tablet_copy_service.cc:342] P 038c5d9d1d7a49c981c4c34fbc00d9e6: Request end of tablet copy session fb6004b8640a4906aa0cfe63e383d40e-fef55863beb6435dbb25bfb6d71e3cd7 received from {username='slave'} at 127.15.255.134:51733
I20240502 21:27:58.027541 24338 tablet_copy_service.cc:433] P 038c5d9d1d7a49c981c4c34fbc00d9e6: ending tablet copy session fb6004b8640a4906aa0cfe63e383d40e-fef55863beb6435dbb25bfb6d71e3cd7 on tablet fef55863beb6435dbb25bfb6d71e3cd7 with peer fb6004b8640a4906aa0cfe63e383d40e
I20240502 21:27:58.028656 29585 tablet_bootstrap.cc:492] T 8f14db150fda449e962be2466f1d2a0e P 4f58800ba2ed45eabd81c2f2e691aacd: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:27:58.029057 29585 tablet_bootstrap.cc:492] T 8f14db150fda449e962be2466f1d2a0e P 4f58800ba2ed45eabd81c2f2e691aacd: Bootstrap complete.
I20240502 21:27:58.029311 29585 ts_tablet_manager.cc:1367] T 8f14db150fda449e962be2466f1d2a0e P 4f58800ba2ed45eabd81c2f2e691aacd: Time spent bootstrapping tablet: real 0.051s user 0.036s sys 0.008s
I20240502 21:27:58.029934 29585 raft_consensus.cc:348] T 8f14db150fda449e962be2466f1d2a0e P 4f58800ba2ed45eabd81c2f2e691aacd [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } attrs { replace: true } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: false } } peers { permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: NON_VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: true } }
I20240502 21:27:58.030203 29585 raft_consensus.cc:729] T 8f14db150fda449e962be2466f1d2a0e P 4f58800ba2ed45eabd81c2f2e691aacd [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4f58800ba2ed45eabd81c2f2e691aacd, State: Initialized, Role: LEARNER
I20240502 21:27:58.030383 29585 consensus_queue.cc:260] T 8f14db150fda449e962be2466f1d2a0e P 4f58800ba2ed45eabd81c2f2e691aacd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } attrs { replace: true } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: false } } peers { permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: NON_VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: true } }
I20240502 21:27:58.030894 29585 ts_tablet_manager.cc:1398] T 8f14db150fda449e962be2466f1d2a0e P 4f58800ba2ed45eabd81c2f2e691aacd: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 21:27:58.031716 23935 tablet_copy_service.cc:342] P 6810a866decb4f5089f7d35c870e4204: Request end of tablet copy session 4f58800ba2ed45eabd81c2f2e691aacd-8f14db150fda449e962be2466f1d2a0e received from {username='slave'} at 127.15.255.137:35119
I20240502 21:27:58.031867 23935 tablet_copy_service.cc:433] P 6810a866decb4f5089f7d35c870e4204: ending tablet copy session 4f58800ba2ed45eabd81c2f2e691aacd-8f14db150fda449e962be2466f1d2a0e on tablet 8f14db150fda449e962be2466f1d2a0e with peer 4f58800ba2ed45eabd81c2f2e691aacd
I20240502 21:27:58.134673 29598 tablet_copy_client.cc:523] T 83b95f3ae7564b81b31a2f5673ee22bb P 4f58800ba2ed45eabd81c2f2e691aacd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:27:58.137312 29598 tablet_bootstrap.cc:492] T 83b95f3ae7564b81b31a2f5673ee22bb P 4f58800ba2ed45eabd81c2f2e691aacd: Bootstrap starting.
I20240502 21:27:58.169770 29598 tablet_bootstrap.cc:492] T 83b95f3ae7564b81b31a2f5673ee22bb P 4f58800ba2ed45eabd81c2f2e691aacd: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:27:58.170193 29598 tablet_bootstrap.cc:492] T 83b95f3ae7564b81b31a2f5673ee22bb P 4f58800ba2ed45eabd81c2f2e691aacd: Bootstrap complete.
I20240502 21:27:58.170459 29598 ts_tablet_manager.cc:1367] T 83b95f3ae7564b81b31a2f5673ee22bb P 4f58800ba2ed45eabd81c2f2e691aacd: Time spent bootstrapping tablet: real 0.033s user 0.032s sys 0.000s
I20240502 21:27:58.171247 29598 raft_consensus.cc:348] T 83b95f3ae7564b81b31a2f5673ee22bb P 4f58800ba2ed45eabd81c2f2e691aacd [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } attrs { replace: true } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } } peers { permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: NON_VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: true } }
I20240502 21:27:58.171504 29598 raft_consensus.cc:729] T 83b95f3ae7564b81b31a2f5673ee22bb P 4f58800ba2ed45eabd81c2f2e691aacd [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4f58800ba2ed45eabd81c2f2e691aacd, State: Initialized, Role: LEARNER
I20240502 21:27:58.171698 29598 consensus_queue.cc:260] T 83b95f3ae7564b81b31a2f5673ee22bb P 4f58800ba2ed45eabd81c2f2e691aacd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } attrs { replace: true } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } } peers { permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: NON_VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: true } }
I20240502 21:27:58.172197 29598 ts_tablet_manager.cc:1398] T 83b95f3ae7564b81b31a2f5673ee22bb P 4f58800ba2ed45eabd81c2f2e691aacd: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 21:27:58.173048 23935 tablet_copy_service.cc:342] P 6810a866decb4f5089f7d35c870e4204: Request end of tablet copy session 4f58800ba2ed45eabd81c2f2e691aacd-83b95f3ae7564b81b31a2f5673ee22bb received from {username='slave'} at 127.15.255.137:35119
I20240502 21:27:58.173206 23935 tablet_copy_service.cc:433] P 6810a866decb4f5089f7d35c870e4204: ending tablet copy session 4f58800ba2ed45eabd81c2f2e691aacd-83b95f3ae7564b81b31a2f5673ee22bb on tablet 83b95f3ae7564b81b31a2f5673ee22bb with peer 4f58800ba2ed45eabd81c2f2e691aacd
I20240502 21:27:58.237403 26012 raft_consensus.cc:1203] T fef55863beb6435dbb25bfb6d71e3cd7 P fb6004b8640a4906aa0cfe63e383d40e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240502 21:27:58.329367 27125 raft_consensus.cc:1203] T 83b95f3ae7564b81b31a2f5673ee22bb P 4f58800ba2ed45eabd81c2f2e691aacd [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240502 21:27:58.436527 27125 raft_consensus.cc:1203] T 8f14db150fda449e962be2466f1d2a0e P 4f58800ba2ed45eabd81c2f2e691aacd [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240502 21:27:58.518296 27125 raft_consensus.cc:1203] T d5697d8b06094211b261f144de9d46f8 P 4f58800ba2ed45eabd81c2f2e691aacd [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240502 21:27:58.703115 29983 raft_consensus.cc:1050] T fef55863beb6435dbb25bfb6d71e3cd7 P 038c5d9d1d7a49c981c4c34fbc00d9e6: attempting to promote NON_VOTER fb6004b8640a4906aa0cfe63e383d40e to VOTER
I20240502 21:27:58.703882 29983 consensus_queue.cc:237] T fef55863beb6435dbb25bfb6d71e3cd7 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } attrs { replace: true } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: false } }
I20240502 21:27:58.706323 26012 raft_consensus.cc:1261] T fef55863beb6435dbb25bfb6d71e3cd7 P fb6004b8640a4906aa0cfe63e383d40e [term 1 LEARNER]: Refusing update from remote peer 038c5d9d1d7a49c981c4c34fbc00d9e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:27:58.706632 23916 raft_consensus.cc:1261] T fef55863beb6435dbb25bfb6d71e3cd7 P 6810a866decb4f5089f7d35c870e4204 [term 1 FOLLOWER]: Refusing update from remote peer 038c5d9d1d7a49c981c4c34fbc00d9e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:27:58.706728 26280 raft_consensus.cc:1261] T fef55863beb6435dbb25bfb6d71e3cd7 P a3e500d7077a48acb5edb33f7e178c81 [term 1 FOLLOWER]: Refusing update from remote peer 038c5d9d1d7a49c981c4c34fbc00d9e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:27:58.706883 29985 consensus_queue.cc:1035] T fef55863beb6435dbb25bfb6d71e3cd7 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:27:58.707262 29983 consensus_queue.cc:1035] T fef55863beb6435dbb25bfb6d71e3cd7 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:27:58.707609 29265 consensus_queue.cc:1035] T fef55863beb6435dbb25bfb6d71e3cd7 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:27:58.722469 29265 raft_consensus.cc:2932] T fef55863beb6435dbb25bfb6d71e3cd7 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, fb6004b8640a4906aa0cfe63e383d40e (127.15.255.134) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } attrs { replace: true } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: false } } }
I20240502 21:27:58.723147 26280 raft_consensus.cc:2932] T fef55863beb6435dbb25bfb6d71e3cd7 P a3e500d7077a48acb5edb33f7e178c81 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, fb6004b8640a4906aa0cfe63e383d40e (127.15.255.134) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } attrs { replace: true } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: false } } }
I20240502 21:27:58.727746 23770 catalog_manager.cc:5484] T fef55863beb6435dbb25bfb6d71e3cd7 P 038c5d9d1d7a49c981c4c34fbc00d9e6 reported cstate change: config changed from index 7 to 8, fb6004b8640a4906aa0cfe63e383d40e (127.15.255.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:58.730525 23916 raft_consensus.cc:2932] T fef55863beb6435dbb25bfb6d71e3cd7 P 6810a866decb4f5089f7d35c870e4204 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, fb6004b8640a4906aa0cfe63e383d40e (127.15.255.134) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } attrs { replace: true } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: false } } }
I20240502 21:27:58.732774 26012 raft_consensus.cc:2932] T fef55863beb6435dbb25bfb6d71e3cd7 P fb6004b8640a4906aa0cfe63e383d40e [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, fb6004b8640a4906aa0cfe63e383d40e (127.15.255.134) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } attrs { replace: true } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: false } } }
I20240502 21:27:58.742080 24316 consensus_queue.cc:237] T fef55863beb6435dbb25bfb6d71e3cd7 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: false } }
I20240502 21:27:58.743753 26012 raft_consensus.cc:1261] T fef55863beb6435dbb25bfb6d71e3cd7 P fb6004b8640a4906aa0cfe63e383d40e [term 1 FOLLOWER]: Refusing update from remote peer 038c5d9d1d7a49c981c4c34fbc00d9e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:27:58.744006 26280 raft_consensus.cc:1261] T fef55863beb6435dbb25bfb6d71e3cd7 P a3e500d7077a48acb5edb33f7e178c81 [term 1 FOLLOWER]: Refusing update from remote peer 038c5d9d1d7a49c981c4c34fbc00d9e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:27:58.744246 29985 consensus_queue.cc:1035] T fef55863beb6435dbb25bfb6d71e3cd7 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:27:58.744565 29265 consensus_queue.cc:1035] T fef55863beb6435dbb25bfb6d71e3cd7 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:27:58.746834 29949 raft_consensus.cc:2932] T fef55863beb6435dbb25bfb6d71e3cd7 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 6810a866decb4f5089f7d35c870e4204 (127.15.255.129) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: false } } }
I20240502 21:27:58.747500 26012 raft_consensus.cc:2932] T fef55863beb6435dbb25bfb6d71e3cd7 P fb6004b8640a4906aa0cfe63e383d40e [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 6810a866decb4f5089f7d35c870e4204 (127.15.255.129) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: false } } }
I20240502 21:27:58.747501 26279 raft_consensus.cc:2932] T fef55863beb6435dbb25bfb6d71e3cd7 P a3e500d7077a48acb5edb33f7e178c81 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 6810a866decb4f5089f7d35c870e4204 (127.15.255.129) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: false } } }
I20240502 21:27:58.751077 23755 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet fef55863beb6435dbb25bfb6d71e3cd7 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:27:58.751982 23771 catalog_manager.cc:5484] T fef55863beb6435dbb25bfb6d71e3cd7 P 038c5d9d1d7a49c981c4c34fbc00d9e6 reported cstate change: config changed from index 8 to 9, VOTER 6810a866decb4f5089f7d35c870e4204 (127.15.255.129) evicted. New cstate: current_term: 1 leader_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:27:58.756361 23894 tablet_service.cc:1514] Processing DeleteTablet for tablet fef55863beb6435dbb25bfb6d71e3cd7 with delete_type TABLET_DATA_TOMBSTONED (TS 6810a866decb4f5089f7d35c870e4204 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:33512
I20240502 21:27:58.757213 29993 tablet_replica.cc:330] T fef55863beb6435dbb25bfb6d71e3cd7 P 6810a866decb4f5089f7d35c870e4204: stopping tablet replica
I20240502 21:27:58.757426 29993 raft_consensus.cc:2229] T fef55863beb6435dbb25bfb6d71e3cd7 P 6810a866decb4f5089f7d35c870e4204 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:58.757759 29993 raft_consensus.cc:2258] T fef55863beb6435dbb25bfb6d71e3cd7 P 6810a866decb4f5089f7d35c870e4204 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:58.758646 29993 ts_tablet_manager.cc:1875] T fef55863beb6435dbb25bfb6d71e3cd7 P 6810a866decb4f5089f7d35c870e4204: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:27:58.763106 29993 ts_tablet_manager.cc:1888] T fef55863beb6435dbb25bfb6d71e3cd7 P 6810a866decb4f5089f7d35c870e4204: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240502 21:27:58.763212 29993 log.cc:1196] T fef55863beb6435dbb25bfb6d71e3cd7 P 6810a866decb4f5089f7d35c870e4204: Deleting WAL directory at /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-0/wal/wals/fef55863beb6435dbb25bfb6d71e3cd7
I20240502 21:27:58.763923 23757 catalog_manager.cc:4830] TS 6810a866decb4f5089f7d35c870e4204 (127.15.255.129:40435): tablet fef55863beb6435dbb25bfb6d71e3cd7 (table rebalance_test_table_8 [id=9a4e7bd31db34cf3aff2242876f0de47]) successfully deleted
I20240502 21:27:58.772692 29951 raft_consensus.cc:1050] T 83b95f3ae7564b81b31a2f5673ee22bb P 6810a866decb4f5089f7d35c870e4204: attempting to promote NON_VOTER 4f58800ba2ed45eabd81c2f2e691aacd to VOTER
I20240502 21:27:58.773444 29951 consensus_queue.cc:237] T 83b95f3ae7564b81b31a2f5673ee22bb P 6810a866decb4f5089f7d35c870e4204 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } attrs { replace: true } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } } peers { permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: false } }
I20240502 21:27:58.775841 27125 raft_consensus.cc:1261] T 83b95f3ae7564b81b31a2f5673ee22bb P 4f58800ba2ed45eabd81c2f2e691aacd [term 1 LEARNER]: Refusing update from remote peer 6810a866decb4f5089f7d35c870e4204: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:27:58.776023 26147 raft_consensus.cc:1261] T 83b95f3ae7564b81b31a2f5673ee22bb P e117afbf8819424e8a98476453dfde2a [term 1 FOLLOWER]: Refusing update from remote peer 6810a866decb4f5089f7d35c870e4204: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:27:58.776031 24316 raft_consensus.cc:1261] T 83b95f3ae7564b81b31a2f5673ee22bb P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 1 FOLLOWER]: Refusing update from remote peer 6810a866decb4f5089f7d35c870e4204: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:27:58.776458 29951 consensus_queue.cc:1035] T 83b95f3ae7564b81b31a2f5673ee22bb P 6810a866decb4f5089f7d35c870e4204 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:27:58.776888 29995 consensus_queue.cc:1035] T 83b95f3ae7564b81b31a2f5673ee22bb P 6810a866decb4f5089f7d35c870e4204 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:27:58.777305 29682 consensus_queue.cc:1035] T 83b95f3ae7564b81b31a2f5673ee22bb P 6810a866decb4f5089f7d35c870e4204 [LEADER]: Connected to new peer: Peer: permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:27:58.780613 29682 raft_consensus.cc:2932] T 83b95f3ae7564b81b31a2f5673ee22bb P 6810a866decb4f5089f7d35c870e4204 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 4f58800ba2ed45eabd81c2f2e691aacd (127.15.255.137) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } attrs { replace: true } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } } peers { permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: false } } }
I20240502 21:27:58.782131 27125 raft_consensus.cc:2932] T 83b95f3ae7564b81b31a2f5673ee22bb P 4f58800ba2ed45eabd81c2f2e691aacd [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 4f58800ba2ed45eabd81c2f2e691aacd (127.15.255.137) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } attrs { replace: true } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } } peers { permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: false } } }
I20240502 21:27:58.784829 26147 raft_consensus.cc:2932] T 83b95f3ae7564b81b31a2f5673ee22bb P e117afbf8819424e8a98476453dfde2a [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 4f58800ba2ed45eabd81c2f2e691aacd (127.15.255.137) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } attrs { replace: true } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } } peers { permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: false } } }
I20240502 21:27:58.786099 23771 catalog_manager.cc:5484] T 83b95f3ae7564b81b31a2f5673ee22bb P 6810a866decb4f5089f7d35c870e4204 reported cstate change: config changed from index 7 to 8, 4f58800ba2ed45eabd81c2f2e691aacd (127.15.255.137) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "6810a866decb4f5089f7d35c870e4204" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:27:58.786262 24316 raft_consensus.cc:2932] T 83b95f3ae7564b81b31a2f5673ee22bb P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 4f58800ba2ed45eabd81c2f2e691aacd (127.15.255.137) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } attrs { replace: true } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } } peers { permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: false } } }
I20240502 21:27:58.794447 23916 consensus_queue.cc:237] T 83b95f3ae7564b81b31a2f5673ee22bb P 6810a866decb4f5089f7d35c870e4204 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } } peers { permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: false } }
I20240502 21:27:58.796093 26147 raft_consensus.cc:1261] T 83b95f3ae7564b81b31a2f5673ee22bb P e117afbf8819424e8a98476453dfde2a [term 1 FOLLOWER]: Refusing update from remote peer 6810a866decb4f5089f7d35c870e4204: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:27:58.796610 27125 raft_consensus.cc:1261] T 83b95f3ae7564b81b31a2f5673ee22bb P 4f58800ba2ed45eabd81c2f2e691aacd [term 1 FOLLOWER]: Refusing update from remote peer 6810a866decb4f5089f7d35c870e4204: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:27:58.796649 29682 consensus_queue.cc:1035] T 83b95f3ae7564b81b31a2f5673ee22bb P 6810a866decb4f5089f7d35c870e4204 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:27:58.797114 29951 consensus_queue.cc:1035] T 83b95f3ae7564b81b31a2f5673ee22bb P 6810a866decb4f5089f7d35c870e4204 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:27:58.798749 29682 raft_consensus.cc:2932] T 83b95f3ae7564b81b31a2f5673ee22bb P 6810a866decb4f5089f7d35c870e4204 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 038c5d9d1d7a49c981c4c34fbc00d9e6 (127.15.255.132) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } } peers { permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: false } } }
I20240502 21:27:58.799371 26147 raft_consensus.cc:2932] T 83b95f3ae7564b81b31a2f5673ee22bb P e117afbf8819424e8a98476453dfde2a [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 038c5d9d1d7a49c981c4c34fbc00d9e6 (127.15.255.132) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } } peers { permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: false } } }
I20240502 21:27:58.800237 27125 raft_consensus.cc:2932] T 83b95f3ae7564b81b31a2f5673ee22bb P 4f58800ba2ed45eabd81c2f2e691aacd [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 038c5d9d1d7a49c981c4c34fbc00d9e6 (127.15.255.132) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } } peers { permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: false } } }
I20240502 21:27:58.802950 23757 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 83b95f3ae7564b81b31a2f5673ee22bb with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:27:58.804147 23765 catalog_manager.cc:5484] T 83b95f3ae7564b81b31a2f5673ee22bb P 6810a866decb4f5089f7d35c870e4204 reported cstate change: config changed from index 8 to 9, VOTER 038c5d9d1d7a49c981c4c34fbc00d9e6 (127.15.255.132) evicted. New cstate: current_term: 1 leader_uuid: "6810a866decb4f5089f7d35c870e4204" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:27:58.808161 24297 tablet_service.cc:1514] Processing DeleteTablet for tablet 83b95f3ae7564b81b31a2f5673ee22bb with delete_type TABLET_DATA_TOMBSTONED (TS 038c5d9d1d7a49c981c4c34fbc00d9e6 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:58268
I20240502 21:27:58.812903 30000 tablet_replica.cc:330] T 83b95f3ae7564b81b31a2f5673ee22bb P 038c5d9d1d7a49c981c4c34fbc00d9e6: stopping tablet replica
I20240502 21:27:58.813088 30000 raft_consensus.cc:2229] T 83b95f3ae7564b81b31a2f5673ee22bb P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:58.813457 30000 raft_consensus.cc:2258] T 83b95f3ae7564b81b31a2f5673ee22bb P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:58.814319 30000 ts_tablet_manager.cc:1875] T 83b95f3ae7564b81b31a2f5673ee22bb P 038c5d9d1d7a49c981c4c34fbc00d9e6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:27:58.818742 30000 ts_tablet_manager.cc:1888] T 83b95f3ae7564b81b31a2f5673ee22bb P 038c5d9d1d7a49c981c4c34fbc00d9e6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240502 21:27:58.818850 30000 log.cc:1196] T 83b95f3ae7564b81b31a2f5673ee22bb P 038c5d9d1d7a49c981c4c34fbc00d9e6: Deleting WAL directory at /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-3/wal/wals/83b95f3ae7564b81b31a2f5673ee22bb
I20240502 21:27:58.819555 23755 catalog_manager.cc:4830] TS 038c5d9d1d7a49c981c4c34fbc00d9e6 (127.15.255.132:46813): tablet 83b95f3ae7564b81b31a2f5673ee22bb (table rebalance_test_table_8 [id=9a4e7bd31db34cf3aff2242876f0de47]) successfully deleted
I20240502 21:27:58.978314 29682 raft_consensus.cc:1050] T 8f14db150fda449e962be2466f1d2a0e P 6810a866decb4f5089f7d35c870e4204: attempting to promote NON_VOTER 4f58800ba2ed45eabd81c2f2e691aacd to VOTER
I20240502 21:27:58.979125 29682 consensus_queue.cc:237] T 8f14db150fda449e962be2466f1d2a0e P 6810a866decb4f5089f7d35c870e4204 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } attrs { replace: true } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: false } } peers { permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: false } }
I20240502 21:27:58.981482 24183 raft_consensus.cc:1261] T 8f14db150fda449e962be2466f1d2a0e P 009d3e7e8a8b4e2497de3a73d01ced49 [term 1 FOLLOWER]: Refusing update from remote peer 6810a866decb4f5089f7d35c870e4204: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:27:58.981562 25878 raft_consensus.cc:1261] T 8f14db150fda449e962be2466f1d2a0e P ee27c3f9e0ec46caa742ac386d6b7c20 [term 1 FOLLOWER]: Refusing update from remote peer 6810a866decb4f5089f7d35c870e4204: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:27:58.981729 27125 raft_consensus.cc:1261] T 8f14db150fda449e962be2466f1d2a0e P 4f58800ba2ed45eabd81c2f2e691aacd [term 1 LEARNER]: Refusing update from remote peer 6810a866decb4f5089f7d35c870e4204: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:27:58.982115 29951 consensus_queue.cc:1035] T 8f14db150fda449e962be2466f1d2a0e P 6810a866decb4f5089f7d35c870e4204 [LEADER]: Connected to new peer: Peer: permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:27:58.982482 29682 consensus_queue.cc:1035] T 8f14db150fda449e962be2466f1d2a0e P 6810a866decb4f5089f7d35c870e4204 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:27:58.982844 29995 consensus_queue.cc:1035] T 8f14db150fda449e962be2466f1d2a0e P 6810a866decb4f5089f7d35c870e4204 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:27:58.986238 29682 raft_consensus.cc:2932] T 8f14db150fda449e962be2466f1d2a0e P 6810a866decb4f5089f7d35c870e4204 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 4f58800ba2ed45eabd81c2f2e691aacd (127.15.255.137) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } attrs { replace: true } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: false } } peers { permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: false } } }
I20240502 21:27:58.986981 25878 raft_consensus.cc:2932] T 8f14db150fda449e962be2466f1d2a0e P ee27c3f9e0ec46caa742ac386d6b7c20 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 4f58800ba2ed45eabd81c2f2e691aacd (127.15.255.137) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } attrs { replace: true } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: false } } peers { permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: false } } }
I20240502 21:27:58.990460 24183 raft_consensus.cc:2932] T 8f14db150fda449e962be2466f1d2a0e P 009d3e7e8a8b4e2497de3a73d01ced49 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 4f58800ba2ed45eabd81c2f2e691aacd (127.15.255.137) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } attrs { replace: true } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: false } } peers { permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: false } } }
I20240502 21:27:58.992063 23771 catalog_manager.cc:5484] T 8f14db150fda449e962be2466f1d2a0e P 6810a866decb4f5089f7d35c870e4204 reported cstate change: config changed from index 7 to 8, 4f58800ba2ed45eabd81c2f2e691aacd (127.15.255.137) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "6810a866decb4f5089f7d35c870e4204" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:58.992146 27125 raft_consensus.cc:2932] T 8f14db150fda449e962be2466f1d2a0e P 4f58800ba2ed45eabd81c2f2e691aacd [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 4f58800ba2ed45eabd81c2f2e691aacd (127.15.255.137) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } attrs { replace: true } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: false } } peers { permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: false } } }
I20240502 21:27:59.000286 23916 consensus_queue.cc:237] T 8f14db150fda449e962be2466f1d2a0e P 6810a866decb4f5089f7d35c870e4204 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: false } } peers { permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: false } }
I20240502 21:27:59.002224 25878 raft_consensus.cc:1261] T 8f14db150fda449e962be2466f1d2a0e P ee27c3f9e0ec46caa742ac386d6b7c20 [term 1 FOLLOWER]: Refusing update from remote peer 6810a866decb4f5089f7d35c870e4204: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:27:59.002225 27125 raft_consensus.cc:1261] T 8f14db150fda449e962be2466f1d2a0e P 4f58800ba2ed45eabd81c2f2e691aacd [term 1 FOLLOWER]: Refusing update from remote peer 6810a866decb4f5089f7d35c870e4204: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:27:59.002733 29682 consensus_queue.cc:1035] T 8f14db150fda449e962be2466f1d2a0e P 6810a866decb4f5089f7d35c870e4204 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:27:59.003046 29951 consensus_queue.cc:1035] T 8f14db150fda449e962be2466f1d2a0e P 6810a866decb4f5089f7d35c870e4204 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:27:59.005211 29951 raft_consensus.cc:2932] T 8f14db150fda449e962be2466f1d2a0e P 6810a866decb4f5089f7d35c870e4204 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 009d3e7e8a8b4e2497de3a73d01ced49 (127.15.255.131) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: false } } peers { permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: false } } }
I20240502 21:27:59.005746 25878 raft_consensus.cc:2932] T 8f14db150fda449e962be2466f1d2a0e P ee27c3f9e0ec46caa742ac386d6b7c20 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 009d3e7e8a8b4e2497de3a73d01ced49 (127.15.255.131) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: false } } peers { permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: false } } }
I20240502 21:27:59.006381 27125 raft_consensus.cc:2932] T 8f14db150fda449e962be2466f1d2a0e P 4f58800ba2ed45eabd81c2f2e691aacd [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 009d3e7e8a8b4e2497de3a73d01ced49 (127.15.255.131) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: false } } peers { permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: false } } }
I20240502 21:27:59.008181 23757 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 8f14db150fda449e962be2466f1d2a0e with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:27:59.009869 23765 catalog_manager.cc:5484] T 8f14db150fda449e962be2466f1d2a0e P 6810a866decb4f5089f7d35c870e4204 reported cstate change: config changed from index 8 to 9, VOTER 009d3e7e8a8b4e2497de3a73d01ced49 (127.15.255.131) evicted. New cstate: current_term: 1 leader_uuid: "6810a866decb4f5089f7d35c870e4204" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:27:59.014014 24162 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f14db150fda449e962be2466f1d2a0e with delete_type TABLET_DATA_TOMBSTONED (TS 009d3e7e8a8b4e2497de3a73d01ced49 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:33102
I20240502 21:27:59.014701 30059 tablet_replica.cc:330] T 8f14db150fda449e962be2466f1d2a0e P 009d3e7e8a8b4e2497de3a73d01ced49: stopping tablet replica
I20240502 21:27:59.014904 30059 raft_consensus.cc:2229] T 8f14db150fda449e962be2466f1d2a0e P 009d3e7e8a8b4e2497de3a73d01ced49 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:27:59.015285 30059 raft_consensus.cc:2258] T 8f14db150fda449e962be2466f1d2a0e P 009d3e7e8a8b4e2497de3a73d01ced49 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:27:59.016157 30059 ts_tablet_manager.cc:1875] T 8f14db150fda449e962be2466f1d2a0e P 009d3e7e8a8b4e2497de3a73d01ced49: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:27:59.021013 30059 ts_tablet_manager.cc:1888] T 8f14db150fda449e962be2466f1d2a0e P 009d3e7e8a8b4e2497de3a73d01ced49: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240502 21:27:59.021119 30059 log.cc:1196] T 8f14db150fda449e962be2466f1d2a0e P 009d3e7e8a8b4e2497de3a73d01ced49: Deleting WAL directory at /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-2/wal/wals/8f14db150fda449e962be2466f1d2a0e
I20240502 21:27:59.021803 23756 catalog_manager.cc:4830] TS 009d3e7e8a8b4e2497de3a73d01ced49 (127.15.255.131:37895): tablet 8f14db150fda449e962be2466f1d2a0e (table rebalance_test_table_4 [id=9d2b330e886e4e83a437506bf99d77a3]) successfully deleted
I20240502 21:27:59.101070 29983 raft_consensus.cc:1050] T d5697d8b06094211b261f144de9d46f8 P 038c5d9d1d7a49c981c4c34fbc00d9e6: attempting to promote NON_VOTER 4f58800ba2ed45eabd81c2f2e691aacd to VOTER
I20240502 21:27:59.101747 29983 consensus_queue.cc:237] T d5697d8b06094211b261f144de9d46f8 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } attrs { replace: true } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } } peers { permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: false } }
I20240502 21:27:59.104027 26279 raft_consensus.cc:1261] T d5697d8b06094211b261f144de9d46f8 P a3e500d7077a48acb5edb33f7e178c81 [term 1 FOLLOWER]: Refusing update from remote peer 038c5d9d1d7a49c981c4c34fbc00d9e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:27:59.104063 27125 raft_consensus.cc:1261] T d5697d8b06094211b261f144de9d46f8 P 4f58800ba2ed45eabd81c2f2e691aacd [term 1 LEARNER]: Refusing update from remote peer 038c5d9d1d7a49c981c4c34fbc00d9e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:27:59.104216 24183 raft_consensus.cc:1261] T d5697d8b06094211b261f144de9d46f8 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 1 FOLLOWER]: Refusing update from remote peer 038c5d9d1d7a49c981c4c34fbc00d9e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:27:59.104645 29985 consensus_queue.cc:1035] T d5697d8b06094211b261f144de9d46f8 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:27:59.104928 29949 consensus_queue.cc:1035] T d5697d8b06094211b261f144de9d46f8 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:27:59.105187 29983 consensus_queue.cc:1035] T d5697d8b06094211b261f144de9d46f8 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:27:59.108521 29983 raft_consensus.cc:2932] T d5697d8b06094211b261f144de9d46f8 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 4f58800ba2ed45eabd81c2f2e691aacd (127.15.255.137) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } attrs { replace: true } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } } peers { permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: false } } }
I20240502 21:27:59.109274 27125 raft_consensus.cc:2932] T d5697d8b06094211b261f144de9d46f8 P 4f58800ba2ed45eabd81c2f2e691aacd [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 4f58800ba2ed45eabd81c2f2e691aacd (127.15.255.137) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } attrs { replace: true } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } } peers { permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: false } } }
I20240502 21:27:59.109423 26279 raft_consensus.cc:2932] T d5697d8b06094211b261f144de9d46f8 P a3e500d7077a48acb5edb33f7e178c81 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 4f58800ba2ed45eabd81c2f2e691aacd (127.15.255.137) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } attrs { replace: true } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } } peers { permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: false } } }
I20240502 21:27:59.114917 23771 catalog_manager.cc:5484] T d5697d8b06094211b261f144de9d46f8 P a3e500d7077a48acb5edb33f7e178c81 reported cstate change: config changed from index 7 to 8, 4f58800ba2ed45eabd81c2f2e691aacd (127.15.255.137) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } attrs { replace: true } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } } peers { permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: false } } }
I20240502 21:27:59.119796 24183 raft_consensus.cc:2932] T d5697d8b06094211b261f144de9d46f8 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 4f58800ba2ed45eabd81c2f2e691aacd (127.15.255.137) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } attrs { replace: true } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } } peers { permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: false } } }
I20240502 21:27:59.161444 24316 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d5697d8b06094211b261f144de9d46f8"
dest_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:37228
I20240502 21:27:59.161656 24316 raft_consensus.cc:595] T d5697d8b06094211b261f144de9d46f8 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 1 LEADER]: Received request to transfer leadership
I20240502 21:27:59.195300 24316 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d5697d8b06094211b261f144de9d46f8"
dest_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:37242
I20240502 21:27:59.195529 24316 raft_consensus.cc:595] T d5697d8b06094211b261f144de9d46f8 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 1 LEADER]: Received request to transfer leadership
I20240502 21:27:59.317368 29860 tablet_copy_client.cc:523] T dc0e070513cc46ca82c1946423ad91c5 P 4f58800ba2ed45eabd81c2f2e691aacd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:27:59.319970 29860 tablet_bootstrap.cc:492] T dc0e070513cc46ca82c1946423ad91c5 P 4f58800ba2ed45eabd81c2f2e691aacd: Bootstrap starting.
I20240502 21:27:59.350070 29860 tablet_bootstrap.cc:492] T dc0e070513cc46ca82c1946423ad91c5 P 4f58800ba2ed45eabd81c2f2e691aacd: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:27:59.350440 29860 tablet_bootstrap.cc:492] T dc0e070513cc46ca82c1946423ad91c5 P 4f58800ba2ed45eabd81c2f2e691aacd: Bootstrap complete.
I20240502 21:27:59.350685 29860 ts_tablet_manager.cc:1367] T dc0e070513cc46ca82c1946423ad91c5 P 4f58800ba2ed45eabd81c2f2e691aacd: Time spent bootstrapping tablet: real 0.031s user 0.031s sys 0.000s
I20240502 21:27:59.351271 29860 raft_consensus.cc:348] T dc0e070513cc46ca82c1946423ad91c5 P 4f58800ba2ed45eabd81c2f2e691aacd [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } attrs { replace: true } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } } peers { permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: NON_VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: true } }
I20240502 21:27:59.351475 29860 raft_consensus.cc:729] T dc0e070513cc46ca82c1946423ad91c5 P 4f58800ba2ed45eabd81c2f2e691aacd [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4f58800ba2ed45eabd81c2f2e691aacd, State: Initialized, Role: LEARNER
I20240502 21:27:59.351656 29860 consensus_queue.cc:260] T dc0e070513cc46ca82c1946423ad91c5 P 4f58800ba2ed45eabd81c2f2e691aacd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } attrs { replace: true } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } } peers { permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: NON_VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: true } }
I20240502 21:27:59.352142 29860 ts_tablet_manager.cc:1398] T dc0e070513cc46ca82c1946423ad91c5 P 4f58800ba2ed45eabd81c2f2e691aacd: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 21:27:59.352857 24070 tablet_copy_service.cc:342] P 6c194a6db6c145acafc0f3068b8ce1fd: Request end of tablet copy session 4f58800ba2ed45eabd81c2f2e691aacd-dc0e070513cc46ca82c1946423ad91c5 received from {username='slave'} at 127.15.255.137:60133
I20240502 21:27:59.352978 24070 tablet_copy_service.cc:433] P 6c194a6db6c145acafc0f3068b8ce1fd: ending tablet copy session 4f58800ba2ed45eabd81c2f2e691aacd-dc0e070513cc46ca82c1946423ad91c5 on tablet dc0e070513cc46ca82c1946423ad91c5 with peer 4f58800ba2ed45eabd81c2f2e691aacd
I20240502 21:27:59.401618 29875 tablet_copy_client.cc:523] T 256ae04a48b849c4a9086b6b1ff5df19 P 4f58800ba2ed45eabd81c2f2e691aacd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:27:59.405108 29875 tablet_bootstrap.cc:492] T 256ae04a48b849c4a9086b6b1ff5df19 P 4f58800ba2ed45eabd81c2f2e691aacd: Bootstrap starting.
I20240502 21:27:59.451102 29875 tablet_bootstrap.cc:492] T 256ae04a48b849c4a9086b6b1ff5df19 P 4f58800ba2ed45eabd81c2f2e691aacd: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:27:59.456810 24316 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d5697d8b06094211b261f144de9d46f8"
dest_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:37246
I20240502 21:27:59.457002 24316 raft_consensus.cc:595] T d5697d8b06094211b261f144de9d46f8 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 1 LEADER]: Received request to transfer leadership
I20240502 21:27:59.472699 29875 tablet_bootstrap.cc:492] T 256ae04a48b849c4a9086b6b1ff5df19 P 4f58800ba2ed45eabd81c2f2e691aacd: Bootstrap complete.
I20240502 21:27:59.472949 29875 ts_tablet_manager.cc:1367] T 256ae04a48b849c4a9086b6b1ff5df19 P 4f58800ba2ed45eabd81c2f2e691aacd: Time spent bootstrapping tablet: real 0.068s user 0.041s sys 0.004s
I20240502 21:27:59.473544 29875 raft_consensus.cc:348] T 256ae04a48b849c4a9086b6b1ff5df19 P 4f58800ba2ed45eabd81c2f2e691aacd [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } attrs { replace: true } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } } peers { permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: NON_VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: true } }
I20240502 21:27:59.473752 29875 raft_consensus.cc:729] T 256ae04a48b849c4a9086b6b1ff5df19 P 4f58800ba2ed45eabd81c2f2e691aacd [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4f58800ba2ed45eabd81c2f2e691aacd, State: Initialized, Role: LEARNER
I20240502 21:27:59.473914 29875 consensus_queue.cc:260] T 256ae04a48b849c4a9086b6b1ff5df19 P 4f58800ba2ed45eabd81c2f2e691aacd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } attrs { replace: true } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } } peers { permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: NON_VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: true } }
I20240502 21:27:59.474392 29875 ts_tablet_manager.cc:1398] T 256ae04a48b849c4a9086b6b1ff5df19 P 4f58800ba2ed45eabd81c2f2e691aacd: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 21:27:59.475178 23935 tablet_copy_service.cc:342] P 6810a866decb4f5089f7d35c870e4204: Request end of tablet copy session 4f58800ba2ed45eabd81c2f2e691aacd-256ae04a48b849c4a9086b6b1ff5df19 received from {username='slave'} at 127.15.255.137:35119
I20240502 21:27:59.475332 23935 tablet_copy_service.cc:433] P 6810a866decb4f5089f7d35c870e4204: ending tablet copy session 4f58800ba2ed45eabd81c2f2e691aacd-256ae04a48b849c4a9086b6b1ff5df19 on tablet 256ae04a48b849c4a9086b6b1ff5df19 with peer 4f58800ba2ed45eabd81c2f2e691aacd
I20240502 21:27:59.558535 29985 raft_consensus.cc:979] T d5697d8b06094211b261f144de9d46f8 P 038c5d9d1d7a49c981c4c34fbc00d9e6: : Instructing follower a3e500d7077a48acb5edb33f7e178c81 to start an election
I20240502 21:27:59.558737 29983 raft_consensus.cc:1067] T d5697d8b06094211b261f144de9d46f8 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 1 LEADER]: Signalling peer a3e500d7077a48acb5edb33f7e178c81 to start an election
I20240502 21:27:59.559412 26279 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d5697d8b06094211b261f144de9d46f8"
dest_uuid: "a3e500d7077a48acb5edb33f7e178c81"
from {username='slave'} at 127.15.255.132:35235
I20240502 21:27:59.559659 26279 raft_consensus.cc:482] T d5697d8b06094211b261f144de9d46f8 P a3e500d7077a48acb5edb33f7e178c81 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:27:59.559772 26279 raft_consensus.cc:3037] T d5697d8b06094211b261f144de9d46f8 P a3e500d7077a48acb5edb33f7e178c81 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:59.561996 26279 raft_consensus.cc:504] T d5697d8b06094211b261f144de9d46f8 P a3e500d7077a48acb5edb33f7e178c81 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } attrs { replace: true } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } } peers { permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: false } }
I20240502 21:27:59.562820 26279 leader_election.cc:288] T d5697d8b06094211b261f144de9d46f8 P a3e500d7077a48acb5edb33f7e178c81 [CANDIDATE]: Term 2 election: Requested vote from peers 009d3e7e8a8b4e2497de3a73d01ced49 (127.15.255.131:37895), 038c5d9d1d7a49c981c4c34fbc00d9e6 (127.15.255.132:46813), 4f58800ba2ed45eabd81c2f2e691aacd (127.15.255.137:40161)
I20240502 21:27:59.563128 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5697d8b06094211b261f144de9d46f8" candidate_uuid: "a3e500d7077a48acb5edb33f7e178c81" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6"
I20240502 21:27:59.563170 24183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5697d8b06094211b261f144de9d46f8" candidate_uuid: "a3e500d7077a48acb5edb33f7e178c81" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "009d3e7e8a8b4e2497de3a73d01ced49"
I20240502 21:27:59.563346 24316 raft_consensus.cc:3032] T d5697d8b06094211b261f144de9d46f8 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 21:27:59.563373 24183 raft_consensus.cc:3037] T d5697d8b06094211b261f144de9d46f8 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:59.563431 24316 raft_consensus.cc:729] T d5697d8b06094211b261f144de9d46f8 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 038c5d9d1d7a49c981c4c34fbc00d9e6, State: Running, Role: LEADER
I20240502 21:27:59.563728 24316 consensus_queue.cc:260] T d5697d8b06094211b261f144de9d46f8 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } attrs { replace: true } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } } peers { permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: false } }
I20240502 21:27:59.564137 24316 raft_consensus.cc:3037] T d5697d8b06094211b261f144de9d46f8 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:59.566131 24183 raft_consensus.cc:2454] T d5697d8b06094211b261f144de9d46f8 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a3e500d7077a48acb5edb33f7e178c81 in term 2.
I20240502 21:27:59.566721 24316 raft_consensus.cc:2454] T d5697d8b06094211b261f144de9d46f8 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a3e500d7077a48acb5edb33f7e178c81 in term 2.
I20240502 21:27:59.567111 26212 leader_election.cc:302] T d5697d8b06094211b261f144de9d46f8 P a3e500d7077a48acb5edb33f7e178c81 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 009d3e7e8a8b4e2497de3a73d01ced49, 038c5d9d1d7a49c981c4c34fbc00d9e6, a3e500d7077a48acb5edb33f7e178c81; no voters:
I20240502 21:27:59.567420 29715 raft_consensus.cc:2781] T d5697d8b06094211b261f144de9d46f8 P a3e500d7077a48acb5edb33f7e178c81 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 21:27:59.567812 29715 raft_consensus.cc:686] T d5697d8b06094211b261f144de9d46f8 P a3e500d7077a48acb5edb33f7e178c81 [term 2 LEADER]: Becoming Leader. State: Replica: a3e500d7077a48acb5edb33f7e178c81, State: Running, Role: LEADER
I20240502 21:27:59.568178 29715 consensus_queue.cc:237] T d5697d8b06094211b261f144de9d46f8 P a3e500d7077a48acb5edb33f7e178c81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } attrs { replace: true } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } } peers { permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: false } }
I20240502 21:27:59.570677 27125 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5697d8b06094211b261f144de9d46f8" candidate_uuid: "a3e500d7077a48acb5edb33f7e178c81" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "4f58800ba2ed45eabd81c2f2e691aacd"
I20240502 21:27:59.570916 27125 raft_consensus.cc:3037] T d5697d8b06094211b261f144de9d46f8 P 4f58800ba2ed45eabd81c2f2e691aacd [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:27:59.571370 23766 catalog_manager.cc:5484] T d5697d8b06094211b261f144de9d46f8 P a3e500d7077a48acb5edb33f7e178c81 reported cstate change: term changed from 1 to 2, leader changed from 038c5d9d1d7a49c981c4c34fbc00d9e6 (127.15.255.132) to a3e500d7077a48acb5edb33f7e178c81 (127.15.255.136). New cstate: current_term: 2 leader_uuid: "a3e500d7077a48acb5edb33f7e178c81" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 21:27:59.573385 27125 raft_consensus.cc:2454] T d5697d8b06094211b261f144de9d46f8 P 4f58800ba2ed45eabd81c2f2e691aacd [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a3e500d7077a48acb5edb33f7e178c81 in term 2.
I20240502 21:27:59.729979 27125 raft_consensus.cc:1203] T dc0e070513cc46ca82c1946423ad91c5 P 4f58800ba2ed45eabd81c2f2e691aacd [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240502 21:27:59.969748 24316 raft_consensus.cc:1261] T d5697d8b06094211b261f144de9d46f8 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 2 FOLLOWER]: Refusing update from remote peer a3e500d7077a48acb5edb33f7e178c81: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 21:27:59.970853 29715 consensus_queue.cc:1035] T d5697d8b06094211b261f144de9d46f8 P a3e500d7077a48acb5edb33f7e178c81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:28:00.002964 27125 raft_consensus.cc:1203] T 256ae04a48b849c4a9086b6b1ff5df19 P 4f58800ba2ed45eabd81c2f2e691aacd [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240502 21:28:00.025064 27125 raft_consensus.cc:1261] T d5697d8b06094211b261f144de9d46f8 P 4f58800ba2ed45eabd81c2f2e691aacd [term 2 FOLLOWER]: Refusing update from remote peer a3e500d7077a48acb5edb33f7e178c81: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 21:28:00.025606 29715 consensus_queue.cc:1035] T d5697d8b06094211b261f144de9d46f8 P a3e500d7077a48acb5edb33f7e178c81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:28:00.028812 24183 raft_consensus.cc:1261] T d5697d8b06094211b261f144de9d46f8 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 2 FOLLOWER]: Refusing update from remote peer a3e500d7077a48acb5edb33f7e178c81: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 21:28:00.029408 29715 consensus_queue.cc:1035] T d5697d8b06094211b261f144de9d46f8 P a3e500d7077a48acb5edb33f7e178c81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:28:00.167974 30166 raft_consensus.cc:1050] T dc0e070513cc46ca82c1946423ad91c5 P 6c194a6db6c145acafc0f3068b8ce1fd: attempting to promote NON_VOTER 4f58800ba2ed45eabd81c2f2e691aacd to VOTER
I20240502 21:28:00.168696 30166 consensus_queue.cc:237] T dc0e070513cc46ca82c1946423ad91c5 P 6c194a6db6c145acafc0f3068b8ce1fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } attrs { replace: true } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } } peers { permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: false } }
I20240502 21:28:00.171110 27125 raft_consensus.cc:1261] T dc0e070513cc46ca82c1946423ad91c5 P 4f58800ba2ed45eabd81c2f2e691aacd [term 1 LEARNER]: Refusing update from remote peer 6c194a6db6c145acafc0f3068b8ce1fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:28:00.171316 24183 raft_consensus.cc:1261] T dc0e070513cc46ca82c1946423ad91c5 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 1 FOLLOWER]: Refusing update from remote peer 6c194a6db6c145acafc0f3068b8ce1fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:28:00.171427 26147 raft_consensus.cc:1261] T dc0e070513cc46ca82c1946423ad91c5 P e117afbf8819424e8a98476453dfde2a [term 1 FOLLOWER]: Refusing update from remote peer 6c194a6db6c145acafc0f3068b8ce1fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:28:00.171810 30166 consensus_queue.cc:1035] T dc0e070513cc46ca82c1946423ad91c5 P 6c194a6db6c145acafc0f3068b8ce1fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:28:00.172128 30191 consensus_queue.cc:1035] T dc0e070513cc46ca82c1946423ad91c5 P 6c194a6db6c145acafc0f3068b8ce1fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:28:00.172508 30189 consensus_queue.cc:1035] T dc0e070513cc46ca82c1946423ad91c5 P 6c194a6db6c145acafc0f3068b8ce1fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:28:00.175392 30191 raft_consensus.cc:2932] T dc0e070513cc46ca82c1946423ad91c5 P 6c194a6db6c145acafc0f3068b8ce1fd [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 4f58800ba2ed45eabd81c2f2e691aacd (127.15.255.137) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } attrs { replace: true } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } } peers { permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: false } } }
I20240502 21:28:00.175896 24183 raft_consensus.cc:2932] T dc0e070513cc46ca82c1946423ad91c5 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 4f58800ba2ed45eabd81c2f2e691aacd (127.15.255.137) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } attrs { replace: true } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } } peers { permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: false } } }
I20240502 21:28:00.180239 23766 catalog_manager.cc:5484] T dc0e070513cc46ca82c1946423ad91c5 P 009d3e7e8a8b4e2497de3a73d01ced49 reported cstate change: config changed from index 7 to 8, 4f58800ba2ed45eabd81c2f2e691aacd (127.15.255.137) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } attrs { replace: true } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } } peers { permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: false } } }
I20240502 21:28:00.182317 26147 raft_consensus.cc:2932] T dc0e070513cc46ca82c1946423ad91c5 P e117afbf8819424e8a98476453dfde2a [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 4f58800ba2ed45eabd81c2f2e691aacd (127.15.255.137) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } attrs { replace: true } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } } peers { permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: false } } }
I20240502 21:28:00.186388 27125 raft_consensus.cc:2932] T dc0e070513cc46ca82c1946423ad91c5 P 4f58800ba2ed45eabd81c2f2e691aacd [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 4f58800ba2ed45eabd81c2f2e691aacd (127.15.255.137) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } attrs { replace: true } } peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } } peers { permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: false } } }
I20240502 21:28:00.190409 24047 consensus_queue.cc:237] T dc0e070513cc46ca82c1946423ad91c5 P 6c194a6db6c145acafc0f3068b8ce1fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } } peers { permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: false } }
I20240502 21:28:00.192070 27125 raft_consensus.cc:1261] T dc0e070513cc46ca82c1946423ad91c5 P 4f58800ba2ed45eabd81c2f2e691aacd [term 1 FOLLOWER]: Refusing update from remote peer 6c194a6db6c145acafc0f3068b8ce1fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:28:00.192226 26147 raft_consensus.cc:1261] T dc0e070513cc46ca82c1946423ad91c5 P e117afbf8819424e8a98476453dfde2a [term 1 FOLLOWER]: Refusing update from remote peer 6c194a6db6c145acafc0f3068b8ce1fd: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:28:00.192721 30054 consensus_queue.cc:1035] T dc0e070513cc46ca82c1946423ad91c5 P 6c194a6db6c145acafc0f3068b8ce1fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:28:00.192981 30189 consensus_queue.cc:1035] T dc0e070513cc46ca82c1946423ad91c5 P 6c194a6db6c145acafc0f3068b8ce1fd [LEADER]: Connected to new peer: Peer: permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:28:00.195012 30054 raft_consensus.cc:2932] T dc0e070513cc46ca82c1946423ad91c5 P 6c194a6db6c145acafc0f3068b8ce1fd [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 009d3e7e8a8b4e2497de3a73d01ced49 (127.15.255.131) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } } peers { permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: false } } }
I20240502 21:28:00.195626 27125 raft_consensus.cc:2932] T dc0e070513cc46ca82c1946423ad91c5 P 4f58800ba2ed45eabd81c2f2e691aacd [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 009d3e7e8a8b4e2497de3a73d01ced49 (127.15.255.131) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } } peers { permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: false } } }
I20240502 21:28:00.196308 26145 raft_consensus.cc:2932] T dc0e070513cc46ca82c1946423ad91c5 P e117afbf8819424e8a98476453dfde2a [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 009d3e7e8a8b4e2497de3a73d01ced49 (127.15.255.131) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } } peers { permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: false } } }
I20240502 21:28:00.198604 23757 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet dc0e070513cc46ca82c1946423ad91c5 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:28:00.200722 23771 catalog_manager.cc:5484] T dc0e070513cc46ca82c1946423ad91c5 P 6c194a6db6c145acafc0f3068b8ce1fd reported cstate change: config changed from index 8 to 9, VOTER 009d3e7e8a8b4e2497de3a73d01ced49 (127.15.255.131) evicted. New cstate: current_term: 1 leader_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:28:00.203990 24162 tablet_service.cc:1514] Processing DeleteTablet for tablet dc0e070513cc46ca82c1946423ad91c5 with delete_type TABLET_DATA_TOMBSTONED (TS 009d3e7e8a8b4e2497de3a73d01ced49 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:33102
I20240502 21:28:00.208847 30199 tablet_replica.cc:330] T dc0e070513cc46ca82c1946423ad91c5 P 009d3e7e8a8b4e2497de3a73d01ced49: stopping tablet replica
I20240502 21:28:00.209039 30199 raft_consensus.cc:2229] T dc0e070513cc46ca82c1946423ad91c5 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:28:00.209329 30199 raft_consensus.cc:2258] T dc0e070513cc46ca82c1946423ad91c5 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:28:00.210052 30199 ts_tablet_manager.cc:1875] T dc0e070513cc46ca82c1946423ad91c5 P 009d3e7e8a8b4e2497de3a73d01ced49: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:28:00.214212 30199 ts_tablet_manager.cc:1888] T dc0e070513cc46ca82c1946423ad91c5 P 009d3e7e8a8b4e2497de3a73d01ced49: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240502 21:28:00.214344 30199 log.cc:1196] T dc0e070513cc46ca82c1946423ad91c5 P 009d3e7e8a8b4e2497de3a73d01ced49: Deleting WAL directory at /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-2/wal/wals/dc0e070513cc46ca82c1946423ad91c5
I20240502 21:28:00.214965 23756 catalog_manager.cc:4830] TS 009d3e7e8a8b4e2497de3a73d01ced49 (127.15.255.131:37895): tablet dc0e070513cc46ca82c1946423ad91c5 (table rebalance_test_table_0 [id=1e673d0be4694336bbf65d9c114c614f]) successfully deleted
I20240502 21:28:00.419684 30225 raft_consensus.cc:1050] T 256ae04a48b849c4a9086b6b1ff5df19 P 6810a866decb4f5089f7d35c870e4204: attempting to promote NON_VOTER 4f58800ba2ed45eabd81c2f2e691aacd to VOTER
I20240502 21:28:00.420454 30225 consensus_queue.cc:237] T 256ae04a48b849c4a9086b6b1ff5df19 P 6810a866decb4f5089f7d35c870e4204 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } attrs { replace: true } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } } peers { permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: false } }
I20240502 21:28:00.423063 26147 raft_consensus.cc:1261] T 256ae04a48b849c4a9086b6b1ff5df19 P e117afbf8819424e8a98476453dfde2a [term 1 FOLLOWER]: Refusing update from remote peer 6810a866decb4f5089f7d35c870e4204: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:28:00.423081 27125 raft_consensus.cc:1261] T 256ae04a48b849c4a9086b6b1ff5df19 P 4f58800ba2ed45eabd81c2f2e691aacd [term 1 LEARNER]: Refusing update from remote peer 6810a866decb4f5089f7d35c870e4204: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:28:00.423657 30173 consensus_queue.cc:1035] T 256ae04a48b849c4a9086b6b1ff5df19 P 6810a866decb4f5089f7d35c870e4204 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:28:00.423794 24183 raft_consensus.cc:1261] T 256ae04a48b849c4a9086b6b1ff5df19 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 1 FOLLOWER]: Refusing update from remote peer 6810a866decb4f5089f7d35c870e4204: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:28:00.423990 30225 consensus_queue.cc:1035] T 256ae04a48b849c4a9086b6b1ff5df19 P 6810a866decb4f5089f7d35c870e4204 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:28:00.424348 29951 consensus_queue.cc:1035] T 256ae04a48b849c4a9086b6b1ff5df19 P 6810a866decb4f5089f7d35c870e4204 [LEADER]: Connected to new peer: Peer: permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:28:00.427363 30173 raft_consensus.cc:2932] T 256ae04a48b849c4a9086b6b1ff5df19 P 6810a866decb4f5089f7d35c870e4204 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 4f58800ba2ed45eabd81c2f2e691aacd (127.15.255.137) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } attrs { replace: true } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } } peers { permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: false } } }
I20240502 21:28:00.428004 26147 raft_consensus.cc:2932] T 256ae04a48b849c4a9086b6b1ff5df19 P e117afbf8819424e8a98476453dfde2a [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 4f58800ba2ed45eabd81c2f2e691aacd (127.15.255.137) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } attrs { replace: true } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } } peers { permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: false } } }
I20240502 21:28:00.433044 24183 raft_consensus.cc:2932] T 256ae04a48b849c4a9086b6b1ff5df19 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 4f58800ba2ed45eabd81c2f2e691aacd (127.15.255.137) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } attrs { replace: true } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } } peers { permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: false } } }
I20240502 21:28:00.436604 23771 catalog_manager.cc:5484] T 256ae04a48b849c4a9086b6b1ff5df19 P 6810a866decb4f5089f7d35c870e4204 reported cstate change: config changed from index 7 to 8, 4f58800ba2ed45eabd81c2f2e691aacd (127.15.255.137) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "6810a866decb4f5089f7d35c870e4204" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 21:28:00.441701 27125 raft_consensus.cc:2932] T 256ae04a48b849c4a9086b6b1ff5df19 P 4f58800ba2ed45eabd81c2f2e691aacd [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 4f58800ba2ed45eabd81c2f2e691aacd (127.15.255.137) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } attrs { replace: true } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } } peers { permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: false } } }
I20240502 21:28:00.443902 23916 consensus_queue.cc:237] T 256ae04a48b849c4a9086b6b1ff5df19 P 6810a866decb4f5089f7d35c870e4204 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } } peers { permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: false } }
I20240502 21:28:00.445442 27125 raft_consensus.cc:1261] T 256ae04a48b849c4a9086b6b1ff5df19 P 4f58800ba2ed45eabd81c2f2e691aacd [term 1 FOLLOWER]: Refusing update from remote peer 6810a866decb4f5089f7d35c870e4204: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:28:00.445746 26147 raft_consensus.cc:1261] T 256ae04a48b849c4a9086b6b1ff5df19 P e117afbf8819424e8a98476453dfde2a [term 1 FOLLOWER]: Refusing update from remote peer 6810a866decb4f5089f7d35c870e4204: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:28:00.446610 30173 consensus_queue.cc:1035] T 256ae04a48b849c4a9086b6b1ff5df19 P 6810a866decb4f5089f7d35c870e4204 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:28:00.446844 29951 consensus_queue.cc:1035] T 256ae04a48b849c4a9086b6b1ff5df19 P 6810a866decb4f5089f7d35c870e4204 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:28:00.448882 29951 raft_consensus.cc:2932] T 256ae04a48b849c4a9086b6b1ff5df19 P 6810a866decb4f5089f7d35c870e4204 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 009d3e7e8a8b4e2497de3a73d01ced49 (127.15.255.131) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } } peers { permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: false } } }
I20240502 21:28:00.449874 27125 raft_consensus.cc:2932] T 256ae04a48b849c4a9086b6b1ff5df19 P 4f58800ba2ed45eabd81c2f2e691aacd [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 009d3e7e8a8b4e2497de3a73d01ced49 (127.15.255.131) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } } peers { permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: false } } }
I20240502 21:28:00.451865 23757 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 256ae04a48b849c4a9086b6b1ff5df19 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:28:00.452045 26147 raft_consensus.cc:2932] T 256ae04a48b849c4a9086b6b1ff5df19 P e117afbf8819424e8a98476453dfde2a [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 009d3e7e8a8b4e2497de3a73d01ced49 (127.15.255.131) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } } peers { permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: false } } }
I20240502 21:28:00.453055 23766 catalog_manager.cc:5484] T 256ae04a48b849c4a9086b6b1ff5df19 P 6810a866decb4f5089f7d35c870e4204 reported cstate change: config changed from index 8 to 9, VOTER 009d3e7e8a8b4e2497de3a73d01ced49 (127.15.255.131) evicted. New cstate: current_term: 1 leader_uuid: "6810a866decb4f5089f7d35c870e4204" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:28:00.458251 24162 tablet_service.cc:1514] Processing DeleteTablet for tablet 256ae04a48b849c4a9086b6b1ff5df19 with delete_type TABLET_DATA_TOMBSTONED (TS 009d3e7e8a8b4e2497de3a73d01ced49 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:33102
I20240502 21:28:00.458631 30199 tablet_replica.cc:330] T 256ae04a48b849c4a9086b6b1ff5df19 P 009d3e7e8a8b4e2497de3a73d01ced49: stopping tablet replica
I20240502 21:28:00.458832 30199 raft_consensus.cc:2229] T 256ae04a48b849c4a9086b6b1ff5df19 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:28:00.459245 30199 raft_consensus.cc:2258] T 256ae04a48b849c4a9086b6b1ff5df19 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:28:00.460033 30199 ts_tablet_manager.cc:1875] T 256ae04a48b849c4a9086b6b1ff5df19 P 009d3e7e8a8b4e2497de3a73d01ced49: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:28:00.464021 30199 ts_tablet_manager.cc:1888] T 256ae04a48b849c4a9086b6b1ff5df19 P 009d3e7e8a8b4e2497de3a73d01ced49: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240502 21:28:00.464111 30199 log.cc:1196] T 256ae04a48b849c4a9086b6b1ff5df19 P 009d3e7e8a8b4e2497de3a73d01ced49: Deleting WAL directory at /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-2/wal/wals/256ae04a48b849c4a9086b6b1ff5df19
I20240502 21:28:00.464733 23756 catalog_manager.cc:4830] TS 009d3e7e8a8b4e2497de3a73d01ced49 (127.15.255.131:37895): tablet 256ae04a48b849c4a9086b6b1ff5df19 (table rebalance_test_table_6 [id=9aaf56e5a31f42a79b45624cc2c43779]) successfully deleted
I20240502 21:28:00.551194 26279 consensus_queue.cc:237] T d5697d8b06094211b261f144de9d46f8 P a3e500d7077a48acb5edb33f7e178c81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 8, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } } peers { permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: false } }
I20240502 21:28:00.553097 27125 raft_consensus.cc:1261] T d5697d8b06094211b261f144de9d46f8 P 4f58800ba2ed45eabd81c2f2e691aacd [term 2 FOLLOWER]: Refusing update from remote peer a3e500d7077a48acb5edb33f7e178c81: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 21:28:00.553373 24183 raft_consensus.cc:1261] T d5697d8b06094211b261f144de9d46f8 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 2 FOLLOWER]: Refusing update from remote peer a3e500d7077a48acb5edb33f7e178c81: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 21:28:00.553925 30232 consensus_queue.cc:1035] T d5697d8b06094211b261f144de9d46f8 P a3e500d7077a48acb5edb33f7e178c81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 21:28:00.554404 30232 consensus_queue.cc:1035] T d5697d8b06094211b261f144de9d46f8 P a3e500d7077a48acb5edb33f7e178c81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 21:28:00.556216 30231 raft_consensus.cc:2932] T d5697d8b06094211b261f144de9d46f8 P a3e500d7077a48acb5edb33f7e178c81 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 038c5d9d1d7a49c981c4c34fbc00d9e6 (127.15.255.132) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } } peers { permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: false } } }
I20240502 21:28:00.556942 27125 raft_consensus.cc:2932] T d5697d8b06094211b261f144de9d46f8 P 4f58800ba2ed45eabd81c2f2e691aacd [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 038c5d9d1d7a49c981c4c34fbc00d9e6 (127.15.255.132) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } } peers { permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: false } } }
I20240502 21:28:00.560089 23756 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet d5697d8b06094211b261f144de9d46f8 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:28:00.562409 23771 catalog_manager.cc:5484] T d5697d8b06094211b261f144de9d46f8 P a3e500d7077a48acb5edb33f7e178c81 reported cstate change: config changed from index 8 to 10, VOTER 038c5d9d1d7a49c981c4c34fbc00d9e6 (127.15.255.132) evicted. New cstate: current_term: 2 leader_uuid: "a3e500d7077a48acb5edb33f7e178c81" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:28:00.566336 24297 tablet_service.cc:1514] Processing DeleteTablet for tablet d5697d8b06094211b261f144de9d46f8 with delete_type TABLET_DATA_TOMBSTONED (TS 038c5d9d1d7a49c981c4c34fbc00d9e6 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:58268
I20240502 21:28:00.568850 30234 tablet_replica.cc:330] T d5697d8b06094211b261f144de9d46f8 P 038c5d9d1d7a49c981c4c34fbc00d9e6: stopping tablet replica
I20240502 21:28:00.569216 30234 raft_consensus.cc:2229] T d5697d8b06094211b261f144de9d46f8 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 21:28:00.569422 30234 raft_consensus.cc:2258] T d5697d8b06094211b261f144de9d46f8 P 038c5d9d1d7a49c981c4c34fbc00d9e6 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 21:28:00.570194 30234 ts_tablet_manager.cc:1875] T d5697d8b06094211b261f144de9d46f8 P 038c5d9d1d7a49c981c4c34fbc00d9e6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:28:00.571182 24183 raft_consensus.cc:2932] T d5697d8b06094211b261f144de9d46f8 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 038c5d9d1d7a49c981c4c34fbc00d9e6 (127.15.255.132) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } } peers { permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: false } } }
I20240502 21:28:00.580446 30234 ts_tablet_manager.cc:1888] T d5697d8b06094211b261f144de9d46f8 P 038c5d9d1d7a49c981c4c34fbc00d9e6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.9
I20240502 21:28:00.580622 30234 log.cc:1196] T d5697d8b06094211b261f144de9d46f8 P 038c5d9d1d7a49c981c4c34fbc00d9e6: Deleting WAL directory at /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-3/wal/wals/d5697d8b06094211b261f144de9d46f8
I20240502 21:28:00.581522 23755 catalog_manager.cc:4830] TS 038c5d9d1d7a49c981c4c34fbc00d9e6 (127.15.255.132:46813): tablet d5697d8b06094211b261f144de9d46f8 (table rebalance_test_table_9 [id=188ff51c46164f47803c423463c2e6e2]) successfully deleted
I20240502 21:28:00.716878 24297 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:28:00.744596 24162 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 21:28:00.756810 24027 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:28:00.758199 23894 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 21:28:00.771898 26260 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:28:00.806712 27104 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:28:00.814010 26127 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:28:00.820261 25859 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:28:00.878401 25992 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:28:00.936327 26279 consensus_queue.cc:237] T d5697d8b06094211b261f144de9d46f8 P a3e500d7077a48acb5edb33f7e178c81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 8, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } attrs { replace: true } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } } peers { permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: false } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: NON_VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: true } }
I20240502 21:28:00.938452 24183 raft_consensus.cc:1261] T d5697d8b06094211b261f144de9d46f8 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 2 FOLLOWER]: Refusing update from remote peer a3e500d7077a48acb5edb33f7e178c81: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240502 21:28:00.938748 27125 raft_consensus.cc:1261] T d5697d8b06094211b261f144de9d46f8 P 4f58800ba2ed45eabd81c2f2e691aacd [term 2 FOLLOWER]: Refusing update from remote peer a3e500d7077a48acb5edb33f7e178c81: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240502 21:28:00.939709 30168 consensus_queue.cc:1035] T d5697d8b06094211b261f144de9d46f8 P a3e500d7077a48acb5edb33f7e178c81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 21:28:00.940119 30232 consensus_queue.cc:1035] T d5697d8b06094211b261f144de9d46f8 P a3e500d7077a48acb5edb33f7e178c81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 21:28:00.942725 30231 raft_consensus.cc:2932] T d5697d8b06094211b261f144de9d46f8 P a3e500d7077a48acb5edb33f7e178c81 [term 2 LEADER]: Committing config change with OpId 2.11: config changed from index 10 to 11, NON_VOTER e117afbf8819424e8a98476453dfde2a (127.15.255.135) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } attrs { replace: true } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } } peers { permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: false } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: NON_VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: true } } }
I20240502 21:28:00.943368 27127 raft_consensus.cc:2932] T d5697d8b06094211b261f144de9d46f8 P 4f58800ba2ed45eabd81c2f2e691aacd [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, NON_VOTER e117afbf8819424e8a98476453dfde2a (127.15.255.135) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } attrs { replace: true } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } } peers { permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: false } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: NON_VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: true } } }
I20240502 21:28:00.943583 24183 raft_consensus.cc:2932] T d5697d8b06094211b261f144de9d46f8 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, NON_VOTER e117afbf8819424e8a98476453dfde2a (127.15.255.135) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } attrs { replace: true } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } } peers { permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: false } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: NON_VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: true } } }
I20240502 21:28:00.947801 23771 catalog_manager.cc:5484] T d5697d8b06094211b261f144de9d46f8 P a3e500d7077a48acb5edb33f7e178c81 reported cstate change: config changed from index 10 to 11, NON_VOTER e117afbf8819424e8a98476453dfde2a (127.15.255.135) added. New cstate: current_term: 2 leader_uuid: "a3e500d7077a48acb5edb33f7e178c81" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: NON_VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240502 21:28:00.952948 26213 consensus_peers.cc:485] T d5697d8b06094211b261f144de9d46f8 P a3e500d7077a48acb5edb33f7e178c81 -> Peer e117afbf8819424e8a98476453dfde2a (127.15.255.135:32775): Couldn't send request to peer e117afbf8819424e8a98476453dfde2a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d5697d8b06094211b261f144de9d46f8. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:28:01.466588 30314 ts_tablet_manager.cc:902] T d5697d8b06094211b261f144de9d46f8 P e117afbf8819424e8a98476453dfde2a: Initiating tablet copy from peer a3e500d7077a48acb5edb33f7e178c81 (127.15.255.136:43595)
I20240502 21:28:01.467679 30314 tablet_copy_client.cc:308] T d5697d8b06094211b261f144de9d46f8 P e117afbf8819424e8a98476453dfde2a: tablet copy: Beginning tablet copy session from remote peer at address 127.15.255.136:43595
I20240502 21:28:01.473343 26300 tablet_copy_service.cc:140] P a3e500d7077a48acb5edb33f7e178c81: Received BeginTabletCopySession request for tablet d5697d8b06094211b261f144de9d46f8 from peer e117afbf8819424e8a98476453dfde2a ({username='slave'} at 127.15.255.135:58587)
I20240502 21:28:01.473500 26300 tablet_copy_service.cc:161] P a3e500d7077a48acb5edb33f7e178c81: Beginning new tablet copy session on tablet d5697d8b06094211b261f144de9d46f8 from peer e117afbf8819424e8a98476453dfde2a at {username='slave'} at 127.15.255.135:58587: session id = e117afbf8819424e8a98476453dfde2a-d5697d8b06094211b261f144de9d46f8
I20240502 21:28:01.475670 26300 tablet_copy_source_session.cc:205] T d5697d8b06094211b261f144de9d46f8 P a3e500d7077a48acb5edb33f7e178c81: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:28:01.476897 30314 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d5697d8b06094211b261f144de9d46f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:01.481321 30314 tablet_copy_client.cc:783] T d5697d8b06094211b261f144de9d46f8 P e117afbf8819424e8a98476453dfde2a: tablet copy: Starting download of 0 data blocks...
I20240502 21:28:01.481604 30314 tablet_copy_client.cc:647] T d5697d8b06094211b261f144de9d46f8 P e117afbf8819424e8a98476453dfde2a: tablet copy: Starting download of 1 WAL segments...
I20240502 21:28:01.483134 30318 tablet_copy_client.cc:1002] T d5697d8b06094211b261f144de9d46f8 P e117afbf8819424e8a98476453dfde2a: tablet copy: Injecting latency into file download: 1500
I20240502 21:28:02.983515 30314 tablet_copy_client.cc:523] T d5697d8b06094211b261f144de9d46f8 P e117afbf8819424e8a98476453dfde2a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:28:02.986243 30314 tablet_bootstrap.cc:492] T d5697d8b06094211b261f144de9d46f8 P e117afbf8819424e8a98476453dfde2a: Bootstrap starting.
I20240502 21:28:03.020705 30314 tablet_bootstrap.cc:492] T d5697d8b06094211b261f144de9d46f8 P e117afbf8819424e8a98476453dfde2a: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:28:03.021030 30314 tablet_bootstrap.cc:492] T d5697d8b06094211b261f144de9d46f8 P e117afbf8819424e8a98476453dfde2a: Bootstrap complete.
I20240502 21:28:03.021265 30314 ts_tablet_manager.cc:1367] T d5697d8b06094211b261f144de9d46f8 P e117afbf8819424e8a98476453dfde2a: Time spent bootstrapping tablet: real 0.035s user 0.028s sys 0.007s
I20240502 21:28:03.021848 30314 raft_consensus.cc:348] T d5697d8b06094211b261f144de9d46f8 P e117afbf8819424e8a98476453dfde2a [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } attrs { replace: true } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } } peers { permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: false } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: NON_VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: true } }
I20240502 21:28:03.022076 30314 raft_consensus.cc:729] T d5697d8b06094211b261f144de9d46f8 P e117afbf8819424e8a98476453dfde2a [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: e117afbf8819424e8a98476453dfde2a, State: Initialized, Role: LEARNER
I20240502 21:28:03.022253 30314 consensus_queue.cc:260] T d5697d8b06094211b261f144de9d46f8 P e117afbf8819424e8a98476453dfde2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 2.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } attrs { replace: true } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } } peers { permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: false } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: NON_VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: true } }
I20240502 21:28:03.023011 30314 ts_tablet_manager.cc:1398] T d5697d8b06094211b261f144de9d46f8 P e117afbf8819424e8a98476453dfde2a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:28:03.023804 26300 tablet_copy_service.cc:342] P a3e500d7077a48acb5edb33f7e178c81: Request end of tablet copy session e117afbf8819424e8a98476453dfde2a-d5697d8b06094211b261f144de9d46f8 received from {username='slave'} at 127.15.255.135:58587
I20240502 21:28:03.023938 26300 tablet_copy_service.cc:433] P a3e500d7077a48acb5edb33f7e178c81: ending tablet copy session e117afbf8819424e8a98476453dfde2a-d5697d8b06094211b261f144de9d46f8 on tablet d5697d8b06094211b261f144de9d46f8 with peer e117afbf8819424e8a98476453dfde2a
I20240502 21:28:03.426311 26147 raft_consensus.cc:1203] T d5697d8b06094211b261f144de9d46f8 P e117afbf8819424e8a98476453dfde2a [term 2 LEARNER]: Deduplicated request from leader. Original: 2.10->[2.11-2.11] Dedup: 2.11->[]
I20240502 21:28:03.859133 30388 raft_consensus.cc:1050] T d5697d8b06094211b261f144de9d46f8 P a3e500d7077a48acb5edb33f7e178c81: attempting to promote NON_VOTER e117afbf8819424e8a98476453dfde2a to VOTER
I20240502 21:28:03.859833 30388 consensus_queue.cc:237] T d5697d8b06094211b261f144de9d46f8 P a3e500d7077a48acb5edb33f7e178c81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 2.11, Last appended by leader: 8, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } attrs { replace: true } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } } peers { permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: false } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } }
I20240502 21:28:03.862474 24183 raft_consensus.cc:1261] T d5697d8b06094211b261f144de9d46f8 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 2 FOLLOWER]: Refusing update from remote peer a3e500d7077a48acb5edb33f7e178c81: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20240502 21:28:03.862766 26147 raft_consensus.cc:1261] T d5697d8b06094211b261f144de9d46f8 P e117afbf8819424e8a98476453dfde2a [term 2 LEARNER]: Refusing update from remote peer a3e500d7077a48acb5edb33f7e178c81: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20240502 21:28:03.863021 27127 raft_consensus.cc:1261] T d5697d8b06094211b261f144de9d46f8 P 4f58800ba2ed45eabd81c2f2e691aacd [term 2 FOLLOWER]: Refusing update from remote peer a3e500d7077a48acb5edb33f7e178c81: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20240502 21:28:03.863019 30389 consensus_queue.cc:1035] T d5697d8b06094211b261f144de9d46f8 P a3e500d7077a48acb5edb33f7e178c81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 21:28:03.863440 30338 consensus_queue.cc:1035] T d5697d8b06094211b261f144de9d46f8 P a3e500d7077a48acb5edb33f7e178c81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 21:28:03.863770 30391 consensus_queue.cc:1035] T d5697d8b06094211b261f144de9d46f8 P a3e500d7077a48acb5edb33f7e178c81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 21:28:03.866710 30338 raft_consensus.cc:2932] T d5697d8b06094211b261f144de9d46f8 P a3e500d7077a48acb5edb33f7e178c81 [term 2 LEADER]: Committing config change with OpId 2.12: config changed from index 11 to 12, e117afbf8819424e8a98476453dfde2a (127.15.255.135) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } attrs { replace: true } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } } peers { permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: false } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } } }
I20240502 21:28:03.867333 27127 raft_consensus.cc:2932] T d5697d8b06094211b261f144de9d46f8 P 4f58800ba2ed45eabd81c2f2e691aacd [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, e117afbf8819424e8a98476453dfde2a (127.15.255.135) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } attrs { replace: true } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } } peers { permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: false } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } } }
I20240502 21:28:03.868889 26147 raft_consensus.cc:2932] T d5697d8b06094211b261f144de9d46f8 P e117afbf8819424e8a98476453dfde2a [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, e117afbf8819424e8a98476453dfde2a (127.15.255.135) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } attrs { replace: true } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } } peers { permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: false } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } } }
I20240502 21:28:03.871013 23770 catalog_manager.cc:5484] T d5697d8b06094211b261f144de9d46f8 P a3e500d7077a48acb5edb33f7e178c81 reported cstate change: config changed from index 11 to 12, e117afbf8819424e8a98476453dfde2a (127.15.255.135) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "a3e500d7077a48acb5edb33f7e178c81" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:28:03.874197 24183 raft_consensus.cc:2932] T d5697d8b06094211b261f144de9d46f8 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, e117afbf8819424e8a98476453dfde2a (127.15.255.135) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "009d3e7e8a8b4e2497de3a73d01ced49" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 37895 } attrs { replace: true } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } } peers { permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: false } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } } }
I20240502 21:28:03.879091 26279 consensus_queue.cc:237] T d5697d8b06094211b261f144de9d46f8 P a3e500d7077a48acb5edb33f7e178c81 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 2.12, Last appended by leader: 8, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } } peers { permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: false } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } }
I20240502 21:28:03.880595 26147 raft_consensus.cc:1261] T d5697d8b06094211b261f144de9d46f8 P e117afbf8819424e8a98476453dfde2a [term 2 FOLLOWER]: Refusing update from remote peer a3e500d7077a48acb5edb33f7e178c81: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20240502 21:28:03.880671 27127 raft_consensus.cc:1261] T d5697d8b06094211b261f144de9d46f8 P 4f58800ba2ed45eabd81c2f2e691aacd [term 2 FOLLOWER]: Refusing update from remote peer a3e500d7077a48acb5edb33f7e178c81: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20240502 21:28:03.881198 30388 consensus_queue.cc:1035] T d5697d8b06094211b261f144de9d46f8 P a3e500d7077a48acb5edb33f7e178c81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240502 21:28:03.882468 30389 consensus_queue.cc:1035] T d5697d8b06094211b261f144de9d46f8 P a3e500d7077a48acb5edb33f7e178c81 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240502 21:28:03.883322 30388 raft_consensus.cc:2932] T d5697d8b06094211b261f144de9d46f8 P a3e500d7077a48acb5edb33f7e178c81 [term 2 LEADER]: Committing config change with OpId 2.13: config changed from index 12 to 13, VOTER 009d3e7e8a8b4e2497de3a73d01ced49 (127.15.255.131) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } } peers { permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: false } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } } }
I20240502 21:28:03.883989 27127 raft_consensus.cc:2932] T d5697d8b06094211b261f144de9d46f8 P 4f58800ba2ed45eabd81c2f2e691aacd [term 2 FOLLOWER]: Committing config change with OpId 2.13: config changed from index 12 to 13, VOTER 009d3e7e8a8b4e2497de3a73d01ced49 (127.15.255.131) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } } peers { permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: false } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } } }
I20240502 21:28:03.886586 23756 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet d5697d8b06094211b261f144de9d46f8 with cas_config_opid_index 12: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:28:03.887284 26147 raft_consensus.cc:2932] T d5697d8b06094211b261f144de9d46f8 P e117afbf8819424e8a98476453dfde2a [term 2 FOLLOWER]: Committing config change with OpId 2.13: config changed from index 12 to 13, VOTER 009d3e7e8a8b4e2497de3a73d01ced49 (127.15.255.131) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } } peers { permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: false } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } } }
I20240502 21:28:03.889554 23770 catalog_manager.cc:5484] T d5697d8b06094211b261f144de9d46f8 P a3e500d7077a48acb5edb33f7e178c81 reported cstate change: config changed from index 12 to 13, VOTER 009d3e7e8a8b4e2497de3a73d01ced49 (127.15.255.131) evicted. New cstate: current_term: 2 leader_uuid: "a3e500d7077a48acb5edb33f7e178c81" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:28:03.893523 24162 tablet_service.cc:1514] Processing DeleteTablet for tablet d5697d8b06094211b261f144de9d46f8 with delete_type TABLET_DATA_TOMBSTONED (TS 009d3e7e8a8b4e2497de3a73d01ced49 not found in new config with opid_index 13) from {username='slave'} at 127.0.0.1:33102
I20240502 21:28:03.894382 30402 tablet_replica.cc:330] T d5697d8b06094211b261f144de9d46f8 P 009d3e7e8a8b4e2497de3a73d01ced49: stopping tablet replica
I20240502 21:28:03.894553 30402 raft_consensus.cc:2229] T d5697d8b06094211b261f144de9d46f8 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 21:28:03.894781 30402 raft_consensus.cc:2258] T d5697d8b06094211b261f144de9d46f8 P 009d3e7e8a8b4e2497de3a73d01ced49 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 21:28:03.895586 30402 ts_tablet_manager.cc:1875] T d5697d8b06094211b261f144de9d46f8 P 009d3e7e8a8b4e2497de3a73d01ced49: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:28:03.899624 30402 ts_tablet_manager.cc:1888] T d5697d8b06094211b261f144de9d46f8 P 009d3e7e8a8b4e2497de3a73d01ced49: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.12
I20240502 21:28:03.899739 30402 log.cc:1196] T d5697d8b06094211b261f144de9d46f8 P 009d3e7e8a8b4e2497de3a73d01ced49: Deleting WAL directory at /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-2/wal/wals/d5697d8b06094211b261f144de9d46f8
I20240502 21:28:03.900465 23756 catalog_manager.cc:4830] TS 009d3e7e8a8b4e2497de3a73d01ced49 (127.15.255.131:37895): tablet d5697d8b06094211b261f144de9d46f8 (table rebalance_test_table_9 [id=188ff51c46164f47803c423463c2e6e2]) successfully deleted
I20240502 21:28:04.039623 24162 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 21:28:04.054085 26260 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:28:04.069530 24297 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:28:04.118137 26127 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:28:04.122874 25859 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:28:04.125962 23894 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 21:28:04.139972 25992 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:28:04.142863 27104 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:28:04.161769 24027 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:28:04.364048 24297 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:28:04.364337 24162 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 21:28:04.382261 23894 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 21:28:04.383878 27104 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:28:04.387794 26260 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:28:04.400260 26127 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:28:04.413077 24027 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:28:04.450686 25859 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:28:04.471258 25992 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:28:05.345059 24162 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 21:28:05.362023 27104 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:28:05.363660 24297 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:28:05.372690 23894 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 21:28:05.427217 26127 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:28:05.430967 25859 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:28:05.445753 24027 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:28:05.461807 26260 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:28:05.488374 25992 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:28:05.682813 24162 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 21:28:05.686430 23894 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 21:28:05.701303 26260 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:28:05.702049 24027 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:28:05.740147 27104 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:28:05.746857 26127 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:28:05.772181 24297 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:28:05.815193 25992 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:28:05.826300 25859 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:28:06.026367 24162 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 21:28:06.037277 27104 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:28:06.041318 24297 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:28:06.045907 23894 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 21:28:06.050565 24027 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:28:06.067126 26260 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:28:06.072824 25992 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:28:06.084506 25859 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:28:06.087112 26127 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:28:06.446323 24297 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:28:06.446533 24162 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 21:28:06.463433 23894 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 21:28:06.470427 26260 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:28:06.472785 26127 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:28:06.492962 24027 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:28:06.494299 27104 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:28:06.496647 25992 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:28:06.549196 25859 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
953efe8866e34dff88dabafc97c90783 | 127.15.255.190:33301 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.15.255.148:46015 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+---------+----------+----------------+-----------------
009d3e7e8a8b4e2497de3a73d01ced49 | 127.15.255.131:37895 | HEALTHY | <none> | 7 | 0
038c5d9d1d7a49c981c4c34fbc00d9e6 | 127.15.255.132:46813 | HEALTHY | <none> | 4 | 0
4f58800ba2ed45eabd81c2f2e691aacd | 127.15.255.137:40161 | HEALTHY | <none> | 1 | 0
6810a866decb4f5089f7d35c870e4204 | 127.15.255.129:40435 | HEALTHY | <none> | 7 | 0
6c194a6db6c145acafc0f3068b8ce1fd | 127.15.255.130:41235 | HEALTHY | <none> | 6 | 0
a3e500d7077a48acb5edb33f7e178c81 | 127.15.255.136:43595 | HEALTHY | <none> | 3 | 0
e117afbf8819424e8a98476453dfde2a | 127.15.255.135:32775 | HEALTHY | <none> | 0 | 0
ee27c3f9e0ec46caa742ac386d6b7c20 | 127.15.255.133:40841 | HEALTHY | <none> | 1 | 0
fb6004b8640a4906aa0cfe63e383d40e | 127.15.255.134:32825 | HEALTHY | <none> | 1 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 9
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
---------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 9 server(s) checked
local_ip_for_outbound_sockets | 127.15.255.129 | experimental | 127.15.255.129:40435
local_ip_for_outbound_sockets | 127.15.255.130 | experimental | 127.15.255.130:41235
local_ip_for_outbound_sockets | 127.15.255.131 | experimental | 127.15.255.131:37895
local_ip_for_outbound_sockets | 127.15.255.132 | experimental | 127.15.255.132:46813
local_ip_for_outbound_sockets | 127.15.255.133 | experimental | 127.15.255.133:40841
local_ip_for_outbound_sockets | 127.15.255.134 | experimental | 127.15.255.134:32825
local_ip_for_outbound_sockets | 127.15.255.135 | experimental | 127.15.255.135:32775
local_ip_for_outbound_sockets | 127.15.255.136 | experimental | 127.15.255.136:43595
local_ip_for_outbound_sockets | 127.15.255.137 | experimental | 127.15.255.137:40161
never_fsync | true | unsafe,advanced | all 9 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 9 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 9 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 9 server(s) checked
server_dump_info_format | pb | hidden | all 9 server(s) checked
server_dump_info_path | /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.15.255.129:40435
server_dump_info_path | /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.15.255.130:41235
server_dump_info_path | /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.15.255.131:37895
server_dump_info_path | /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.15.255.132:46813
server_dump_info_path | /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.15.255.133:40841
server_dump_info_path | /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.15.255.134:32825
server_dump_info_path | /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-6/data/info.pb | hidden | 127.15.255.135:32775
server_dump_info_path | /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-7/data/info.pb | hidden | 127.15.255.136:43595
server_dump_info_path | /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.TserverAddedDuringRebalancingTest.TserverStarts_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-8/data/info.pb | hidden | 127.15.255.137:40161
tablet_copy_download_file_inject_latency_ms | 1500 | hidden | all 9 server(s) checked
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.15.255.148:46015 | all 9 server(s) checked
time_source | builtin | all 9 server(s) checked
Version Summary
Version | Servers
-----------------+--------------------------
1.18.0-SNAPSHOT | all 10 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------------------+----+---------+---------------+---------+------------+------------------+-------------
rebalance_test_table_0 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_1 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_2 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_3 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_4 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_5 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_6 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_7 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_8 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_9 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 10
First Quartile | 10
Median | 10
Third Quartile | 10
Maximum | 10
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 9
Tables | 10
Tablets | 30
Replicas | 90
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20240502 21:28:06.623775 16382 log_verifier.cc:126] Checking tablet 0b3c514213bb40ca98955d6409a3b372
I20240502 21:28:06.665355 16382 log_verifier.cc:177] Verified matching terms for 10 ops in tablet 0b3c514213bb40ca98955d6409a3b372
I20240502 21:28:06.665513 16382 log_verifier.cc:126] Checking tablet 11cb9a2fbda946fb9f303f2c465bccb6
I20240502 21:28:06.737391 16382 log_verifier.cc:177] Verified matching terms for 9 ops in tablet 11cb9a2fbda946fb9f303f2c465bccb6
I20240502 21:28:06.737479 16382 log_verifier.cc:126] Checking tablet 16bb47f03a2f46fdb7344e018f07fdcb
I20240502 21:28:06.772033 16382 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 16bb47f03a2f46fdb7344e018f07fdcb
I20240502 21:28:06.772130 16382 log_verifier.cc:126] Checking tablet 256ae04a48b849c4a9086b6b1ff5df19
I20240502 21:28:06.808707 16382 log_verifier.cc:177] Verified matching terms for 9 ops in tablet 256ae04a48b849c4a9086b6b1ff5df19
I20240502 21:28:06.808846 16382 log_verifier.cc:126] Checking tablet 30236241b670443489240f191d9b9d50
I20240502 21:28:06.853626 16382 log_verifier.cc:177] Verified matching terms for 10 ops in tablet 30236241b670443489240f191d9b9d50
I20240502 21:28:06.853724 16382 log_verifier.cc:126] Checking tablet 3cb3e62ca0e94e659efb3b7b0c2c9c91
I20240502 21:28:06.888597 16382 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 3cb3e62ca0e94e659efb3b7b0c2c9c91
I20240502 21:28:06.888715 16382 log_verifier.cc:126] Checking tablet 4eebf4508766421eb6c547a0fbbc28b1
I20240502 21:28:06.926549 16382 log_verifier.cc:177] Verified matching terms for 9 ops in tablet 4eebf4508766421eb6c547a0fbbc28b1
I20240502 21:28:06.926652 16382 log_verifier.cc:126] Checking tablet 52f621ea3f174490948eae854c706a92
I20240502 21:28:06.999796 16382 log_verifier.cc:177] Verified matching terms for 10 ops in tablet 52f621ea3f174490948eae854c706a92
I20240502 21:28:06.999890 16382 log_verifier.cc:126] Checking tablet 549666d19e724a96801db183a07809d8
I20240502 21:28:07.038460 16382 log_verifier.cc:177] Verified matching terms for 9 ops in tablet 549666d19e724a96801db183a07809d8
I20240502 21:28:07.038564 16382 log_verifier.cc:126] Checking tablet 6bd759d700a24e9f8ad11f243cf7ebfa
I20240502 21:28:07.074688 16382 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 6bd759d700a24e9f8ad11f243cf7ebfa
I20240502 21:28:07.074810 16382 log_verifier.cc:126] Checking tablet 77fc7226878f44d3884ccb1fc41c44a4
I20240502 21:28:07.123628 16382 log_verifier.cc:177] Verified matching terms for 9 ops in tablet 77fc7226878f44d3884ccb1fc41c44a4
I20240502 21:28:07.123747 16382 log_verifier.cc:126] Checking tablet 83b95f3ae7564b81b31a2f5673ee22bb
I20240502 21:28:07.163739 16382 log_verifier.cc:177] Verified matching terms for 9 ops in tablet 83b95f3ae7564b81b31a2f5673ee22bb
I20240502 21:28:07.163851 16382 log_verifier.cc:126] Checking tablet 85e77d5faa8346cd8a99dcbcdf75f250
I20240502 21:28:07.201002 16382 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 85e77d5faa8346cd8a99dcbcdf75f250
I20240502 21:28:07.201179 16382 log_verifier.cc:126] Checking tablet 89f86d58fe414b579845214534d338d8
I20240502 21:28:07.273540 16382 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 89f86d58fe414b579845214534d338d8
I20240502 21:28:07.273648 16382 log_verifier.cc:126] Checking tablet 8af15fc41246471db31f7824985b5903
I20240502 21:28:07.313663 16382 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 8af15fc41246471db31f7824985b5903
I20240502 21:28:07.313760 16382 log_verifier.cc:126] Checking tablet 8eba3872a77046ecb9c9477d4cc0e18b
I20240502 21:28:07.348011 16382 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 8eba3872a77046ecb9c9477d4cc0e18b
I20240502 21:28:07.348104 16382 log_verifier.cc:126] Checking tablet 8f14db150fda449e962be2466f1d2a0e
I20240502 21:28:07.385520 16382 log_verifier.cc:177] Verified matching terms for 9 ops in tablet 8f14db150fda449e962be2466f1d2a0e
I20240502 21:28:07.385625 16382 log_verifier.cc:126] Checking tablet 906cfe7530754271bb1ee57c470109bd
I20240502 21:28:07.422181 16382 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 906cfe7530754271bb1ee57c470109bd
I20240502 21:28:07.422272 16382 log_verifier.cc:126] Checking tablet a274546743354e79bffa26ae7000e895
I20240502 21:28:07.459452 16382 log_verifier.cc:177] Verified matching terms for 9 ops in tablet a274546743354e79bffa26ae7000e895
I20240502 21:28:07.459556 16382 log_verifier.cc:126] Checking tablet a2a0537a60f849daae2061c2b0c4c89f
I20240502 21:28:07.533587 16382 log_verifier.cc:177] Verified matching terms for 6 ops in tablet a2a0537a60f849daae2061c2b0c4c89f
I20240502 21:28:07.533687 16382 log_verifier.cc:126] Checking tablet adf80b9d71314010a7a4cf052fd8a511
I20240502 21:28:07.579001 16382 log_verifier.cc:177] Verified matching terms for 9 ops in tablet adf80b9d71314010a7a4cf052fd8a511
I20240502 21:28:07.579107 16382 log_verifier.cc:126] Checking tablet c9b5be01b5e941b1848df79c7a9db348
I20240502 21:28:07.616714 16382 log_verifier.cc:177] Verified matching terms for 9 ops in tablet c9b5be01b5e941b1848df79c7a9db348
I20240502 21:28:07.616823 16382 log_verifier.cc:126] Checking tablet cca45c89b9fe40e7800d95c1598a9e52
I20240502 21:28:07.653296 16382 log_verifier.cc:177] Verified matching terms for 9 ops in tablet cca45c89b9fe40e7800d95c1598a9e52
I20240502 21:28:07.653407 16382 log_verifier.cc:126] Checking tablet d5697d8b06094211b261f144de9d46f8
I20240502 21:28:07.693143 16382 log_verifier.cc:177] Verified matching terms for 13 ops in tablet d5697d8b06094211b261f144de9d46f8
I20240502 21:28:07.693271 16382 log_verifier.cc:126] Checking tablet d79eec4f59634df480161815d888bcac
I20240502 21:28:07.735105 16382 log_verifier.cc:177] Verified matching terms for 6 ops in tablet d79eec4f59634df480161815d888bcac
I20240502 21:28:07.735215 16382 log_verifier.cc:126] Checking tablet dc0e070513cc46ca82c1946423ad91c5
I20240502 21:28:07.804502 16382 log_verifier.cc:177] Verified matching terms for 9 ops in tablet dc0e070513cc46ca82c1946423ad91c5
I20240502 21:28:07.804600 16382 log_verifier.cc:126] Checking tablet f84bbe304597450c978ae28e0c4e7a3e
I20240502 21:28:07.850919 16382 log_verifier.cc:177] Verified matching terms for 10 ops in tablet f84bbe304597450c978ae28e0c4e7a3e
I20240502 21:28:07.851022 16382 log_verifier.cc:126] Checking tablet fcc9cbeae6f343af9f84c73349f3dadb
I20240502 21:28:07.885469 16382 log_verifier.cc:177] Verified matching terms for 7 ops in tablet fcc9cbeae6f343af9f84c73349f3dadb
I20240502 21:28:07.885581 16382 log_verifier.cc:126] Checking tablet fe044374ad7b4999afc72ab812466749
I20240502 21:28:07.921651 16382 log_verifier.cc:177] Verified matching terms for 9 ops in tablet fe044374ad7b4999afc72ab812466749
I20240502 21:28:07.921761 16382 log_verifier.cc:126] Checking tablet fef55863beb6435dbb25bfb6d71e3cd7
I20240502 21:28:07.959249 16382 log_verifier.cc:177] Verified matching terms for 9 ops in tablet fef55863beb6435dbb25bfb6d71e3cd7
I20240502 21:28:07.960104 16382 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvNciQZ/build/asan/bin/kudu pid 23831
I20240502 21:28:07.966784 23957 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 21:28:08.207427 16382 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvNciQZ/build/asan/bin/kudu with pid 23831
W20240502 21:28:08.242599 25814 connection.cc:535] client connection to 127.15.255.129:40435 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 21:28:08.242872 25814 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 21:28:08.243649 26213 connection.cc:535] server connection from 127.15.255.129:44953 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 21:28:08.244037 16382 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvNciQZ/build/asan/bin/kudu pid 23965
W20240502 21:28:08.246438 25814 consensus_peers.cc:485] T 52f621ea3f174490948eae854c706a92 P ee27c3f9e0ec46caa742ac386d6b7c20 -> Peer 6810a866decb4f5089f7d35c870e4204 (127.15.255.129:40435): Couldn't send request to peer 6810a866decb4f5089f7d35c870e4204. Status: Network error: Client connection negotiation failed: client connection to 127.15.255.129:40435: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 21:28:08.250450 24092 generic_service.cc:196] Checking for leaks (request via RPC)
W20240502 21:28:08.250614 24118 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.255.129:40435: connect: Connection refused (error 111)
W20240502 21:28:08.252207 24118 consensus_peers.cc:485] T 6bd759d700a24e9f8ad11f243cf7ebfa P 009d3e7e8a8b4e2497de3a73d01ced49 -> Peer 6810a866decb4f5089f7d35c870e4204 (127.15.255.129:40435): Couldn't send request to peer 6810a866decb4f5089f7d35c870e4204. Status: Network error: Client connection negotiation failed: client connection to 127.15.255.129:40435: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:28:08.322129 24253 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.255.129:40435: connect: Connection refused (error 111)
W20240502 21:28:08.323666 24253 consensus_peers.cc:485] T 8eba3872a77046ecb9c9477d4cc0e18b P 038c5d9d1d7a49c981c4c34fbc00d9e6 -> Peer 6810a866decb4f5089f7d35c870e4204 (127.15.255.129:40435): Couldn't send request to peer 6810a866decb4f5089f7d35c870e4204. Status: Network error: Client connection negotiation failed: client connection to 127.15.255.129:40435: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 21:28:08.475090 16382 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvNciQZ/build/asan/bin/kudu with pid 23965
W20240502 21:28:08.506366 26081 connection.cc:535] server connection from 127.15.255.130:38799 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 21:28:08.507020 16382 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvNciQZ/build/asan/bin/kudu pid 24100
W20240502 21:28:08.507164 24118 connection.cc:535] server connection from 127.15.255.130:37311 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 21:28:08.513257 24226 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 21:28:08.736021 16382 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvNciQZ/build/asan/bin/kudu with pid 24100
I20240502 21:28:08.767954 16382 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvNciQZ/build/asan/bin/kudu pid 24234
I20240502 21:28:08.773998 24358 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 21:28:09.002264 16382 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvNciQZ/build/asan/bin/kudu with pid 24234
W20240502 21:28:09.035800 25811 connection.cc:535] server connection from 127.15.255.132:42901 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 21:28:09.036720 16382 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvNciQZ/build/asan/bin/kudu pid 25795
I20240502 21:28:09.042865 25918 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 21:28:09.100203 28380 raft_consensus.cc:482] T 906cfe7530754271bb1ee57c470109bd P fb6004b8640a4906aa0cfe63e383d40e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6810a866decb4f5089f7d35c870e4204)
I20240502 21:28:09.100456 28380 raft_consensus.cc:504] T 906cfe7530754271bb1ee57c470109bd P fb6004b8640a4906aa0cfe63e383d40e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "038c5d9d1d7a49c981c4c34fbc00d9e6" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 46813 } } peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: false } }
I20240502 21:28:09.101526 28380 leader_election.cc:288] T 906cfe7530754271bb1ee57c470109bd P fb6004b8640a4906aa0cfe63e383d40e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 038c5d9d1d7a49c981c4c34fbc00d9e6 (127.15.255.132:46813), 6810a866decb4f5089f7d35c870e4204 (127.15.255.129:40435)
W20240502 21:28:09.102686 25945 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.255.132:46813: connect: Connection refused (error 111)
W20240502 21:28:09.104467 25945 leader_election.cc:334] T 906cfe7530754271bb1ee57c470109bd P fb6004b8640a4906aa0cfe63e383d40e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 038c5d9d1d7a49c981c4c34fbc00d9e6 (127.15.255.132:46813): Network error: Client connection negotiation failed: client connection to 127.15.255.132:46813: connect: Connection refused (error 111)
W20240502 21:28:09.104619 25947 leader_election.cc:334] T 906cfe7530754271bb1ee57c470109bd P fb6004b8640a4906aa0cfe63e383d40e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6810a866decb4f5089f7d35c870e4204 (127.15.255.129:40435): Network error: Client connection negotiation failed: client connection to 127.15.255.129:40435: connect: Connection refused (error 111)
I20240502 21:28:09.104763 25947 leader_election.cc:302] T 906cfe7530754271bb1ee57c470109bd P fb6004b8640a4906aa0cfe63e383d40e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fb6004b8640a4906aa0cfe63e383d40e; no voters: 038c5d9d1d7a49c981c4c34fbc00d9e6, 6810a866decb4f5089f7d35c870e4204
I20240502 21:28:09.105029 28380 raft_consensus.cc:2726] T 906cfe7530754271bb1ee57c470109bd P fb6004b8640a4906aa0cfe63e383d40e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 21:28:09.164199 30391 raft_consensus.cc:482] T 4eebf4508766421eb6c547a0fbbc28b1 P a3e500d7077a48acb5edb33f7e178c81 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6810a866decb4f5089f7d35c870e4204)
I20240502 21:28:09.164465 30391 raft_consensus.cc:504] T 4eebf4508766421eb6c547a0fbbc28b1 P a3e500d7077a48acb5edb33f7e178c81 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } }
I20240502 21:28:09.165243 30391 leader_election.cc:288] T 4eebf4508766421eb6c547a0fbbc28b1 P a3e500d7077a48acb5edb33f7e178c81 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6810a866decb4f5089f7d35c870e4204 (127.15.255.129:40435), e117afbf8819424e8a98476453dfde2a (127.15.255.135:32775)
I20240502 21:28:09.165807 30492 raft_consensus.cc:482] T 8f14db150fda449e962be2466f1d2a0e P 4f58800ba2ed45eabd81c2f2e691aacd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6810a866decb4f5089f7d35c870e4204)
I20240502 21:28:09.166031 26147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4eebf4508766421eb6c547a0fbbc28b1" candidate_uuid: "a3e500d7077a48acb5edb33f7e178c81" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "e117afbf8819424e8a98476453dfde2a" is_pre_election: true
I20240502 21:28:09.166006 30492 raft_consensus.cc:504] T 8f14db150fda449e962be2466f1d2a0e P 4f58800ba2ed45eabd81c2f2e691aacd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: false } } peers { permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: false } }
I20240502 21:28:09.167047 30492 leader_election.cc:288] T 8f14db150fda449e962be2466f1d2a0e P 4f58800ba2ed45eabd81c2f2e691aacd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6810a866decb4f5089f7d35c870e4204 (127.15.255.129:40435), ee27c3f9e0ec46caa742ac386d6b7c20 (127.15.255.133:40841)
W20240502 21:28:09.170091 27033 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.255.129:40435: connect: Connection refused (error 111)
W20240502 21:28:09.169795 26214 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.255.129:40435: connect: Connection refused (error 111)
W20240502 21:28:09.171520 27033 leader_election.cc:334] T 8f14db150fda449e962be2466f1d2a0e P 4f58800ba2ed45eabd81c2f2e691aacd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6810a866decb4f5089f7d35c870e4204 (127.15.255.129:40435): Network error: Client connection negotiation failed: client connection to 127.15.255.129:40435: connect: Connection refused (error 111)
W20240502 21:28:09.171842 26214 leader_election.cc:334] T 4eebf4508766421eb6c547a0fbbc28b1 P a3e500d7077a48acb5edb33f7e178c81 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6810a866decb4f5089f7d35c870e4204 (127.15.255.129:40435): Network error: Client connection negotiation failed: client connection to 127.15.255.129:40435: connect: Connection refused (error 111)
I20240502 21:28:09.171970 26214 leader_election.cc:302] T 4eebf4508766421eb6c547a0fbbc28b1 P a3e500d7077a48acb5edb33f7e178c81 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a3e500d7077a48acb5edb33f7e178c81; no voters: 6810a866decb4f5089f7d35c870e4204, e117afbf8819424e8a98476453dfde2a
I20240502 21:28:09.172240 30391 raft_consensus.cc:2726] T 4eebf4508766421eb6c547a0fbbc28b1 P a3e500d7077a48acb5edb33f7e178c81 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 21:28:09.193639 30564 raft_consensus.cc:482] T 8f14db150fda449e962be2466f1d2a0e P ee27c3f9e0ec46caa742ac386d6b7c20 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6810a866decb4f5089f7d35c870e4204)
I20240502 21:28:09.193882 30564 raft_consensus.cc:504] T 8f14db150fda449e962be2466f1d2a0e P ee27c3f9e0ec46caa742ac386d6b7c20 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: false } } peers { permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: false } }
I20240502 21:28:09.194988 30564 leader_election.cc:288] T 8f14db150fda449e962be2466f1d2a0e P ee27c3f9e0ec46caa742ac386d6b7c20 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6810a866decb4f5089f7d35c870e4204 (127.15.255.129:40435), 4f58800ba2ed45eabd81c2f2e691aacd (127.15.255.137:40161)
I20240502 21:28:09.196540 16382 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvNciQZ/build/asan/bin/kudu with pid 25795
I20240502 21:28:09.196570 27127 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f14db150fda449e962be2466f1d2a0e" candidate_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" is_pre_election: true
I20240502 21:28:09.196822 27127 raft_consensus.cc:2454] T 8f14db150fda449e962be2466f1d2a0e P 4f58800ba2ed45eabd81c2f2e691aacd [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee27c3f9e0ec46caa742ac386d6b7c20 in term 1.
W20240502 21:28:09.208832 25946 consensus_peers.cc:485] T 89f86d58fe414b579845214534d338d8 P fb6004b8640a4906aa0cfe63e383d40e -> Peer 009d3e7e8a8b4e2497de3a73d01ced49 (127.15.255.131:37895): Couldn't send request to peer 009d3e7e8a8b4e2497de3a73d01ced49. Status: Network error: Client connection negotiation failed: client connection to 127.15.255.131:37895: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:28:09.216954 26212 consensus_peers.cc:485] T 30236241b670443489240f191d9b9d50 P a3e500d7077a48acb5edb33f7e178c81 -> Peer 038c5d9d1d7a49c981c4c34fbc00d9e6 (127.15.255.132:46813): Couldn't send request to peer 038c5d9d1d7a49c981c4c34fbc00d9e6. Status: Network error: Client connection negotiation failed: client connection to 127.15.255.132:46813: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 21:28:09.219919 27028 connection.cc:535] server connection from 127.15.255.133:36401 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 21:28:09.220113 27024 connection.cc:535] client connection to 127.15.255.133:40841 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 21:28:09.220552 16382 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvNciQZ/build/asan/bin/kudu pid 25928
W20240502 21:28:09.221122 27024 leader_election.cc:334] T 8f14db150fda449e962be2466f1d2a0e P 4f58800ba2ed45eabd81c2f2e691aacd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ee27c3f9e0ec46caa742ac386d6b7c20 (127.15.255.133:40841): Network error: Client connection negotiation failed: client connection to 127.15.255.133:40841: connect: Connection refused (error 111)
I20240502 21:28:09.221261 27024 leader_election.cc:302] T 8f14db150fda449e962be2466f1d2a0e P 4f58800ba2ed45eabd81c2f2e691aacd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4f58800ba2ed45eabd81c2f2e691aacd; no voters: 6810a866decb4f5089f7d35c870e4204, ee27c3f9e0ec46caa742ac386d6b7c20
I20240502 21:28:09.221602 30492 raft_consensus.cc:2726] T 8f14db150fda449e962be2466f1d2a0e P 4f58800ba2ed45eabd81c2f2e691aacd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 21:28:09.226959 26053 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 21:28:09.252488 30492 raft_consensus.cc:482] T 256ae04a48b849c4a9086b6b1ff5df19 P 4f58800ba2ed45eabd81c2f2e691aacd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6810a866decb4f5089f7d35c870e4204)
I20240502 21:28:09.252691 30492 raft_consensus.cc:504] T 256ae04a48b849c4a9086b6b1ff5df19 P 4f58800ba2ed45eabd81c2f2e691aacd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } } peers { permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: false } }
I20240502 21:28:09.253558 30492 leader_election.cc:288] T 256ae04a48b849c4a9086b6b1ff5df19 P 4f58800ba2ed45eabd81c2f2e691aacd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6810a866decb4f5089f7d35c870e4204 (127.15.255.129:40435), e117afbf8819424e8a98476453dfde2a (127.15.255.135:32775)
W20240502 21:28:09.254877 27033 leader_election.cc:334] T 256ae04a48b849c4a9086b6b1ff5df19 P 4f58800ba2ed45eabd81c2f2e691aacd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6810a866decb4f5089f7d35c870e4204 (127.15.255.129:40435): Network error: Client connection negotiation failed: client connection to 127.15.255.129:40435: connect: Connection refused (error 111)
I20240502 21:28:09.259438 26147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256ae04a48b849c4a9086b6b1ff5df19" candidate_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "e117afbf8819424e8a98476453dfde2a" is_pre_election: true
I20240502 21:28:09.259915 27028 leader_election.cc:302] T 256ae04a48b849c4a9086b6b1ff5df19 P 4f58800ba2ed45eabd81c2f2e691aacd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4f58800ba2ed45eabd81c2f2e691aacd; no voters: 6810a866decb4f5089f7d35c870e4204, e117afbf8819424e8a98476453dfde2a
I20240502 21:28:09.260200 30492 raft_consensus.cc:2726] T 256ae04a48b849c4a9086b6b1ff5df19 P 4f58800ba2ed45eabd81c2f2e691aacd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 21:28:09.269865 30719 raft_consensus.cc:482] T 256ae04a48b849c4a9086b6b1ff5df19 P e117afbf8819424e8a98476453dfde2a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6810a866decb4f5089f7d35c870e4204)
I20240502 21:28:09.270112 30719 raft_consensus.cc:504] T 256ae04a48b849c4a9086b6b1ff5df19 P e117afbf8819424e8a98476453dfde2a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } } peers { permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: false } }
I20240502 21:28:09.271601 30719 leader_election.cc:288] T 256ae04a48b849c4a9086b6b1ff5df19 P e117afbf8819424e8a98476453dfde2a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6810a866decb4f5089f7d35c870e4204 (127.15.255.129:40435), 4f58800ba2ed45eabd81c2f2e691aacd (127.15.255.137:40161)
W20240502 21:28:09.272598 26081 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.255.129:40435: connect: Connection refused (error 111)
W20240502 21:28:09.275188 26081 leader_election.cc:334] T 256ae04a48b849c4a9086b6b1ff5df19 P e117afbf8819424e8a98476453dfde2a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6810a866decb4f5089f7d35c870e4204 (127.15.255.129:40435): Network error: Client connection negotiation failed: client connection to 127.15.255.129:40435: connect: Connection refused (error 111)
I20240502 21:28:09.279125 27127 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256ae04a48b849c4a9086b6b1ff5df19" candidate_uuid: "e117afbf8819424e8a98476453dfde2a" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" is_pre_election: true
I20240502 21:28:09.279316 27127 raft_consensus.cc:2454] T 256ae04a48b849c4a9086b6b1ff5df19 P 4f58800ba2ed45eabd81c2f2e691aacd [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e117afbf8819424e8a98476453dfde2a in term 1.
I20240502 21:28:09.279695 26079 leader_election.cc:302] T 256ae04a48b849c4a9086b6b1ff5df19 P e117afbf8819424e8a98476453dfde2a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4f58800ba2ed45eabd81c2f2e691aacd, e117afbf8819424e8a98476453dfde2a; no voters: 6810a866decb4f5089f7d35c870e4204
I20240502 21:28:09.280009 30719 raft_consensus.cc:2781] T 256ae04a48b849c4a9086b6b1ff5df19 P e117afbf8819424e8a98476453dfde2a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 21:28:09.280151 30719 raft_consensus.cc:482] T 256ae04a48b849c4a9086b6b1ff5df19 P e117afbf8819424e8a98476453dfde2a [term 1 FOLLOWER]: Starting leader election (detected failure of leader 6810a866decb4f5089f7d35c870e4204)
I20240502 21:28:09.280259 30719 raft_consensus.cc:3037] T 256ae04a48b849c4a9086b6b1ff5df19 P e117afbf8819424e8a98476453dfde2a [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:28:09.283162 30719 raft_consensus.cc:504] T 256ae04a48b849c4a9086b6b1ff5df19 P e117afbf8819424e8a98476453dfde2a [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } } peers { permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: false } }
I20240502 21:28:09.283818 30719 leader_election.cc:288] T 256ae04a48b849c4a9086b6b1ff5df19 P e117afbf8819424e8a98476453dfde2a [CANDIDATE]: Term 2 election: Requested vote from peers 6810a866decb4f5089f7d35c870e4204 (127.15.255.129:40435), 4f58800ba2ed45eabd81c2f2e691aacd (127.15.255.137:40161)
I20240502 21:28:09.284303 27127 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "256ae04a48b849c4a9086b6b1ff5df19" candidate_uuid: "e117afbf8819424e8a98476453dfde2a" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "4f58800ba2ed45eabd81c2f2e691aacd"
I20240502 21:28:09.284540 27127 raft_consensus.cc:3037] T 256ae04a48b849c4a9086b6b1ff5df19 P 4f58800ba2ed45eabd81c2f2e691aacd [term 1 FOLLOWER]: Advancing to term 2
W20240502 21:28:09.285399 26081 leader_election.cc:334] T 256ae04a48b849c4a9086b6b1ff5df19 P e117afbf8819424e8a98476453dfde2a [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 6810a866decb4f5089f7d35c870e4204 (127.15.255.129:40435): Network error: Client connection negotiation failed: client connection to 127.15.255.129:40435: connect: Connection refused (error 111)
I20240502 21:28:09.286679 27127 raft_consensus.cc:2454] T 256ae04a48b849c4a9086b6b1ff5df19 P 4f58800ba2ed45eabd81c2f2e691aacd [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e117afbf8819424e8a98476453dfde2a in term 2.
I20240502 21:28:09.287029 26079 leader_election.cc:302] T 256ae04a48b849c4a9086b6b1ff5df19 P e117afbf8819424e8a98476453dfde2a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4f58800ba2ed45eabd81c2f2e691aacd, e117afbf8819424e8a98476453dfde2a; no voters: 6810a866decb4f5089f7d35c870e4204
I20240502 21:28:09.287325 30719 raft_consensus.cc:2781] T 256ae04a48b849c4a9086b6b1ff5df19 P e117afbf8819424e8a98476453dfde2a [term 2 FOLLOWER]: Leader election won for term 2
I20240502 21:28:09.287789 30719 raft_consensus.cc:686] T 256ae04a48b849c4a9086b6b1ff5df19 P e117afbf8819424e8a98476453dfde2a [term 2 LEADER]: Becoming Leader. State: Replica: e117afbf8819424e8a98476453dfde2a, State: Running, Role: LEADER
W20240502 21:28:09.288336 26212 consensus_peers.cc:485] T f84bbe304597450c978ae28e0c4e7a3e P a3e500d7077a48acb5edb33f7e178c81 -> Peer 038c5d9d1d7a49c981c4c34fbc00d9e6 (127.15.255.132:46813): Couldn't send request to peer 038c5d9d1d7a49c981c4c34fbc00d9e6. Status: Network error: Client connection negotiation failed: client connection to 127.15.255.132:46813: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 21:28:09.288532 30719 consensus_queue.cc:237] T 256ae04a48b849c4a9086b6b1ff5df19 P e117afbf8819424e8a98476453dfde2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } } peers { permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: false } }
I20240502 21:28:09.292094 23770 catalog_manager.cc:5484] T 256ae04a48b849c4a9086b6b1ff5df19 P e117afbf8819424e8a98476453dfde2a reported cstate change: term changed from 1 to 2, leader changed from 6810a866decb4f5089f7d35c870e4204 (127.15.255.129) to e117afbf8819424e8a98476453dfde2a (127.15.255.135). New cstate: current_term: 2 leader_uuid: "e117afbf8819424e8a98476453dfde2a" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20240502 21:28:09.323912 27025 consensus_peers.cc:485] T 0b3c514213bb40ca98955d6409a3b372 P 4f58800ba2ed45eabd81c2f2e691aacd -> Peer 038c5d9d1d7a49c981c4c34fbc00d9e6 (127.15.255.132:46813): Couldn't send request to peer 038c5d9d1d7a49c981c4c34fbc00d9e6. Status: Network error: Client connection negotiation failed: client connection to 127.15.255.132:46813: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 21:28:09.346973 30719 raft_consensus.cc:482] T 4eebf4508766421eb6c547a0fbbc28b1 P e117afbf8819424e8a98476453dfde2a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6810a866decb4f5089f7d35c870e4204)
I20240502 21:28:09.347216 30719 raft_consensus.cc:504] T 4eebf4508766421eb6c547a0fbbc28b1 P e117afbf8819424e8a98476453dfde2a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } }
I20240502 21:28:09.348004 30719 leader_election.cc:288] T 4eebf4508766421eb6c547a0fbbc28b1 P e117afbf8819424e8a98476453dfde2a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6810a866decb4f5089f7d35c870e4204 (127.15.255.129:40435), a3e500d7077a48acb5edb33f7e178c81 (127.15.255.136:43595)
I20240502 21:28:09.348742 26279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4eebf4508766421eb6c547a0fbbc28b1" candidate_uuid: "e117afbf8819424e8a98476453dfde2a" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "a3e500d7077a48acb5edb33f7e178c81" is_pre_election: true
I20240502 21:28:09.349073 26279 raft_consensus.cc:2454] T 4eebf4508766421eb6c547a0fbbc28b1 P a3e500d7077a48acb5edb33f7e178c81 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e117afbf8819424e8a98476453dfde2a in term 1.
I20240502 21:28:09.349586 26080 leader_election.cc:302] T 4eebf4508766421eb6c547a0fbbc28b1 P e117afbf8819424e8a98476453dfde2a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a3e500d7077a48acb5edb33f7e178c81, e117afbf8819424e8a98476453dfde2a; no voters:
W20240502 21:28:09.349915 26081 leader_election.cc:334] T 4eebf4508766421eb6c547a0fbbc28b1 P e117afbf8819424e8a98476453dfde2a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6810a866decb4f5089f7d35c870e4204 (127.15.255.129:40435): Network error: Client connection negotiation failed: client connection to 127.15.255.129:40435: connect: Connection refused (error 111)
I20240502 21:28:09.349939 30719 raft_consensus.cc:2781] T 4eebf4508766421eb6c547a0fbbc28b1 P e117afbf8819424e8a98476453dfde2a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 21:28:09.350045 30719 raft_consensus.cc:482] T 4eebf4508766421eb6c547a0fbbc28b1 P e117afbf8819424e8a98476453dfde2a [term 1 FOLLOWER]: Starting leader election (detected failure of leader 6810a866decb4f5089f7d35c870e4204)
I20240502 21:28:09.350167 30719 raft_consensus.cc:3037] T 4eebf4508766421eb6c547a0fbbc28b1 P e117afbf8819424e8a98476453dfde2a [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:28:09.353034 30719 raft_consensus.cc:504] T 4eebf4508766421eb6c547a0fbbc28b1 P e117afbf8819424e8a98476453dfde2a [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } }
I20240502 21:28:09.353675 30719 leader_election.cc:288] T 4eebf4508766421eb6c547a0fbbc28b1 P e117afbf8819424e8a98476453dfde2a [CANDIDATE]: Term 2 election: Requested vote from peers 6810a866decb4f5089f7d35c870e4204 (127.15.255.129:40435), a3e500d7077a48acb5edb33f7e178c81 (127.15.255.136:43595)
I20240502 21:28:09.354120 26279 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4eebf4508766421eb6c547a0fbbc28b1" candidate_uuid: "e117afbf8819424e8a98476453dfde2a" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "a3e500d7077a48acb5edb33f7e178c81"
I20240502 21:28:09.354310 26279 raft_consensus.cc:3037] T 4eebf4508766421eb6c547a0fbbc28b1 P a3e500d7077a48acb5edb33f7e178c81 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:28:09.356155 26279 raft_consensus.cc:2454] T 4eebf4508766421eb6c547a0fbbc28b1 P a3e500d7077a48acb5edb33f7e178c81 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e117afbf8819424e8a98476453dfde2a in term 2.
I20240502 21:28:09.356606 26080 leader_election.cc:302] T 4eebf4508766421eb6c547a0fbbc28b1 P e117afbf8819424e8a98476453dfde2a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a3e500d7077a48acb5edb33f7e178c81, e117afbf8819424e8a98476453dfde2a; no voters:
I20240502 21:28:09.356916 30719 raft_consensus.cc:2781] T 4eebf4508766421eb6c547a0fbbc28b1 P e117afbf8819424e8a98476453dfde2a [term 2 FOLLOWER]: Leader election won for term 2
W20240502 21:28:09.357121 26081 leader_election.cc:334] T 4eebf4508766421eb6c547a0fbbc28b1 P e117afbf8819424e8a98476453dfde2a [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 6810a866decb4f5089f7d35c870e4204 (127.15.255.129:40435): Network error: Client connection negotiation failed: client connection to 127.15.255.129:40435: connect: Connection refused (error 111)
I20240502 21:28:09.357359 30719 raft_consensus.cc:686] T 4eebf4508766421eb6c547a0fbbc28b1 P e117afbf8819424e8a98476453dfde2a [term 2 LEADER]: Becoming Leader. State: Replica: e117afbf8819424e8a98476453dfde2a, State: Running, Role: LEADER
I20240502 21:28:09.357636 30719 consensus_queue.cc:237] T 4eebf4508766421eb6c547a0fbbc28b1 P e117afbf8819424e8a98476453dfde2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } }
I20240502 21:28:09.360925 23770 catalog_manager.cc:5484] T 4eebf4508766421eb6c547a0fbbc28b1 P e117afbf8819424e8a98476453dfde2a reported cstate change: term changed from 1 to 2, leader changed from 6810a866decb4f5089f7d35c870e4204 (127.15.255.129) to e117afbf8819424e8a98476453dfde2a (127.15.255.135). New cstate: current_term: 2 leader_uuid: "e117afbf8819424e8a98476453dfde2a" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 21:28:09.382725 16382 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvNciQZ/build/asan/bin/kudu with pid 25928
I20240502 21:28:09.404861 16382 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvNciQZ/build/asan/bin/kudu pid 26061
I20240502 21:28:09.410290 26188 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 21:28:09.433619 30492 raft_consensus.cc:482] T 83b95f3ae7564b81b31a2f5673ee22bb P 4f58800ba2ed45eabd81c2f2e691aacd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6810a866decb4f5089f7d35c870e4204)
I20240502 21:28:09.433794 30492 raft_consensus.cc:504] T 83b95f3ae7564b81b31a2f5673ee22bb P 4f58800ba2ed45eabd81c2f2e691aacd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } } peers { permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: false } }
I20240502 21:28:09.434494 30492 leader_election.cc:288] T 83b95f3ae7564b81b31a2f5673ee22bb P 4f58800ba2ed45eabd81c2f2e691aacd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6810a866decb4f5089f7d35c870e4204 (127.15.255.129:40435), e117afbf8819424e8a98476453dfde2a (127.15.255.135:32775)
W20240502 21:28:09.435690 27033 leader_election.cc:334] T 83b95f3ae7564b81b31a2f5673ee22bb P 4f58800ba2ed45eabd81c2f2e691aacd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6810a866decb4f5089f7d35c870e4204 (127.15.255.129:40435): Network error: Client connection negotiation failed: client connection to 127.15.255.129:40435: connect: Connection refused (error 111)
I20240502 21:28:09.454217 30391 raft_consensus.cc:482] T adf80b9d71314010a7a4cf052fd8a511 P a3e500d7077a48acb5edb33f7e178c81 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6810a866decb4f5089f7d35c870e4204)
I20240502 21:28:09.454423 30391 raft_consensus.cc:504] T adf80b9d71314010a7a4cf052fd8a511 P a3e500d7077a48acb5edb33f7e178c81 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } } peers { permanent_uuid: "fb6004b8640a4906aa0cfe63e383d40e" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 32825 } attrs { promote: false } }
I20240502 21:28:09.455276 30391 leader_election.cc:288] T adf80b9d71314010a7a4cf052fd8a511 P a3e500d7077a48acb5edb33f7e178c81 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6810a866decb4f5089f7d35c870e4204 (127.15.255.129:40435), fb6004b8640a4906aa0cfe63e383d40e (127.15.255.134:32825)
W20240502 21:28:09.456727 26211 leader_election.cc:334] T adf80b9d71314010a7a4cf052fd8a511 P a3e500d7077a48acb5edb33f7e178c81 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer fb6004b8640a4906aa0cfe63e383d40e (127.15.255.134:32825): Network error: Client connection negotiation failed: client connection to 127.15.255.134:32825: connect: Connection refused (error 111)
W20240502 21:28:09.456887 26214 leader_election.cc:334] T adf80b9d71314010a7a4cf052fd8a511 P a3e500d7077a48acb5edb33f7e178c81 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6810a866decb4f5089f7d35c870e4204 (127.15.255.129:40435): Network error: Client connection negotiation failed: client connection to 127.15.255.129:40435: connect: Connection refused (error 111)
I20240502 21:28:09.457037 26214 leader_election.cc:302] T adf80b9d71314010a7a4cf052fd8a511 P a3e500d7077a48acb5edb33f7e178c81 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a3e500d7077a48acb5edb33f7e178c81; no voters: 6810a866decb4f5089f7d35c870e4204, fb6004b8640a4906aa0cfe63e383d40e
I20240502 21:28:09.457295 30391 raft_consensus.cc:2726] T adf80b9d71314010a7a4cf052fd8a511 P a3e500d7077a48acb5edb33f7e178c81 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 21:28:09.521994 26211 consensus_peers.cc:485] T f84bbe304597450c978ae28e0c4e7a3e P a3e500d7077a48acb5edb33f7e178c81 -> Peer fb6004b8640a4906aa0cfe63e383d40e (127.15.255.134:32825): Couldn't send request to peer fb6004b8640a4906aa0cfe63e383d40e. Status: Network error: Client connection negotiation failed: client connection to 127.15.255.134:32825: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 21:28:09.524184 30492 raft_consensus.cc:482] T 77fc7226878f44d3884ccb1fc41c44a4 P 4f58800ba2ed45eabd81c2f2e691aacd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6c194a6db6c145acafc0f3068b8ce1fd)
I20240502 21:28:09.524366 30492 raft_consensus.cc:504] T 77fc7226878f44d3884ccb1fc41c44a4 P 4f58800ba2ed45eabd81c2f2e691aacd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } } peers { permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: false } }
I20240502 21:28:09.525130 30492 leader_election.cc:288] T 77fc7226878f44d3884ccb1fc41c44a4 P 4f58800ba2ed45eabd81c2f2e691aacd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6c194a6db6c145acafc0f3068b8ce1fd (127.15.255.130:41235), e117afbf8819424e8a98476453dfde2a (127.15.255.135:32775)
W20240502 21:28:09.526255 27033 leader_election.cc:334] T 77fc7226878f44d3884ccb1fc41c44a4 P 4f58800ba2ed45eabd81c2f2e691aacd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6c194a6db6c145acafc0f3068b8ce1fd (127.15.255.130:41235): Network error: Client connection negotiation failed: client connection to 127.15.255.130:41235: connect: Connection refused (error 111)
I20240502 21:28:09.559322 26145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77fc7226878f44d3884ccb1fc41c44a4" candidate_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "e117afbf8819424e8a98476453dfde2a" is_pre_election: true
I20240502 21:28:09.559440 26147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83b95f3ae7564b81b31a2f5673ee22bb" candidate_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" candidate_term: 2 candidate_status { last_received { term: 1 index: 9 } } ignore_live_leader: false dest_uuid: "e117afbf8819424e8a98476453dfde2a" is_pre_election: true
I20240502 21:28:09.560058 30724 raft_consensus.cc:482] T 77fc7226878f44d3884ccb1fc41c44a4 P e117afbf8819424e8a98476453dfde2a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6c194a6db6c145acafc0f3068b8ce1fd)
I20240502 21:28:09.560012 30719 raft_consensus.cc:482] T 83b95f3ae7564b81b31a2f5673ee22bb P e117afbf8819424e8a98476453dfde2a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6810a866decb4f5089f7d35c870e4204)
I20240502 21:28:09.560284 16382 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvNciQZ/build/asan/bin/kudu with pid 26061
I20240502 21:28:09.582736 16382 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvNciQZ/build/asan/bin/kudu pid 26196
W20240502 21:28:09.583339 27028 leader_election.cc:334] T 83b95f3ae7564b81b31a2f5673ee22bb P 4f58800ba2ed45eabd81c2f2e691aacd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer e117afbf8819424e8a98476453dfde2a (127.15.255.135:32775): Network error: Client connection negotiation failed: client connection to 127.15.255.135:32775: connect: Connection refused (error 111)
I20240502 21:28:09.583483 27028 leader_election.cc:302] T 83b95f3ae7564b81b31a2f5673ee22bb P 4f58800ba2ed45eabd81c2f2e691aacd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4f58800ba2ed45eabd81c2f2e691aacd; no voters: 6810a866decb4f5089f7d35c870e4204, e117afbf8819424e8a98476453dfde2a
W20240502 21:28:09.583806 27028 leader_election.cc:334] T 77fc7226878f44d3884ccb1fc41c44a4 P 4f58800ba2ed45eabd81c2f2e691aacd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer e117afbf8819424e8a98476453dfde2a (127.15.255.135:32775): Network error: Client connection negotiation failed: client connection to 127.15.255.135:32775: connect: Connection refused (error 111)
I20240502 21:28:09.583889 27028 leader_election.cc:302] T 77fc7226878f44d3884ccb1fc41c44a4 P 4f58800ba2ed45eabd81c2f2e691aacd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4f58800ba2ed45eabd81c2f2e691aacd; no voters: 6c194a6db6c145acafc0f3068b8ce1fd, e117afbf8819424e8a98476453dfde2a
I20240502 21:28:09.584241 30492 raft_consensus.cc:2726] T 83b95f3ae7564b81b31a2f5673ee22bb P 4f58800ba2ed45eabd81c2f2e691aacd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 21:28:09.584434 30492 raft_consensus.cc:2726] T 77fc7226878f44d3884ccb1fc41c44a4 P 4f58800ba2ed45eabd81c2f2e691aacd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 21:28:09.587574 27024 consensus_peers.cc:485] T 0b3c514213bb40ca98955d6409a3b372 P 4f58800ba2ed45eabd81c2f2e691aacd -> Peer fb6004b8640a4906aa0cfe63e383d40e (127.15.255.134:32825): Couldn't send request to peer fb6004b8640a4906aa0cfe63e383d40e. Status: Network error: Client connection negotiation failed: client connection to 127.15.255.134:32825: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 21:28:09.587904 30492 raft_consensus.cc:482] T c9b5be01b5e941b1848df79c7a9db348 P 4f58800ba2ed45eabd81c2f2e691aacd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6c194a6db6c145acafc0f3068b8ce1fd)
I20240502 21:28:09.588081 30492 raft_consensus.cc:504] T c9b5be01b5e941b1848df79c7a9db348 P 4f58800ba2ed45eabd81c2f2e691aacd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } } peers { permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: false } }
I20240502 21:28:09.588793 30492 leader_election.cc:288] T c9b5be01b5e941b1848df79c7a9db348 P 4f58800ba2ed45eabd81c2f2e691aacd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6c194a6db6c145acafc0f3068b8ce1fd (127.15.255.130:41235), e117afbf8819424e8a98476453dfde2a (127.15.255.135:32775)
W20240502 21:28:09.591147 27033 leader_election.cc:334] T c9b5be01b5e941b1848df79c7a9db348 P 4f58800ba2ed45eabd81c2f2e691aacd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6c194a6db6c145acafc0f3068b8ce1fd (127.15.255.130:41235): Network error: Client connection negotiation failed: client connection to 127.15.255.130:41235: connect: Connection refused (error 111)
W20240502 21:28:09.595950 27028 leader_election.cc:334] T c9b5be01b5e941b1848df79c7a9db348 P 4f58800ba2ed45eabd81c2f2e691aacd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer e117afbf8819424e8a98476453dfde2a (127.15.255.135:32775): Network error: Client connection negotiation failed: client connection to 127.15.255.135:32775: connect: Connection refused (error 111)
I20240502 21:28:09.596119 27028 leader_election.cc:302] T c9b5be01b5e941b1848df79c7a9db348 P 4f58800ba2ed45eabd81c2f2e691aacd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4f58800ba2ed45eabd81c2f2e691aacd; no voters: 6c194a6db6c145acafc0f3068b8ce1fd, e117afbf8819424e8a98476453dfde2a
I20240502 21:28:09.596690 30734 raft_consensus.cc:2726] T c9b5be01b5e941b1848df79c7a9db348 P 4f58800ba2ed45eabd81c2f2e691aacd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 21:28:09.599853 26321 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 21:28:09.722990 30734 raft_consensus.cc:482] T a274546743354e79bffa26ae7000e895 P 4f58800ba2ed45eabd81c2f2e691aacd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6c194a6db6c145acafc0f3068b8ce1fd)
I20240502 21:28:09.723300 30734 raft_consensus.cc:504] T a274546743354e79bffa26ae7000e895 P 4f58800ba2ed45eabd81c2f2e691aacd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6c194a6db6c145acafc0f3068b8ce1fd" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 41235 } } peers { permanent_uuid: "e117afbf8819424e8a98476453dfde2a" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 32775 } attrs { promote: false } } peers { permanent_uuid: "4f58800ba2ed45eabd81c2f2e691aacd" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 40161 } attrs { promote: false } }
W20240502 21:28:09.725459 27033 leader_election.cc:334] T a274546743354e79bffa26ae7000e895 P 4f58800ba2ed45eabd81c2f2e691aacd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6c194a6db6c145acafc0f3068b8ce1fd (127.15.255.130:41235): Network error: Client connection negotiation failed: client connection to 127.15.255.130:41235: connect: Connection refused (error 111)
I20240502 21:28:09.725888 30734 leader_election.cc:288] T a274546743354e79bffa26ae7000e895 P 4f58800ba2ed45eabd81c2f2e691aacd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6c194a6db6c145acafc0f3068b8ce1fd (127.15.255.130:41235), e117afbf8819424e8a98476453dfde2a (127.15.255.135:32775)
W20240502 21:28:09.727262 27028 leader_election.cc:334] T a274546743354e79bffa26ae7000e895 P 4f58800ba2ed45eabd81c2f2e691aacd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer e117afbf8819424e8a98476453dfde2a (127.15.255.135:32775): Network error: Client connection negotiation failed: client connection to 127.15.255.135:32775: connect: Connection refused (error 111)
I20240502 21:28:09.727392 27028 leader_election.cc:302] T a274546743354e79bffa26ae7000e895 P 4f58800ba2ed45eabd81c2f2e691aacd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4f58800ba2ed45eabd81c2f2e691aacd; no voters: 6c194a6db6c145acafc0f3068b8ce1fd, e117afbf8819424e8a98476453dfde2a
I20240502 21:28:09.727648 30734 raft_consensus.cc:2726] T a274546743354e79bffa26ae7000e895 P 4f58800ba2ed45eabd81c2f2e691aacd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 21:28:09.757769 30736 raft_consensus.cc:482] T 549666d19e724a96801db183a07809d8 P a3e500d7077a48acb5edb33f7e178c81 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6810a866decb4f5089f7d35c870e4204)
I20240502 21:28:09.757977 30736 raft_consensus.cc:504] T 549666d19e724a96801db183a07809d8 P a3e500d7077a48acb5edb33f7e178c81 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "6810a866decb4f5089f7d35c870e4204" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40435 } } peers { permanent_uuid: "a3e500d7077a48acb5edb33f7e178c81" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 43595 } attrs { promote: false } } peers { permanent_uuid: "ee27c3f9e0ec46caa742ac386d6b7c20" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 40841 } attrs { promote: false } }
W20240502 21:28:09.758612 26211 consensus_peers.cc:485] T 30236241b670443489240f191d9b9d50 P a3e500d7077a48acb5edb33f7e178c81 -> Peer fb6004b8640a4906aa0cfe63e383d40e (127.15.255.134:32825): Couldn't send request to peer fb6004b8640a4906aa0cfe63e383d40e. Status: Network error: Client connection negotiation failed: client connection to 127.15.255.134:32825: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 21:28:09.758823 30736 leader_election.cc:288] T 549666d19e724a96801db183a07809d8 P a3e500d7077a48acb5edb33f7e178c81 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6810a866decb4f5089f7d35c870e4204 (127.15.255.129:40435), ee27c3f9e0ec46caa742ac386d6b7c20 (127.15.255.133:40841)
I20240502 21:28:09.759507 16382 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvNciQZ/build/asan/bin/kudu with pid 26196
W20240502 21:28:09.782604 27028 connection.cc:535] server connection from 127.15.255.136:44479 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 21:28:09.782985 16382 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvNciQZ/build/asan/bin/kudu pid 26928
I20240502 21:28:09.788343 27174 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 21:28:09.921953 16382 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvNciQZ/build/asan/bin/kudu with pid 26928
I20240502 21:28:09.942819 16382 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvNciQZ/build/asan/bin/kudu pid 23738
I20240502 21:28:09.943814 23801 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 21:28:10.162133 16382 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvNciQZ/build/asan/bin/kudu with pid 23738
2024-05-02T21:28:10Z chronyd exiting
[ OK ] TserverAddedDuringRebalancingTest.TserverStarts/1 (52235 ms)
[----------] 1 test from TserverAddedDuringRebalancingTest (52235 ms total)
[----------] 1 test from RebalancerAndSingleReplicaTablets
[ RUN ] RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove/5
I20240502 21:28:10.342597 16382 test_util.cc:268] Using random seed: 1201793205
I20240502 21:28:10.344159 16382 ts_itest-base.cc:112] Starting cluster with:
I20240502 21:28:10.344238 16382 ts_itest-base.cc:113] --------------
I20240502 21:28:10.344285 16382 ts_itest-base.cc:114] 4 tablet servers
I20240502 21:28:10.344352 16382 ts_itest-base.cc:115] 1 replicas per TS
I20240502 21:28:10.344400 16382 ts_itest-base.cc:116] --------------
2024-05-02T21:28:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-02T21:28:10Z Disabled control of system clock
I20240502 21:28:10.396212 16382 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvNciQZ/build/asan/bin/kudu
/tmp/dist-test-taskvNciQZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714685120732928-16382-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714685120732928-16382-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714685120732928-16382-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714685120732928-16382-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.255.190:35921
--webserver_interface=127.15.255.190
--webserver_port=0
--builtin_ntp_servers=127.15.255.148:38143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.255.190:35921
--raft_prepare_replacement_before_eviction=true
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false with env {}
W20240502 21:28:10.717492 30750 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:28:10.717792 30750 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:28:10.717942 30750 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:28:10.725181 30750 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 21:28:10.725301 30750 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240502 21:28:10.725380 30750 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:28:10.725457 30750 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240502 21:28:10.725517 30750 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240502 21:28:10.734367 30750 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.255.148:38143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714685120732928-16382-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714685120732928-16382-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.15.255.190:35921
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.255.190:35921
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714685120732928-16382-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.15.255.190
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714685120732928-16382-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:28:10.735615 30750 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:28:10.737316 30750 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:28:10.747387 30755 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:28:10.747861 30758 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:28:10.748453 30756 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:28:10.748966 30750 server_base.cc:1034] running on GCE node
I20240502 21:28:10.749614 30750 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:28:10.750881 30750 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:28:10.752082 30750 hybrid_clock.cc:648] HybridClock initialized: now 1714685290752031 us; error 39 us; skew 500 ppm
I20240502 21:28:10.752393 30750 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:28:10.758765 30750 webserver.cc:460] Webserver started at http://127.15.255.190:44025/ using document root <none> and password file <none>
I20240502 21:28:10.759438 30750 fs_manager.cc:359] Metadata directory not provided
I20240502 21:28:10.759549 30750 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:28:10.759829 30750 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:28:10.762359 30750 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714685120732928-16382-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "d381c63cae664ab9942526dbc30cfefa"
format_stamp: "Formatted at 2024-05-02 21:28:10 on dist-test-slave-jdfx"
I20240502 21:28:10.763080 30750 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714685120732928-16382-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "d381c63cae664ab9942526dbc30cfefa"
format_stamp: "Formatted at 2024-05-02 21:28:10 on dist-test-slave-jdfx"
I20240502 21:28:10.767761 30750 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20240502 21:28:10.770875 30764 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:28:10.772080 30750 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 21:28:10.772238 30750 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714685120732928-16382-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714685120732928-16382-0/raft_consensus-itest-cluster/master-0/wal
uuid: "d381c63cae664ab9942526dbc30cfefa"
format_stamp: "Formatted at 2024-05-02 21:28:10 on dist-test-slave-jdfx"
I20240502 21:28:10.772444 30750 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714685120732928-16382-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714685120732928-16382-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714685120732928-16382-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:28:10.787367 30750 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:28:10.788080 30750 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:28:10.788334 30750 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:28:10.805019 30750 rpc_server.cc:307] RPC server started. Bound to: 127.15.255.190:35921
I20240502 21:28:10.805048 30815 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.255.190:35921 every 8 connection(s)
I20240502 21:28:10.806267 30750 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714685120732928-16382-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240502 21:28:10.809868 30816 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:10.812351 16382 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvNciQZ/build/asan/bin/kudu as pid 30750
I20240502 21:28:10.812628 16382 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714685120732928-16382-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240502 21:28:10.820689 30816 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d381c63cae664ab9942526dbc30cfefa: Bootstrap starting.
I20240502 21:28:10.823575 30816 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d381c63cae664ab9942526dbc30cfefa: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:10.824858 30816 log.cc:824] T 00000000000000000000000000000000 P d381c63cae664ab9942526dbc30cfefa: Log is configured to *not* fsync() on all Append() calls
I20240502 21:28:10.827315 30816 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d381c63cae664ab9942526dbc30cfefa: No bootstrap required, opened a new log
I20240502 21:28:10.833140 30816 raft_consensus.cc:348] T 00000000000000000000000000000000 P d381c63cae664ab9942526dbc30cfefa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d381c63cae664ab9942526dbc30cfefa" member_type: VOTER last_known_addr { host: "127.15.255.190" port: 35921 } }
I20240502 21:28:10.833441 30816 raft_consensus.cc:374] T 00000000000000000000000000000000 P d381c63cae664ab9942526dbc30cfefa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:10.833519 30816 raft_consensus.cc:729] T 00000000000000000000000000000000 P d381c63cae664ab9942526dbc30cfefa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d381c63cae664ab9942526dbc30cfefa, State: Initialized, Role: FOLLOWER
I20240502 21:28:10.833998 30816 consensus_queue.cc:260] T 00000000000000000000000000000000 P d381c63cae664ab9942526dbc30cfefa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d381c63cae664ab9942526dbc30cfefa" member_type: VOTER last_known_addr { host: "127.15.255.190" port: 35921 } }
I20240502 21:28:10.834173 30816 raft_consensus.cc:388] T 00000000000000000000000000000000 P d381c63cae664ab9942526dbc30cfefa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 21:28:10.834270 30816 raft_consensus.cc:482] T 00000000000000000000000000000000 P d381c63cae664ab9942526dbc30cfefa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 21:28:10.834385 30816 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d381c63cae664ab9942526dbc30cfefa [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:10.836313 30816 raft_consensus.cc:504] T 00000000000000000000000000000000 P d381c63cae664ab9942526dbc30cfefa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d381c63cae664ab9942526dbc30cfefa" member_type: VOTER last_known_addr { host: "127.15.255.190" port: 35921 } }
I20240502 21:28:10.836802 30816 leader_election.cc:302] T 00000000000000000000000000000000 P d381c63cae664ab9942526dbc30cfefa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d381c63cae664ab9942526dbc30cfefa; no voters:
I20240502 21:28:10.837289 30816 leader_election.cc:288] T 00000000000000000000000000000000 P d381c63cae664ab9942526dbc30cfefa [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 21:28:10.837414 30821 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d381c63cae664ab9942526dbc30cfefa [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:28:10.837997 30821 raft_consensus.cc:686] T 00000000000000000000000000000000 P d381c63cae664ab9942526dbc30cfefa [term 1 LEADER]: Becoming Leader. State: Replica: d381c63cae664ab9942526dbc30cfefa, State: Running, Role: LEADER
I20240502 21:28:10.838469 30821 consensus_queue.cc:237] T 00000000000000000000000000000000 P d381c63cae664ab9942526dbc30cfefa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d381c63cae664ab9942526dbc30cfefa" member_type: VOTER last_known_addr { host: "127.15.255.190" port: 35921 } }
I20240502 21:28:10.839017 30816 sys_catalog.cc:564] T 00000000000000000000000000000000 P d381c63cae664ab9942526dbc30cfefa [sys.catalog]: configured and running, proceeding with master startup.
I20240502 21:28:10.840890 30822 sys_catalog.cc:455] T 00000000000000000000000000000000 P d381c63cae664ab9942526dbc30cfefa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d381c63cae664ab9942526dbc30cfefa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d381c63cae664ab9942526dbc30cfefa" member_type: VOTER last_known_addr { host: "127.15.255.190" port: 35921 } } }
I20240502 21:28:10.841146 30822 sys_catalog.cc:458] T 00000000000000000000000000000000 P d381c63cae664ab9942526dbc30cfefa [sys.catalog]: This master's current role is: LEADER
I20240502 21:28:10.841742 30823 sys_catalog.cc:455] T 00000000000000000000000000000000 P d381c63cae664ab9942526dbc30cfefa [sys.catalog]: SysCatalogTable state changed. Reason: New leader d381c63cae664ab9942526dbc30cfefa. Latest consensus state: current_term: 1 leader_uuid: "d381c63cae664ab9942526dbc30cfefa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d381c63cae664ab9942526dbc30cfefa" member_type: VOTER last_known_addr { host: "127.15.255.190" port: 35921 } } }
I20240502 21:28:10.842046 30823 sys_catalog.cc:458] T 00000000000000000000000000000000 P d381c63cae664ab9942526dbc30cfefa [sys.catalog]: This master's current role is: LEADER
I20240502 21:28:10.843189 30827 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 21:28:10.848227 30827 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 21:28:10.857466 30827 catalog_manager.cc:1347] Generated new cluster ID: d1ffd9009b7c479da27848f01ecf4b6c
I20240502 21:28:10.857573 30827 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 21:28:10.880474 30827 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 21:28:10.881551 30827 catalog_manager.cc:1504] Loading token signing keys...
I20240502 21:28:10.893244 30827 catalog_manager.cc:5857] T 00000000000000000000000000000000 P d381c63cae664ab9942526dbc30cfefa: Generated new TSK 0
I20240502 21:28:10.893957 30827 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 21:28:10.903232 16382 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvNciQZ/build/asan/bin/kudu
/tmp/dist-test-taskvNciQZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.255.129:0
--local_ip_for_outbound_sockets=127.15.255.129
--webserver_interface=127.15.255.129
--webserver_port=0
--tserver_master_addrs=127.15.255.190:35921
--builtin_ntp_servers=127.15.255.148:38143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240502 21:28:11.244113 30841 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:28:11.244437 30841 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:28:11.244550 30841 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:28:11.251896 30841 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 21:28:11.252010 30841 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:28:11.252127 30841 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.255.129
I20240502 21:28:11.260984 30841 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.255.148:38143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.255.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.15.255.129
--webserver_port=0
--tserver_master_addrs=127.15.255.190:35921
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.255.129
--log_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:28:11.262214 30841 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:28:11.263854 30841 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:28:11.275086 30847 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:28:11.275537 30849 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:28:11.276144 30846 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:28:11.276686 30841 server_base.cc:1034] running on GCE node
I20240502 21:28:11.277209 30841 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:28:11.278069 30841 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:28:11.279273 30841 hybrid_clock.cc:648] HybridClock initialized: now 1714685291279228 us; error 32 us; skew 500 ppm
I20240502 21:28:11.279573 30841 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:28:11.281649 30841 webserver.cc:460] Webserver started at http://127.15.255.129:44511/ using document root <none> and password file <none>
I20240502 21:28:11.282328 30841 fs_manager.cc:359] Metadata directory not provided
I20240502 21:28:11.282444 30841 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:28:11.282722 30841 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:28:11.285225 30841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "2bff96c137244df6b6beef103aecd997"
format_stamp: "Formatted at 2024-05-02 21:28:11 on dist-test-slave-jdfx"
I20240502 21:28:11.285924 30841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "2bff96c137244df6b6beef103aecd997"
format_stamp: "Formatted at 2024-05-02 21:28:11 on dist-test-slave-jdfx"
I20240502 21:28:11.290679 30841 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20240502 21:28:11.293830 30855 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:28:11.294957 30841 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240502 21:28:11.295105 30841 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "2bff96c137244df6b6beef103aecd997"
format_stamp: "Formatted at 2024-05-02 21:28:11 on dist-test-slave-jdfx"
I20240502 21:28:11.295289 30841 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:28:11.310348 30841 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:28:11.311070 30841 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:28:11.311324 30841 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:28:11.312147 30841 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:28:11.313506 30841 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:28:11.313598 30841 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:28:11.313699 30841 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:28:11.313768 30841 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 21:28:11.341992 30841 rpc_server.cc:307] RPC server started. Bound to: 127.15.255.129:44455
I20240502 21:28:11.342062 30967 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.255.129:44455 every 8 connection(s)
I20240502 21:28:11.343231 30841 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240502 21:28:11.350178 16382 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvNciQZ/build/asan/bin/kudu as pid 30841
I20240502 21:28:11.350375 16382 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240502 21:28:11.353778 16382 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvNciQZ/build/asan/bin/kudu
/tmp/dist-test-taskvNciQZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.255.130:0
--local_ip_for_outbound_sockets=127.15.255.130
--webserver_interface=127.15.255.130
--webserver_port=0
--tserver_master_addrs=127.15.255.190:35921
--builtin_ntp_servers=127.15.255.148:38143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 21:28:11.355489 30968 heartbeater.cc:347] Connected to a master server at 127.15.255.190:35921
I20240502 21:28:11.355755 30968 heartbeater.cc:464] Registering TS with master...
I20240502 21:28:11.356525 30968 heartbeater.cc:511] Master 127.15.255.190:35921 requested a full tablet report, sending...
I20240502 21:28:11.358608 30781 ts_manager.cc:194] Registered new tserver with Master: 2bff96c137244df6b6beef103aecd997 (127.15.255.129:44455)
I20240502 21:28:11.360528 30781 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.255.129:34431
W20240502 21:28:11.674639 30972 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:28:11.674891 30972 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:28:11.675014 30972 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:28:11.682112 30972 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 21:28:11.682233 30972 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:28:11.682374 30972 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.255.130
I20240502 21:28:11.690702 30972 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.255.148:38143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.255.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.15.255.130
--webserver_port=0
--tserver_master_addrs=127.15.255.190:35921
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.255.130
--log_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:28:11.691776 30972 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:28:11.693356 30972 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:28:11.704957 30978 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:28:11.705545 30979 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:28:11.705957 30981 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:28:11.706507 30972 server_base.cc:1034] running on GCE node
I20240502 21:28:11.707000 30972 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:28:11.707791 30972 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:28:11.708972 30972 hybrid_clock.cc:648] HybridClock initialized: now 1714685291708926 us; error 37 us; skew 500 ppm
I20240502 21:28:11.709254 30972 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:28:11.711194 30972 webserver.cc:460] Webserver started at http://127.15.255.130:42519/ using document root <none> and password file <none>
I20240502 21:28:11.711838 30972 fs_manager.cc:359] Metadata directory not provided
I20240502 21:28:11.711951 30972 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:28:11.712239 30972 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:28:11.714578 30972 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "a27039d23a9041e4831027412c5416c9"
format_stamp: "Formatted at 2024-05-02 21:28:11 on dist-test-slave-jdfx"
I20240502 21:28:11.715276 30972 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "a27039d23a9041e4831027412c5416c9"
format_stamp: "Formatted at 2024-05-02 21:28:11 on dist-test-slave-jdfx"
I20240502 21:28:11.719780 30972 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20240502 21:28:11.722796 30987 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:28:11.723932 30972 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 21:28:11.724068 30972 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "a27039d23a9041e4831027412c5416c9"
format_stamp: "Formatted at 2024-05-02 21:28:11 on dist-test-slave-jdfx"
I20240502 21:28:11.724246 30972 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:28:11.750504 30972 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:28:11.751235 30972 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:28:11.751475 30972 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:28:11.752314 30972 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:28:11.753705 30972 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:28:11.753798 30972 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:28:11.753901 30972 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:28:11.753969 30972 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 21:28:11.782848 30972 rpc_server.cc:307] RPC server started. Bound to: 127.15.255.130:42697
I20240502 21:28:11.782913 31100 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.255.130:42697 every 8 connection(s)
I20240502 21:28:11.784106 30972 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240502 21:28:11.794113 16382 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvNciQZ/build/asan/bin/kudu as pid 30972
I20240502 21:28:11.794315 16382 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240502 21:28:11.795959 31101 heartbeater.cc:347] Connected to a master server at 127.15.255.190:35921
I20240502 21:28:11.796272 31101 heartbeater.cc:464] Registering TS with master...
I20240502 21:28:11.797260 31101 heartbeater.cc:511] Master 127.15.255.190:35921 requested a full tablet report, sending...
I20240502 21:28:11.797605 16382 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvNciQZ/build/asan/bin/kudu
/tmp/dist-test-taskvNciQZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.255.131:0
--local_ip_for_outbound_sockets=127.15.255.131
--webserver_interface=127.15.255.131
--webserver_port=0
--tserver_master_addrs=127.15.255.190:35921
--builtin_ntp_servers=127.15.255.148:38143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 21:28:11.798691 30781 ts_manager.cc:194] Registered new tserver with Master: a27039d23a9041e4831027412c5416c9 (127.15.255.130:42697)
I20240502 21:28:11.799445 30781 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.255.130:60311
W20240502 21:28:12.127528 31105 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:28:12.127789 31105 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:28:12.127918 31105 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:28:12.135879 31105 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 21:28:12.135993 31105 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:28:12.136147 31105 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.255.131
I20240502 21:28:12.144800 31105 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.255.148:38143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.255.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.15.255.131
--webserver_port=0
--tserver_master_addrs=127.15.255.190:35921
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.255.131
--log_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:28:12.146054 31105 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:28:12.147634 31105 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:28:12.158782 31110 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:28:12.159169 31111 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:28:12.159529 31113 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:28:12.160243 31105 server_base.cc:1034] running on GCE node
I20240502 21:28:12.160807 31105 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:28:12.161661 31105 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:28:12.162853 31105 hybrid_clock.cc:648] HybridClock initialized: now 1714685292162810 us; error 39 us; skew 500 ppm
I20240502 21:28:12.163152 31105 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:28:12.165282 31105 webserver.cc:460] Webserver started at http://127.15.255.131:32787/ using document root <none> and password file <none>
I20240502 21:28:12.165947 31105 fs_manager.cc:359] Metadata directory not provided
I20240502 21:28:12.166061 31105 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:28:12.166347 31105 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:28:12.168879 31105 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "bbfd9ca2d42547dca3f678bd6ed2964e"
format_stamp: "Formatted at 2024-05-02 21:28:12 on dist-test-slave-jdfx"
I20240502 21:28:12.169579 31105 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "bbfd9ca2d42547dca3f678bd6ed2964e"
format_stamp: "Formatted at 2024-05-02 21:28:12 on dist-test-slave-jdfx"
I20240502 21:28:12.174299 31105 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20240502 21:28:12.177397 31119 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:28:12.178587 31105 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20240502 21:28:12.178731 31105 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "bbfd9ca2d42547dca3f678bd6ed2964e"
format_stamp: "Formatted at 2024-05-02 21:28:12 on dist-test-slave-jdfx"
I20240502 21:28:12.178908 31105 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:28:12.204742 31105 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:28:12.205600 31105 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:28:12.205881 31105 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:28:12.206800 31105 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:28:12.208256 31105 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:28:12.208351 31105 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:28:12.208479 31105 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:28:12.208551 31105 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 21:28:12.237448 31105 rpc_server.cc:307] RPC server started. Bound to: 127.15.255.131:37625
I20240502 21:28:12.237512 31232 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.255.131:37625 every 8 connection(s)
I20240502 21:28:12.238708 31105 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240502 21:28:12.242376 16382 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvNciQZ/build/asan/bin/kudu as pid 31105
I20240502 21:28:12.242583 16382 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240502 21:28:12.245611 16382 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvNciQZ/build/asan/bin/kudu
/tmp/dist-test-taskvNciQZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.255.132:0
--local_ip_for_outbound_sockets=127.15.255.132
--webserver_interface=127.15.255.132
--webserver_port=0
--tserver_master_addrs=127.15.255.190:35921
--builtin_ntp_servers=127.15.255.148:38143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 21:28:12.250792 31233 heartbeater.cc:347] Connected to a master server at 127.15.255.190:35921
I20240502 21:28:12.251046 31233 heartbeater.cc:464] Registering TS with master...
I20240502 21:28:12.251770 31233 heartbeater.cc:511] Master 127.15.255.190:35921 requested a full tablet report, sending...
I20240502 21:28:12.253186 30781 ts_manager.cc:194] Registered new tserver with Master: bbfd9ca2d42547dca3f678bd6ed2964e (127.15.255.131:37625)
I20240502 21:28:12.254101 30781 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.255.131:34359
I20240502 21:28:12.363173 30968 heartbeater.cc:503] Master 127.15.255.190:35921 was elected leader, sending a full tablet report...
W20240502 21:28:12.562409 31236 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:28:12.562665 31236 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:28:12.562791 31236 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:28:12.569944 31236 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 21:28:12.570066 31236 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:28:12.570204 31236 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.255.132
I20240502 21:28:12.578817 31236 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.255.148:38143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.255.132:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.15.255.132
--webserver_port=0
--tserver_master_addrs=127.15.255.190:35921
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.255.132
--log_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:28:12.579998 31236 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:28:12.581636 31236 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:28:12.592816 31241 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:28:12.593293 31244 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:28:12.593704 31242 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:28:12.594127 31236 server_base.cc:1034] running on GCE node
I20240502 21:28:12.594642 31236 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:28:12.595480 31236 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:28:12.596665 31236 hybrid_clock.cc:648] HybridClock initialized: now 1714685292596629 us; error 26 us; skew 500 ppm
I20240502 21:28:12.596943 31236 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:28:12.598910 31236 webserver.cc:460] Webserver started at http://127.15.255.132:39927/ using document root <none> and password file <none>
I20240502 21:28:12.599610 31236 fs_manager.cc:359] Metadata directory not provided
I20240502 21:28:12.599725 31236 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:28:12.600003 31236 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:28:12.602404 31236 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "17faa5ec5e6a42c9846441cf2da4eb4d"
format_stamp: "Formatted at 2024-05-02 21:28:12 on dist-test-slave-jdfx"
I20240502 21:28:12.603093 31236 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "17faa5ec5e6a42c9846441cf2da4eb4d"
format_stamp: "Formatted at 2024-05-02 21:28:12 on dist-test-slave-jdfx"
I20240502 21:28:12.607650 31236 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20240502 21:28:12.610649 31250 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:28:12.611776 31236 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 21:28:12.611912 31236 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "17faa5ec5e6a42c9846441cf2da4eb4d"
format_stamp: "Formatted at 2024-05-02 21:28:12 on dist-test-slave-jdfx"
I20240502 21:28:12.612083 31236 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:28:12.625651 31236 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:28:12.626353 31236 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:28:12.626588 31236 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:28:12.627420 31236 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:28:12.628800 31236 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:28:12.628891 31236 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:28:12.628991 31236 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:28:12.629058 31236 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 21:28:12.657509 31236 rpc_server.cc:307] RPC server started. Bound to: 127.15.255.132:45045
I20240502 21:28:12.657570 31366 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.255.132:45045 every 8 connection(s)
I20240502 21:28:12.658700 31236 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240502 21:28:12.665513 16382 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvNciQZ/build/asan/bin/kudu as pid 31236
I20240502 21:28:12.665695 16382 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240502 21:28:12.670258 31367 heartbeater.cc:347] Connected to a master server at 127.15.255.190:35921
I20240502 21:28:12.670500 31367 heartbeater.cc:464] Registering TS with master...
I20240502 21:28:12.671209 31367 heartbeater.cc:511] Master 127.15.255.190:35921 requested a full tablet report, sending...
I20240502 21:28:12.672391 30781 ts_manager.cc:194] Registered new tserver with Master: 17faa5ec5e6a42c9846441cf2da4eb4d (127.15.255.132:45045)
I20240502 21:28:12.673321 30781 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.255.132:44443
I20240502 21:28:12.680963 16382 external_mini_cluster.cc:930] 4 TS(s) registered with all masters
I20240502 21:28:12.694926 30781 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46594:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20240502 21:28:12.719343 31300 tablet_service.cc:1467] Processing CreateTablet for tablet 6c5cb1507e5a470abaa5de5ec29b99d7 (DEFAULT_TABLE table=TestTable [id=79642cb7c4394951abe2581d2de07ecc]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:12.720525 31300 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c5cb1507e5a470abaa5de5ec29b99d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:12.729637 31381 tablet_bootstrap.cc:492] T 6c5cb1507e5a470abaa5de5ec29b99d7 P 17faa5ec5e6a42c9846441cf2da4eb4d: Bootstrap starting.
I20240502 21:28:12.732374 31381 tablet_bootstrap.cc:654] T 6c5cb1507e5a470abaa5de5ec29b99d7 P 17faa5ec5e6a42c9846441cf2da4eb4d: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:12.733331 31381 log.cc:824] T 6c5cb1507e5a470abaa5de5ec29b99d7 P 17faa5ec5e6a42c9846441cf2da4eb4d: Log is configured to *not* fsync() on all Append() calls
I20240502 21:28:12.735347 31381 tablet_bootstrap.cc:492] T 6c5cb1507e5a470abaa5de5ec29b99d7 P 17faa5ec5e6a42c9846441cf2da4eb4d: No bootstrap required, opened a new log
I20240502 21:28:12.735569 31381 ts_tablet_manager.cc:1367] T 6c5cb1507e5a470abaa5de5ec29b99d7 P 17faa5ec5e6a42c9846441cf2da4eb4d: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20240502 21:28:12.741451 31381 raft_consensus.cc:348] T 6c5cb1507e5a470abaa5de5ec29b99d7 P 17faa5ec5e6a42c9846441cf2da4eb4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17faa5ec5e6a42c9846441cf2da4eb4d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 45045 } }
I20240502 21:28:12.741724 31381 raft_consensus.cc:374] T 6c5cb1507e5a470abaa5de5ec29b99d7 P 17faa5ec5e6a42c9846441cf2da4eb4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:12.741827 31381 raft_consensus.cc:729] T 6c5cb1507e5a470abaa5de5ec29b99d7 P 17faa5ec5e6a42c9846441cf2da4eb4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17faa5ec5e6a42c9846441cf2da4eb4d, State: Initialized, Role: FOLLOWER
I20240502 21:28:12.742341 31381 consensus_queue.cc:260] T 6c5cb1507e5a470abaa5de5ec29b99d7 P 17faa5ec5e6a42c9846441cf2da4eb4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17faa5ec5e6a42c9846441cf2da4eb4d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 45045 } }
I20240502 21:28:12.742558 31381 raft_consensus.cc:388] T 6c5cb1507e5a470abaa5de5ec29b99d7 P 17faa5ec5e6a42c9846441cf2da4eb4d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 21:28:12.742681 31381 raft_consensus.cc:482] T 6c5cb1507e5a470abaa5de5ec29b99d7 P 17faa5ec5e6a42c9846441cf2da4eb4d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 21:28:12.742810 31381 raft_consensus.cc:3037] T 6c5cb1507e5a470abaa5de5ec29b99d7 P 17faa5ec5e6a42c9846441cf2da4eb4d [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:12.744751 31381 raft_consensus.cc:504] T 6c5cb1507e5a470abaa5de5ec29b99d7 P 17faa5ec5e6a42c9846441cf2da4eb4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17faa5ec5e6a42c9846441cf2da4eb4d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 45045 } }
I20240502 21:28:12.745172 31381 leader_election.cc:302] T 6c5cb1507e5a470abaa5de5ec29b99d7 P 17faa5ec5e6a42c9846441cf2da4eb4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 17faa5ec5e6a42c9846441cf2da4eb4d; no voters:
I20240502 21:28:12.745677 31381 leader_election.cc:288] T 6c5cb1507e5a470abaa5de5ec29b99d7 P 17faa5ec5e6a42c9846441cf2da4eb4d [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 21:28:12.745780 31383 raft_consensus.cc:2781] T 6c5cb1507e5a470abaa5de5ec29b99d7 P 17faa5ec5e6a42c9846441cf2da4eb4d [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:28:12.746335 31383 raft_consensus.cc:686] T 6c5cb1507e5a470abaa5de5ec29b99d7 P 17faa5ec5e6a42c9846441cf2da4eb4d [term 1 LEADER]: Becoming Leader. State: Replica: 17faa5ec5e6a42c9846441cf2da4eb4d, State: Running, Role: LEADER
I20240502 21:28:12.746716 31367 heartbeater.cc:503] Master 127.15.255.190:35921 was elected leader, sending a full tablet report...
I20240502 21:28:12.746876 31383 consensus_queue.cc:237] T 6c5cb1507e5a470abaa5de5ec29b99d7 P 17faa5ec5e6a42c9846441cf2da4eb4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17faa5ec5e6a42c9846441cf2da4eb4d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 45045 } }
I20240502 21:28:12.747901 31381 ts_tablet_manager.cc:1398] T 6c5cb1507e5a470abaa5de5ec29b99d7 P 17faa5ec5e6a42c9846441cf2da4eb4d: Time spent starting tablet: real 0.012s user 0.008s sys 0.004s
I20240502 21:28:12.751571 30781 catalog_manager.cc:5484] T 6c5cb1507e5a470abaa5de5ec29b99d7 P 17faa5ec5e6a42c9846441cf2da4eb4d reported cstate change: term changed from 0 to 1, leader changed from <none> to 17faa5ec5e6a42c9846441cf2da4eb4d (127.15.255.132). New cstate: current_term: 1 leader_uuid: "17faa5ec5e6a42c9846441cf2da4eb4d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17faa5ec5e6a42c9846441cf2da4eb4d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 45045 } health_report { overall_health: HEALTHY } } }
I20240502 21:28:12.762174 16382 external_mini_cluster.cc:930] 4 TS(s) registered with all masters
I20240502 21:28:12.763891 16382 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 2bff96c137244df6b6beef103aecd997 to finish bootstrapping
I20240502 21:28:12.772162 16382 ts_itest-base.cc:233] Waiting for 0 tablets on tserver a27039d23a9041e4831027412c5416c9 to finish bootstrapping
I20240502 21:28:12.779963 16382 ts_itest-base.cc:233] Waiting for 0 tablets on tserver bbfd9ca2d42547dca3f678bd6ed2964e to finish bootstrapping
I20240502 21:28:12.788587 16382 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 17faa5ec5e6a42c9846441cf2da4eb4d to finish bootstrapping
I20240502 21:28:12.795212 16382 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvNciQZ/build/asan/bin/kudu pid 31105
I20240502 21:28:12.796026 31228 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 21:28:12.802138 31101 heartbeater.cc:503] Master 127.15.255.190:35921 was elected leader, sending a full tablet report...
I20240502 21:28:12.873605 16382 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvNciQZ/build/asan/bin/kudu with pid 31105
I20240502 21:28:12.888650 16382 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvNciQZ/build/asan/bin/kudu pid 31236
I20240502 21:28:12.889709 31362 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 21:28:12.973667 16382 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvNciQZ/build/asan/bin/kudu with pid 31236
I20240502 21:28:16.739965 30781 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46594:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 21:28:16.770251 31035 tablet_service.cc:1467] Processing CreateTablet for tablet 19573c9e33484d14b7f5b3b866597593 (DEFAULT_TABLE table=rebalance_test_table_0 [id=4832d4eadc654d4db08ef4446ad39406]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:16.770416 30903 tablet_service.cc:1467] Processing CreateTablet for tablet 9fbb7fe776a541cfb3a383b5e7ab611e (DEFAULT_TABLE table=rebalance_test_table_0 [id=4832d4eadc654d4db08ef4446ad39406]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:16.770453 30902 tablet_service.cc:1467] Processing CreateTablet for tablet 6566df65606643b584597fa2fc5a307d (DEFAULT_TABLE table=rebalance_test_table_0 [id=4832d4eadc654d4db08ef4446ad39406]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:16.771488 31035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19573c9e33484d14b7f5b3b866597593. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:16.771823 30902 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6566df65606643b584597fa2fc5a307d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:16.772272 30903 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9fbb7fe776a541cfb3a383b5e7ab611e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:16.782548 31402 tablet_bootstrap.cc:492] T 6566df65606643b584597fa2fc5a307d P 2bff96c137244df6b6beef103aecd997: Bootstrap starting.
I20240502 21:28:16.783886 31403 tablet_bootstrap.cc:492] T 19573c9e33484d14b7f5b3b866597593 P a27039d23a9041e4831027412c5416c9: Bootstrap starting.
I20240502 21:28:16.785768 31402 tablet_bootstrap.cc:654] T 6566df65606643b584597fa2fc5a307d P 2bff96c137244df6b6beef103aecd997: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:16.786466 31403 tablet_bootstrap.cc:654] T 19573c9e33484d14b7f5b3b866597593 P a27039d23a9041e4831027412c5416c9: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:16.787240 31402 log.cc:824] T 6566df65606643b584597fa2fc5a307d P 2bff96c137244df6b6beef103aecd997: Log is configured to *not* fsync() on all Append() calls
I20240502 21:28:16.787606 31403 log.cc:824] T 19573c9e33484d14b7f5b3b866597593 P a27039d23a9041e4831027412c5416c9: Log is configured to *not* fsync() on all Append() calls
I20240502 21:28:16.793018 31403 tablet_bootstrap.cc:492] T 19573c9e33484d14b7f5b3b866597593 P a27039d23a9041e4831027412c5416c9: No bootstrap required, opened a new log
I20240502 21:28:16.793258 31403 ts_tablet_manager.cc:1367] T 19573c9e33484d14b7f5b3b866597593 P a27039d23a9041e4831027412c5416c9: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20240502 21:28:16.794078 31402 tablet_bootstrap.cc:492] T 6566df65606643b584597fa2fc5a307d P 2bff96c137244df6b6beef103aecd997: No bootstrap required, opened a new log
I20240502 21:28:16.794309 31402 ts_tablet_manager.cc:1367] T 6566df65606643b584597fa2fc5a307d P 2bff96c137244df6b6beef103aecd997: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.000s
I20240502 21:28:16.800541 31403 raft_consensus.cc:348] T 19573c9e33484d14b7f5b3b866597593 P a27039d23a9041e4831027412c5416c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a27039d23a9041e4831027412c5416c9" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 42697 } }
I20240502 21:28:16.800964 31403 raft_consensus.cc:374] T 19573c9e33484d14b7f5b3b866597593 P a27039d23a9041e4831027412c5416c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:16.801093 31403 raft_consensus.cc:729] T 19573c9e33484d14b7f5b3b866597593 P a27039d23a9041e4831027412c5416c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a27039d23a9041e4831027412c5416c9, State: Initialized, Role: FOLLOWER
I20240502 21:28:16.802584 31403 consensus_queue.cc:260] T 19573c9e33484d14b7f5b3b866597593 P a27039d23a9041e4831027412c5416c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a27039d23a9041e4831027412c5416c9" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 42697 } }
I20240502 21:28:16.802872 31403 raft_consensus.cc:388] T 19573c9e33484d14b7f5b3b866597593 P a27039d23a9041e4831027412c5416c9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 21:28:16.803004 31403 raft_consensus.cc:482] T 19573c9e33484d14b7f5b3b866597593 P a27039d23a9041e4831027412c5416c9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 21:28:16.803153 31403 raft_consensus.cc:3037] T 19573c9e33484d14b7f5b3b866597593 P a27039d23a9041e4831027412c5416c9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:16.802940 31402 raft_consensus.cc:348] T 6566df65606643b584597fa2fc5a307d P 2bff96c137244df6b6beef103aecd997 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bff96c137244df6b6beef103aecd997" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 44455 } }
I20240502 21:28:16.803284 31402 raft_consensus.cc:374] T 6566df65606643b584597fa2fc5a307d P 2bff96c137244df6b6beef103aecd997 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:16.803382 31402 raft_consensus.cc:729] T 6566df65606643b584597fa2fc5a307d P 2bff96c137244df6b6beef103aecd997 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2bff96c137244df6b6beef103aecd997, State: Initialized, Role: FOLLOWER
I20240502 21:28:16.804028 31402 consensus_queue.cc:260] T 6566df65606643b584597fa2fc5a307d P 2bff96c137244df6b6beef103aecd997 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bff96c137244df6b6beef103aecd997" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 44455 } }
I20240502 21:28:16.804261 31402 raft_consensus.cc:388] T 6566df65606643b584597fa2fc5a307d P 2bff96c137244df6b6beef103aecd997 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 21:28:16.804388 31402 raft_consensus.cc:482] T 6566df65606643b584597fa2fc5a307d P 2bff96c137244df6b6beef103aecd997 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 21:28:16.804582 31402 raft_consensus.cc:3037] T 6566df65606643b584597fa2fc5a307d P 2bff96c137244df6b6beef103aecd997 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:16.805356 31403 raft_consensus.cc:504] T 19573c9e33484d14b7f5b3b866597593 P a27039d23a9041e4831027412c5416c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a27039d23a9041e4831027412c5416c9" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 42697 } }
I20240502 21:28:16.805778 31403 leader_election.cc:302] T 19573c9e33484d14b7f5b3b866597593 P a27039d23a9041e4831027412c5416c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a27039d23a9041e4831027412c5416c9; no voters:
I20240502 21:28:16.806368 31403 leader_election.cc:288] T 19573c9e33484d14b7f5b3b866597593 P a27039d23a9041e4831027412c5416c9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 21:28:16.806491 31406 raft_consensus.cc:2781] T 19573c9e33484d14b7f5b3b866597593 P a27039d23a9041e4831027412c5416c9 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:28:16.807313 31402 raft_consensus.cc:504] T 6566df65606643b584597fa2fc5a307d P 2bff96c137244df6b6beef103aecd997 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bff96c137244df6b6beef103aecd997" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 44455 } }
I20240502 21:28:16.807837 31402 leader_election.cc:302] T 6566df65606643b584597fa2fc5a307d P 2bff96c137244df6b6beef103aecd997 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2bff96c137244df6b6beef103aecd997; no voters:
I20240502 21:28:16.815632 31403 ts_tablet_manager.cc:1398] T 19573c9e33484d14b7f5b3b866597593 P a27039d23a9041e4831027412c5416c9: Time spent starting tablet: real 0.022s user 0.016s sys 0.003s
I20240502 21:28:16.816150 31406 raft_consensus.cc:686] T 19573c9e33484d14b7f5b3b866597593 P a27039d23a9041e4831027412c5416c9 [term 1 LEADER]: Becoming Leader. State: Replica: a27039d23a9041e4831027412c5416c9, State: Running, Role: LEADER
I20240502 21:28:16.816515 31402 leader_election.cc:288] T 6566df65606643b584597fa2fc5a307d P 2bff96c137244df6b6beef103aecd997 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 21:28:16.816632 31409 raft_consensus.cc:2781] T 6566df65606643b584597fa2fc5a307d P 2bff96c137244df6b6beef103aecd997 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:28:16.816744 31406 consensus_queue.cc:237] T 19573c9e33484d14b7f5b3b866597593 P a27039d23a9041e4831027412c5416c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a27039d23a9041e4831027412c5416c9" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 42697 } }
I20240502 21:28:16.823246 31409 raft_consensus.cc:686] T 6566df65606643b584597fa2fc5a307d P 2bff96c137244df6b6beef103aecd997 [term 1 LEADER]: Becoming Leader. State: Replica: 2bff96c137244df6b6beef103aecd997, State: Running, Role: LEADER
I20240502 21:28:16.823870 31409 consensus_queue.cc:237] T 6566df65606643b584597fa2fc5a307d P 2bff96c137244df6b6beef103aecd997 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bff96c137244df6b6beef103aecd997" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 44455 } }
I20240502 21:28:16.825068 31402 ts_tablet_manager.cc:1398] T 6566df65606643b584597fa2fc5a307d P 2bff96c137244df6b6beef103aecd997: Time spent starting tablet: real 0.031s user 0.016s sys 0.014s
I20240502 21:28:16.825428 31402 tablet_bootstrap.cc:492] T 9fbb7fe776a541cfb3a383b5e7ab611e P 2bff96c137244df6b6beef103aecd997: Bootstrap starting.
I20240502 21:28:16.827183 31402 tablet_bootstrap.cc:654] T 9fbb7fe776a541cfb3a383b5e7ab611e P 2bff96c137244df6b6beef103aecd997: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:16.829154 30781 catalog_manager.cc:5484] T 6566df65606643b584597fa2fc5a307d P 2bff96c137244df6b6beef103aecd997 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2bff96c137244df6b6beef103aecd997 (127.15.255.129). New cstate: current_term: 1 leader_uuid: "2bff96c137244df6b6beef103aecd997" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bff96c137244df6b6beef103aecd997" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 44455 } health_report { overall_health: HEALTHY } } }
I20240502 21:28:16.831269 30780 catalog_manager.cc:5484] T 19573c9e33484d14b7f5b3b866597593 P a27039d23a9041e4831027412c5416c9 reported cstate change: term changed from 0 to 1, leader changed from <none> to a27039d23a9041e4831027412c5416c9 (127.15.255.130). New cstate: current_term: 1 leader_uuid: "a27039d23a9041e4831027412c5416c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a27039d23a9041e4831027412c5416c9" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 42697 } health_report { overall_health: HEALTHY } } }
I20240502 21:28:16.833312 31402 tablet_bootstrap.cc:492] T 9fbb7fe776a541cfb3a383b5e7ab611e P 2bff96c137244df6b6beef103aecd997: No bootstrap required, opened a new log
I20240502 21:28:16.833464 31402 ts_tablet_manager.cc:1367] T 9fbb7fe776a541cfb3a383b5e7ab611e P 2bff96c137244df6b6beef103aecd997: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.001s
I20240502 21:28:16.834635 31402 raft_consensus.cc:348] T 9fbb7fe776a541cfb3a383b5e7ab611e P 2bff96c137244df6b6beef103aecd997 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bff96c137244df6b6beef103aecd997" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 44455 } }
I20240502 21:28:16.834767 31402 raft_consensus.cc:374] T 9fbb7fe776a541cfb3a383b5e7ab611e P 2bff96c137244df6b6beef103aecd997 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:16.834882 31402 raft_consensus.cc:729] T 9fbb7fe776a541cfb3a383b5e7ab611e P 2bff96c137244df6b6beef103aecd997 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2bff96c137244df6b6beef103aecd997, State: Initialized, Role: FOLLOWER
I20240502 21:28:16.835857 31402 consensus_queue.cc:260] T 9fbb7fe776a541cfb3a383b5e7ab611e P 2bff96c137244df6b6beef103aecd997 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bff96c137244df6b6beef103aecd997" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 44455 } }
I20240502 21:28:16.836068 31402 raft_consensus.cc:388] T 9fbb7fe776a541cfb3a383b5e7ab611e P 2bff96c137244df6b6beef103aecd997 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 21:28:16.836217 31402 raft_consensus.cc:482] T 9fbb7fe776a541cfb3a383b5e7ab611e P 2bff96c137244df6b6beef103aecd997 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 21:28:16.836336 31402 raft_consensus.cc:3037] T 9fbb7fe776a541cfb3a383b5e7ab611e P 2bff96c137244df6b6beef103aecd997 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:16.838166 31402 raft_consensus.cc:504] T 9fbb7fe776a541cfb3a383b5e7ab611e P 2bff96c137244df6b6beef103aecd997 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bff96c137244df6b6beef103aecd997" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 44455 } }
I20240502 21:28:16.838356 31402 leader_election.cc:302] T 9fbb7fe776a541cfb3a383b5e7ab611e P 2bff96c137244df6b6beef103aecd997 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2bff96c137244df6b6beef103aecd997; no voters:
I20240502 21:28:16.838487 31402 leader_election.cc:288] T 9fbb7fe776a541cfb3a383b5e7ab611e P 2bff96c137244df6b6beef103aecd997 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 21:28:16.838564 31409 raft_consensus.cc:2781] T 9fbb7fe776a541cfb3a383b5e7ab611e P 2bff96c137244df6b6beef103aecd997 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:28:16.838749 31409 raft_consensus.cc:686] T 9fbb7fe776a541cfb3a383b5e7ab611e P 2bff96c137244df6b6beef103aecd997 [term 1 LEADER]: Becoming Leader. State: Replica: 2bff96c137244df6b6beef103aecd997, State: Running, Role: LEADER
I20240502 21:28:16.838912 31402 ts_tablet_manager.cc:1398] T 9fbb7fe776a541cfb3a383b5e7ab611e P 2bff96c137244df6b6beef103aecd997: Time spent starting tablet: real 0.005s user 0.002s sys 0.002s
I20240502 21:28:16.838955 31409 consensus_queue.cc:237] T 9fbb7fe776a541cfb3a383b5e7ab611e P 2bff96c137244df6b6beef103aecd997 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bff96c137244df6b6beef103aecd997" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 44455 } }
I20240502 21:28:16.840631 30781 catalog_manager.cc:5484] T 9fbb7fe776a541cfb3a383b5e7ab611e P 2bff96c137244df6b6beef103aecd997 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2bff96c137244df6b6beef103aecd997 (127.15.255.129). New cstate: current_term: 1 leader_uuid: "2bff96c137244df6b6beef103aecd997" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bff96c137244df6b6beef103aecd997" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 44455 } health_report { overall_health: HEALTHY } } }
I20240502 21:28:18.185957 30781 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46594:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 21:28:18.194789 31034 tablet_service.cc:1467] Processing CreateTablet for tablet 21b9b0408e6f4ffaafaebec4b02c3c83 (DEFAULT_TABLE table=rebalance_test_table_1 [id=8016910c70a94d929ffd34f244caf3b1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:18.194725 31035 tablet_service.cc:1467] Processing CreateTablet for tablet ad16c12ce187416caabf8074b9499f3b (DEFAULT_TABLE table=rebalance_test_table_1 [id=8016910c70a94d929ffd34f244caf3b1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:18.195039 30902 tablet_service.cc:1467] Processing CreateTablet for tablet c450518f0a1b4c6889538b31d86e7e29 (DEFAULT_TABLE table=rebalance_test_table_1 [id=8016910c70a94d929ffd34f244caf3b1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:18.195240 31034 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21b9b0408e6f4ffaafaebec4b02c3c83. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:18.195503 30902 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c450518f0a1b4c6889538b31d86e7e29. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:18.195539 31035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad16c12ce187416caabf8074b9499f3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:18.201848 31443 tablet_bootstrap.cc:492] T 21b9b0408e6f4ffaafaebec4b02c3c83 P a27039d23a9041e4831027412c5416c9: Bootstrap starting.
I20240502 21:28:18.201921 31444 tablet_bootstrap.cc:492] T c450518f0a1b4c6889538b31d86e7e29 P 2bff96c137244df6b6beef103aecd997: Bootstrap starting.
I20240502 21:28:18.203065 31443 tablet_bootstrap.cc:654] T 21b9b0408e6f4ffaafaebec4b02c3c83 P a27039d23a9041e4831027412c5416c9: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:18.203220 31444 tablet_bootstrap.cc:654] T c450518f0a1b4c6889538b31d86e7e29 P 2bff96c137244df6b6beef103aecd997: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:18.204953 31444 tablet_bootstrap.cc:492] T c450518f0a1b4c6889538b31d86e7e29 P 2bff96c137244df6b6beef103aecd997: No bootstrap required, opened a new log
I20240502 21:28:18.205073 31444 ts_tablet_manager.cc:1367] T c450518f0a1b4c6889538b31d86e7e29 P 2bff96c137244df6b6beef103aecd997: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.001s
I20240502 21:28:18.205282 31443 tablet_bootstrap.cc:492] T 21b9b0408e6f4ffaafaebec4b02c3c83 P a27039d23a9041e4831027412c5416c9: No bootstrap required, opened a new log
I20240502 21:28:18.205410 31443 ts_tablet_manager.cc:1367] T 21b9b0408e6f4ffaafaebec4b02c3c83 P a27039d23a9041e4831027412c5416c9: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20240502 21:28:18.205716 31444 raft_consensus.cc:348] T c450518f0a1b4c6889538b31d86e7e29 P 2bff96c137244df6b6beef103aecd997 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bff96c137244df6b6beef103aecd997" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 44455 } }
I20240502 21:28:18.205852 31444 raft_consensus.cc:374] T c450518f0a1b4c6889538b31d86e7e29 P 2bff96c137244df6b6beef103aecd997 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:18.205930 31444 raft_consensus.cc:729] T c450518f0a1b4c6889538b31d86e7e29 P 2bff96c137244df6b6beef103aecd997 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2bff96c137244df6b6beef103aecd997, State: Initialized, Role: FOLLOWER
I20240502 21:28:18.206122 31444 consensus_queue.cc:260] T c450518f0a1b4c6889538b31d86e7e29 P 2bff96c137244df6b6beef103aecd997 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bff96c137244df6b6beef103aecd997" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 44455 } }
I20240502 21:28:18.206187 31443 raft_consensus.cc:348] T 21b9b0408e6f4ffaafaebec4b02c3c83 P a27039d23a9041e4831027412c5416c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a27039d23a9041e4831027412c5416c9" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 42697 } }
I20240502 21:28:18.206264 31444 raft_consensus.cc:388] T c450518f0a1b4c6889538b31d86e7e29 P 2bff96c137244df6b6beef103aecd997 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 21:28:18.206342 31443 raft_consensus.cc:374] T 21b9b0408e6f4ffaafaebec4b02c3c83 P a27039d23a9041e4831027412c5416c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:18.206333 31444 raft_consensus.cc:482] T c450518f0a1b4c6889538b31d86e7e29 P 2bff96c137244df6b6beef103aecd997 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 21:28:18.206440 31444 raft_consensus.cc:3037] T c450518f0a1b4c6889538b31d86e7e29 P 2bff96c137244df6b6beef103aecd997 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:18.206434 31443 raft_consensus.cc:729] T 21b9b0408e6f4ffaafaebec4b02c3c83 P a27039d23a9041e4831027412c5416c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a27039d23a9041e4831027412c5416c9, State: Initialized, Role: FOLLOWER
I20240502 21:28:18.206650 31443 consensus_queue.cc:260] T 21b9b0408e6f4ffaafaebec4b02c3c83 P a27039d23a9041e4831027412c5416c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a27039d23a9041e4831027412c5416c9" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 42697 } }
I20240502 21:28:18.206806 31443 raft_consensus.cc:388] T 21b9b0408e6f4ffaafaebec4b02c3c83 P a27039d23a9041e4831027412c5416c9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 21:28:18.206880 31443 raft_consensus.cc:482] T 21b9b0408e6f4ffaafaebec4b02c3c83 P a27039d23a9041e4831027412c5416c9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 21:28:18.206977 31443 raft_consensus.cc:3037] T 21b9b0408e6f4ffaafaebec4b02c3c83 P a27039d23a9041e4831027412c5416c9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:18.208230 31444 raft_consensus.cc:504] T c450518f0a1b4c6889538b31d86e7e29 P 2bff96c137244df6b6beef103aecd997 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bff96c137244df6b6beef103aecd997" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 44455 } }
I20240502 21:28:18.208492 31444 leader_election.cc:302] T c450518f0a1b4c6889538b31d86e7e29 P 2bff96c137244df6b6beef103aecd997 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2bff96c137244df6b6beef103aecd997; no voters:
I20240502 21:28:18.208817 31443 raft_consensus.cc:504] T 21b9b0408e6f4ffaafaebec4b02c3c83 P a27039d23a9041e4831027412c5416c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a27039d23a9041e4831027412c5416c9" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 42697 } }
I20240502 21:28:18.209074 31443 leader_election.cc:302] T 21b9b0408e6f4ffaafaebec4b02c3c83 P a27039d23a9041e4831027412c5416c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a27039d23a9041e4831027412c5416c9; no voters:
I20240502 21:28:18.209260 31444 leader_election.cc:288] T c450518f0a1b4c6889538b31d86e7e29 P 2bff96c137244df6b6beef103aecd997 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 21:28:18.209425 31447 raft_consensus.cc:2781] T c450518f0a1b4c6889538b31d86e7e29 P 2bff96c137244df6b6beef103aecd997 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:28:18.210086 31444 ts_tablet_manager.cc:1398] T c450518f0a1b4c6889538b31d86e7e29 P 2bff96c137244df6b6beef103aecd997: Time spent starting tablet: real 0.005s user 0.001s sys 0.003s
I20240502 21:28:18.210382 31447 raft_consensus.cc:686] T c450518f0a1b4c6889538b31d86e7e29 P 2bff96c137244df6b6beef103aecd997 [term 1 LEADER]: Becoming Leader. State: Replica: 2bff96c137244df6b6beef103aecd997, State: Running, Role: LEADER
I20240502 21:28:18.210597 31443 leader_election.cc:288] T 21b9b0408e6f4ffaafaebec4b02c3c83 P a27039d23a9041e4831027412c5416c9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 21:28:18.210727 31448 raft_consensus.cc:2781] T 21b9b0408e6f4ffaafaebec4b02c3c83 P a27039d23a9041e4831027412c5416c9 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:28:18.210676 31447 consensus_queue.cc:237] T c450518f0a1b4c6889538b31d86e7e29 P 2bff96c137244df6b6beef103aecd997 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bff96c137244df6b6beef103aecd997" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 44455 } }
I20240502 21:28:18.211155 31448 raft_consensus.cc:686] T 21b9b0408e6f4ffaafaebec4b02c3c83 P a27039d23a9041e4831027412c5416c9 [term 1 LEADER]: Becoming Leader. State: Replica: a27039d23a9041e4831027412c5416c9, State: Running, Role: LEADER
I20240502 21:28:18.211469 31448 consensus_queue.cc:237] T 21b9b0408e6f4ffaafaebec4b02c3c83 P a27039d23a9041e4831027412c5416c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a27039d23a9041e4831027412c5416c9" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 42697 } }
I20240502 21:28:18.213984 30780 catalog_manager.cc:5484] T c450518f0a1b4c6889538b31d86e7e29 P 2bff96c137244df6b6beef103aecd997 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2bff96c137244df6b6beef103aecd997 (127.15.255.129). New cstate: current_term: 1 leader_uuid: "2bff96c137244df6b6beef103aecd997" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bff96c137244df6b6beef103aecd997" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 44455 } health_report { overall_health: HEALTHY } } }
I20240502 21:28:18.214211 31443 ts_tablet_manager.cc:1398] T 21b9b0408e6f4ffaafaebec4b02c3c83 P a27039d23a9041e4831027412c5416c9: Time spent starting tablet: real 0.009s user 0.007s sys 0.002s
I20240502 21:28:18.214480 31443 tablet_bootstrap.cc:492] T ad16c12ce187416caabf8074b9499f3b P a27039d23a9041e4831027412c5416c9: Bootstrap starting.
I20240502 21:28:18.213948 30781 catalog_manager.cc:5484] T 21b9b0408e6f4ffaafaebec4b02c3c83 P a27039d23a9041e4831027412c5416c9 reported cstate change: term changed from 0 to 1, leader changed from <none> to a27039d23a9041e4831027412c5416c9 (127.15.255.130). New cstate: current_term: 1 leader_uuid: "a27039d23a9041e4831027412c5416c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a27039d23a9041e4831027412c5416c9" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 42697 } health_report { overall_health: HEALTHY } } }
I20240502 21:28:18.216193 31443 tablet_bootstrap.cc:654] T ad16c12ce187416caabf8074b9499f3b P a27039d23a9041e4831027412c5416c9: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:18.227319 31443 tablet_bootstrap.cc:492] T ad16c12ce187416caabf8074b9499f3b P a27039d23a9041e4831027412c5416c9: No bootstrap required, opened a new log
I20240502 21:28:18.227473 31443 ts_tablet_manager.cc:1367] T ad16c12ce187416caabf8074b9499f3b P a27039d23a9041e4831027412c5416c9: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.008s
I20240502 21:28:18.228077 31443 raft_consensus.cc:348] T ad16c12ce187416caabf8074b9499f3b P a27039d23a9041e4831027412c5416c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a27039d23a9041e4831027412c5416c9" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 42697 } }
I20240502 21:28:18.228230 31443 raft_consensus.cc:374] T ad16c12ce187416caabf8074b9499f3b P a27039d23a9041e4831027412c5416c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:18.228299 31443 raft_consensus.cc:729] T ad16c12ce187416caabf8074b9499f3b P a27039d23a9041e4831027412c5416c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a27039d23a9041e4831027412c5416c9, State: Initialized, Role: FOLLOWER
I20240502 21:28:18.228546 31443 consensus_queue.cc:260] T ad16c12ce187416caabf8074b9499f3b P a27039d23a9041e4831027412c5416c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a27039d23a9041e4831027412c5416c9" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 42697 } }
I20240502 21:28:18.228688 31443 raft_consensus.cc:388] T ad16c12ce187416caabf8074b9499f3b P a27039d23a9041e4831027412c5416c9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 21:28:18.228778 31443 raft_consensus.cc:482] T ad16c12ce187416caabf8074b9499f3b P a27039d23a9041e4831027412c5416c9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 21:28:18.228865 31443 raft_consensus.cc:3037] T ad16c12ce187416caabf8074b9499f3b P a27039d23a9041e4831027412c5416c9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:18.230574 31443 raft_consensus.cc:504] T ad16c12ce187416caabf8074b9499f3b P a27039d23a9041e4831027412c5416c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a27039d23a9041e4831027412c5416c9" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 42697 } }
I20240502 21:28:18.230777 31443 leader_election.cc:302] T ad16c12ce187416caabf8074b9499f3b P a27039d23a9041e4831027412c5416c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a27039d23a9041e4831027412c5416c9; no voters:
I20240502 21:28:18.230962 31443 leader_election.cc:288] T ad16c12ce187416caabf8074b9499f3b P a27039d23a9041e4831027412c5416c9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 21:28:18.231007 31448 raft_consensus.cc:2781] T ad16c12ce187416caabf8074b9499f3b P a27039d23a9041e4831027412c5416c9 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:28:18.231180 31448 raft_consensus.cc:686] T ad16c12ce187416caabf8074b9499f3b P a27039d23a9041e4831027412c5416c9 [term 1 LEADER]: Becoming Leader. State: Replica: a27039d23a9041e4831027412c5416c9, State: Running, Role: LEADER
I20240502 21:28:18.231407 31443 ts_tablet_manager.cc:1398] T ad16c12ce187416caabf8074b9499f3b P a27039d23a9041e4831027412c5416c9: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 21:28:18.231444 31448 consensus_queue.cc:237] T ad16c12ce187416caabf8074b9499f3b P a27039d23a9041e4831027412c5416c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a27039d23a9041e4831027412c5416c9" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 42697 } }
I20240502 21:28:18.233395 30779 catalog_manager.cc:5484] T ad16c12ce187416caabf8074b9499f3b P a27039d23a9041e4831027412c5416c9 reported cstate change: term changed from 0 to 1, leader changed from <none> to a27039d23a9041e4831027412c5416c9 (127.15.255.130). New cstate: current_term: 1 leader_uuid: "a27039d23a9041e4831027412c5416c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a27039d23a9041e4831027412c5416c9" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 42697 } health_report { overall_health: HEALTHY } } }
I20240502 21:28:18.711031 30779 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46594:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 21:28:18.718647 30902 tablet_service.cc:1467] Processing CreateTablet for tablet 9c53de279b95409ca718939cfd90165b (DEFAULT_TABLE table=rebalance_test_table_2 [id=bc8e2465cdae4b77902184dd0ca18eb4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:18.718839 31035 tablet_service.cc:1467] Processing CreateTablet for tablet 886418f21f1e4eef95d1a0cd69417204 (DEFAULT_TABLE table=rebalance_test_table_2 [id=bc8e2465cdae4b77902184dd0ca18eb4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:18.719123 30902 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c53de279b95409ca718939cfd90165b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:18.719301 31035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 886418f21f1e4eef95d1a0cd69417204. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:18.719377 30903 tablet_service.cc:1467] Processing CreateTablet for tablet 493201e6661b4ba1b3c50bab507496a1 (DEFAULT_TABLE table=rebalance_test_table_2 [id=bc8e2465cdae4b77902184dd0ca18eb4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:18.719724 30903 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 493201e6661b4ba1b3c50bab507496a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:18.724396 31483 tablet_bootstrap.cc:492] T 9c53de279b95409ca718939cfd90165b P 2bff96c137244df6b6beef103aecd997: Bootstrap starting.
I20240502 21:28:18.725260 31443 tablet_bootstrap.cc:492] T 886418f21f1e4eef95d1a0cd69417204 P a27039d23a9041e4831027412c5416c9: Bootstrap starting.
I20240502 21:28:18.726084 31483 tablet_bootstrap.cc:654] T 9c53de279b95409ca718939cfd90165b P 2bff96c137244df6b6beef103aecd997: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:18.726742 31443 tablet_bootstrap.cc:654] T 886418f21f1e4eef95d1a0cd69417204 P a27039d23a9041e4831027412c5416c9: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:18.727813 31483 tablet_bootstrap.cc:492] T 9c53de279b95409ca718939cfd90165b P 2bff96c137244df6b6beef103aecd997: No bootstrap required, opened a new log
I20240502 21:28:18.727934 31483 ts_tablet_manager.cc:1367] T 9c53de279b95409ca718939cfd90165b P 2bff96c137244df6b6beef103aecd997: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20240502 21:28:18.728574 31443 tablet_bootstrap.cc:492] T 886418f21f1e4eef95d1a0cd69417204 P a27039d23a9041e4831027412c5416c9: No bootstrap required, opened a new log
I20240502 21:28:18.728559 31483 raft_consensus.cc:348] T 9c53de279b95409ca718939cfd90165b P 2bff96c137244df6b6beef103aecd997 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bff96c137244df6b6beef103aecd997" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 44455 } }
I20240502 21:28:18.728695 31443 ts_tablet_manager.cc:1367] T 886418f21f1e4eef95d1a0cd69417204 P a27039d23a9041e4831027412c5416c9: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.001s
I20240502 21:28:18.728715 31483 raft_consensus.cc:374] T 9c53de279b95409ca718939cfd90165b P 2bff96c137244df6b6beef103aecd997 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:18.728796 31483 raft_consensus.cc:729] T 9c53de279b95409ca718939cfd90165b P 2bff96c137244df6b6beef103aecd997 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2bff96c137244df6b6beef103aecd997, State: Initialized, Role: FOLLOWER
I20240502 21:28:18.729051 31483 consensus_queue.cc:260] T 9c53de279b95409ca718939cfd90165b P 2bff96c137244df6b6beef103aecd997 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bff96c137244df6b6beef103aecd997" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 44455 } }
I20240502 21:28:18.729208 31483 raft_consensus.cc:388] T 9c53de279b95409ca718939cfd90165b P 2bff96c137244df6b6beef103aecd997 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 21:28:18.729282 31483 raft_consensus.cc:482] T 9c53de279b95409ca718939cfd90165b P 2bff96c137244df6b6beef103aecd997 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 21:28:18.729380 31483 raft_consensus.cc:3037] T 9c53de279b95409ca718939cfd90165b P 2bff96c137244df6b6beef103aecd997 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:18.729372 31443 raft_consensus.cc:348] T 886418f21f1e4eef95d1a0cd69417204 P a27039d23a9041e4831027412c5416c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a27039d23a9041e4831027412c5416c9" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 42697 } }
I20240502 21:28:18.729535 31443 raft_consensus.cc:374] T 886418f21f1e4eef95d1a0cd69417204 P a27039d23a9041e4831027412c5416c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:18.729624 31443 raft_consensus.cc:729] T 886418f21f1e4eef95d1a0cd69417204 P a27039d23a9041e4831027412c5416c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a27039d23a9041e4831027412c5416c9, State: Initialized, Role: FOLLOWER
I20240502 21:28:18.730084 31443 consensus_queue.cc:260] T 886418f21f1e4eef95d1a0cd69417204 P a27039d23a9041e4831027412c5416c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a27039d23a9041e4831027412c5416c9" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 42697 } }
I20240502 21:28:18.730309 31443 raft_consensus.cc:388] T 886418f21f1e4eef95d1a0cd69417204 P a27039d23a9041e4831027412c5416c9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 21:28:18.730386 31443 raft_consensus.cc:482] T 886418f21f1e4eef95d1a0cd69417204 P a27039d23a9041e4831027412c5416c9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 21:28:18.730485 31443 raft_consensus.cc:3037] T 886418f21f1e4eef95d1a0cd69417204 P a27039d23a9041e4831027412c5416c9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:18.731338 31483 raft_consensus.cc:504] T 9c53de279b95409ca718939cfd90165b P 2bff96c137244df6b6beef103aecd997 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bff96c137244df6b6beef103aecd997" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 44455 } }
I20240502 21:28:18.731564 31483 leader_election.cc:302] T 9c53de279b95409ca718939cfd90165b P 2bff96c137244df6b6beef103aecd997 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2bff96c137244df6b6beef103aecd997; no voters:
I20240502 21:28:18.731947 31483 leader_election.cc:288] T 9c53de279b95409ca718939cfd90165b P 2bff96c137244df6b6beef103aecd997 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 21:28:18.732100 31486 raft_consensus.cc:2781] T 9c53de279b95409ca718939cfd90165b P 2bff96c137244df6b6beef103aecd997 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:28:18.732585 31443 raft_consensus.cc:504] T 886418f21f1e4eef95d1a0cd69417204 P a27039d23a9041e4831027412c5416c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a27039d23a9041e4831027412c5416c9" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 42697 } }
I20240502 21:28:18.732858 31443 leader_election.cc:302] T 886418f21f1e4eef95d1a0cd69417204 P a27039d23a9041e4831027412c5416c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a27039d23a9041e4831027412c5416c9; no voters:
I20240502 21:28:18.732888 31483 ts_tablet_manager.cc:1398] T 9c53de279b95409ca718939cfd90165b P 2bff96c137244df6b6beef103aecd997: Time spent starting tablet: real 0.005s user 0.004s sys 0.001s
I20240502 21:28:18.733124 31483 tablet_bootstrap.cc:492] T 493201e6661b4ba1b3c50bab507496a1 P 2bff96c137244df6b6beef103aecd997: Bootstrap starting.
I20240502 21:28:18.733112 31448 raft_consensus.cc:2781] T 886418f21f1e4eef95d1a0cd69417204 P a27039d23a9041e4831027412c5416c9 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:28:18.733629 31486 raft_consensus.cc:686] T 9c53de279b95409ca718939cfd90165b P 2bff96c137244df6b6beef103aecd997 [term 1 LEADER]: Becoming Leader. State: Replica: 2bff96c137244df6b6beef103aecd997, State: Running, Role: LEADER
I20240502 21:28:18.733899 31486 consensus_queue.cc:237] T 9c53de279b95409ca718939cfd90165b P 2bff96c137244df6b6beef103aecd997 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bff96c137244df6b6beef103aecd997" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 44455 } }
I20240502 21:28:18.733950 31448 raft_consensus.cc:686] T 886418f21f1e4eef95d1a0cd69417204 P a27039d23a9041e4831027412c5416c9 [term 1 LEADER]: Becoming Leader. State: Replica: a27039d23a9041e4831027412c5416c9, State: Running, Role: LEADER
I20240502 21:28:18.734232 31448 consensus_queue.cc:237] T 886418f21f1e4eef95d1a0cd69417204 P a27039d23a9041e4831027412c5416c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a27039d23a9041e4831027412c5416c9" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 42697 } }
I20240502 21:28:18.734679 31483 tablet_bootstrap.cc:654] T 493201e6661b4ba1b3c50bab507496a1 P 2bff96c137244df6b6beef103aecd997: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:18.734704 31443 leader_election.cc:288] T 886418f21f1e4eef95d1a0cd69417204 P a27039d23a9041e4831027412c5416c9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 21:28:18.736698 30779 catalog_manager.cc:5484] T 9c53de279b95409ca718939cfd90165b P 2bff96c137244df6b6beef103aecd997 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2bff96c137244df6b6beef103aecd997 (127.15.255.129). New cstate: current_term: 1 leader_uuid: "2bff96c137244df6b6beef103aecd997" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bff96c137244df6b6beef103aecd997" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 44455 } health_report { overall_health: HEALTHY } } }
I20240502 21:28:18.739419 31483 tablet_bootstrap.cc:492] T 493201e6661b4ba1b3c50bab507496a1 P 2bff96c137244df6b6beef103aecd997: No bootstrap required, opened a new log
I20240502 21:28:18.739682 31483 ts_tablet_manager.cc:1367] T 493201e6661b4ba1b3c50bab507496a1 P 2bff96c137244df6b6beef103aecd997: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.002s
I20240502 21:28:18.740516 31483 raft_consensus.cc:348] T 493201e6661b4ba1b3c50bab507496a1 P 2bff96c137244df6b6beef103aecd997 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bff96c137244df6b6beef103aecd997" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 44455 } }
I20240502 21:28:18.740780 31483 raft_consensus.cc:374] T 493201e6661b4ba1b3c50bab507496a1 P 2bff96c137244df6b6beef103aecd997 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:18.740906 31483 raft_consensus.cc:729] T 493201e6661b4ba1b3c50bab507496a1 P 2bff96c137244df6b6beef103aecd997 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2bff96c137244df6b6beef103aecd997, State: Initialized, Role: FOLLOWER
I20240502 21:28:18.741182 31483 consensus_queue.cc:260] T 493201e6661b4ba1b3c50bab507496a1 P 2bff96c137244df6b6beef103aecd997 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bff96c137244df6b6beef103aecd997" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 44455 } }
I20240502 21:28:18.741405 31483 raft_consensus.cc:388] T 493201e6661b4ba1b3c50bab507496a1 P 2bff96c137244df6b6beef103aecd997 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 21:28:18.741308 30781 catalog_manager.cc:5484] T 886418f21f1e4eef95d1a0cd69417204 P a27039d23a9041e4831027412c5416c9 reported cstate change: term changed from 0 to 1, leader changed from <none> to a27039d23a9041e4831027412c5416c9 (127.15.255.130). New cstate: current_term: 1 leader_uuid: "a27039d23a9041e4831027412c5416c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a27039d23a9041e4831027412c5416c9" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 42697 } health_report { overall_health: HEALTHY } } }
I20240502 21:28:18.741542 31483 raft_consensus.cc:482] T 493201e6661b4ba1b3c50bab507496a1 P 2bff96c137244df6b6beef103aecd997 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 21:28:18.741730 31483 raft_consensus.cc:3037] T 493201e6661b4ba1b3c50bab507496a1 P 2bff96c137244df6b6beef103aecd997 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:18.744185 31483 raft_consensus.cc:504] T 493201e6661b4ba1b3c50bab507496a1 P 2bff96c137244df6b6beef103aecd997 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bff96c137244df6b6beef103aecd997" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 44455 } }
I20240502 21:28:18.744376 31483 leader_election.cc:302] T 493201e6661b4ba1b3c50bab507496a1 P 2bff96c137244df6b6beef103aecd997 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2bff96c137244df6b6beef103aecd997; no voters:
I20240502 21:28:18.744632 31483 leader_election.cc:288] T 493201e6661b4ba1b3c50bab507496a1 P 2bff96c137244df6b6beef103aecd997 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 21:28:18.744657 31486 raft_consensus.cc:2781] T 493201e6661b4ba1b3c50bab507496a1 P 2bff96c137244df6b6beef103aecd997 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:28:18.744894 31486 raft_consensus.cc:686] T 493201e6661b4ba1b3c50bab507496a1 P 2bff96c137244df6b6beef103aecd997 [term 1 LEADER]: Becoming Leader. State: Replica: 2bff96c137244df6b6beef103aecd997, State: Running, Role: LEADER
I20240502 21:28:18.745190 31483 ts_tablet_manager.cc:1398] T 493201e6661b4ba1b3c50bab507496a1 P 2bff96c137244df6b6beef103aecd997: Time spent starting tablet: real 0.005s user 0.002s sys 0.003s
I20240502 21:28:18.745149 31486 consensus_queue.cc:237] T 493201e6661b4ba1b3c50bab507496a1 P 2bff96c137244df6b6beef103aecd997 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bff96c137244df6b6beef103aecd997" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 44455 } }
I20240502 21:28:18.747323 30779 catalog_manager.cc:5484] T 493201e6661b4ba1b3c50bab507496a1 P 2bff96c137244df6b6beef103aecd997 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2bff96c137244df6b6beef103aecd997 (127.15.255.129). New cstate: current_term: 1 leader_uuid: "2bff96c137244df6b6beef103aecd997" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bff96c137244df6b6beef103aecd997" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 44455 } health_report { overall_health: HEALTHY } } }
I20240502 21:28:18.749809 31443 ts_tablet_manager.cc:1398] T 886418f21f1e4eef95d1a0cd69417204 P a27039d23a9041e4831027412c5416c9: Time spent starting tablet: real 0.021s user 0.005s sys 0.010s
I20240502 21:28:20.070222 30779 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46594:
name: "rebalance_test_table_3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 21:28:20.080606 30903 tablet_service.cc:1467] Processing CreateTablet for tablet b71736f0c4c245a29cca4198dc0dfbf8 (DEFAULT_TABLE table=rebalance_test_table_3 [id=87667fd8192d4d81848423acef48b282]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:20.080607 31035 tablet_service.cc:1467] Processing CreateTablet for tablet f6db5ed2663f4019a13adeaae12c4e44 (DEFAULT_TABLE table=rebalance_test_table_3 [id=87667fd8192d4d81848423acef48b282]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:20.081004 30903 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b71736f0c4c245a29cca4198dc0dfbf8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:20.081024 31035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6db5ed2663f4019a13adeaae12c4e44. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:20.084887 31034 tablet_service.cc:1467] Processing CreateTablet for tablet 87102d031b724bce8dd11d4f2a5c2e1a (DEFAULT_TABLE table=rebalance_test_table_3 [id=87667fd8192d4d81848423acef48b282]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:20.085273 31034 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87102d031b724bce8dd11d4f2a5c2e1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:20.085968 31521 tablet_bootstrap.cc:492] T b71736f0c4c245a29cca4198dc0dfbf8 P 2bff96c137244df6b6beef103aecd997: Bootstrap starting.
I20240502 21:28:20.087594 31521 tablet_bootstrap.cc:654] T b71736f0c4c245a29cca4198dc0dfbf8 P 2bff96c137244df6b6beef103aecd997: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:20.089644 31521 tablet_bootstrap.cc:492] T b71736f0c4c245a29cca4198dc0dfbf8 P 2bff96c137244df6b6beef103aecd997: No bootstrap required, opened a new log
I20240502 21:28:20.089841 31521 ts_tablet_manager.cc:1367] T b71736f0c4c245a29cca4198dc0dfbf8 P 2bff96c137244df6b6beef103aecd997: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240502 21:28:20.090648 31521 raft_consensus.cc:348] T b71736f0c4c245a29cca4198dc0dfbf8 P 2bff96c137244df6b6beef103aecd997 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bff96c137244df6b6beef103aecd997" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 44455 } }
I20240502 21:28:20.090893 31521 raft_consensus.cc:374] T b71736f0c4c245a29cca4198dc0dfbf8 P 2bff96c137244df6b6beef103aecd997 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:20.091028 31521 raft_consensus.cc:729] T b71736f0c4c245a29cca4198dc0dfbf8 P 2bff96c137244df6b6beef103aecd997 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2bff96c137244df6b6beef103aecd997, State: Initialized, Role: FOLLOWER
I20240502 21:28:20.091336 31521 consensus_queue.cc:260] T b71736f0c4c245a29cca4198dc0dfbf8 P 2bff96c137244df6b6beef103aecd997 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bff96c137244df6b6beef103aecd997" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 44455 } }
I20240502 21:28:20.091599 31521 raft_consensus.cc:388] T b71736f0c4c245a29cca4198dc0dfbf8 P 2bff96c137244df6b6beef103aecd997 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 21:28:20.091725 31521 raft_consensus.cc:482] T b71736f0c4c245a29cca4198dc0dfbf8 P 2bff96c137244df6b6beef103aecd997 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 21:28:20.091878 31521 raft_consensus.cc:3037] T b71736f0c4c245a29cca4198dc0dfbf8 P 2bff96c137244df6b6beef103aecd997 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:20.094420 31521 raft_consensus.cc:504] T b71736f0c4c245a29cca4198dc0dfbf8 P 2bff96c137244df6b6beef103aecd997 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bff96c137244df6b6beef103aecd997" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 44455 } }
I20240502 21:28:20.094751 31521 leader_election.cc:302] T b71736f0c4c245a29cca4198dc0dfbf8 P 2bff96c137244df6b6beef103aecd997 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2bff96c137244df6b6beef103aecd997; no voters:
I20240502 21:28:20.095225 31521 leader_election.cc:288] T b71736f0c4c245a29cca4198dc0dfbf8 P 2bff96c137244df6b6beef103aecd997 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 21:28:20.095363 31524 raft_consensus.cc:2781] T b71736f0c4c245a29cca4198dc0dfbf8 P 2bff96c137244df6b6beef103aecd997 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:28:20.095542 31522 tablet_bootstrap.cc:492] T f6db5ed2663f4019a13adeaae12c4e44 P a27039d23a9041e4831027412c5416c9: Bootstrap starting.
I20240502 21:28:20.096128 31524 raft_consensus.cc:686] T b71736f0c4c245a29cca4198dc0dfbf8 P 2bff96c137244df6b6beef103aecd997 [term 1 LEADER]: Becoming Leader. State: Replica: 2bff96c137244df6b6beef103aecd997, State: Running, Role: LEADER
I20240502 21:28:20.096450 31524 consensus_queue.cc:237] T b71736f0c4c245a29cca4198dc0dfbf8 P 2bff96c137244df6b6beef103aecd997 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bff96c137244df6b6beef103aecd997" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 44455 } }
I20240502 21:28:20.098140 31522 tablet_bootstrap.cc:654] T f6db5ed2663f4019a13adeaae12c4e44 P a27039d23a9041e4831027412c5416c9: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:20.098731 30779 catalog_manager.cc:5484] T b71736f0c4c245a29cca4198dc0dfbf8 P 2bff96c137244df6b6beef103aecd997 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2bff96c137244df6b6beef103aecd997 (127.15.255.129). New cstate: current_term: 1 leader_uuid: "2bff96c137244df6b6beef103aecd997" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bff96c137244df6b6beef103aecd997" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 44455 } health_report { overall_health: HEALTHY } } }
I20240502 21:28:20.099800 31521 ts_tablet_manager.cc:1398] T b71736f0c4c245a29cca4198dc0dfbf8 P 2bff96c137244df6b6beef103aecd997: Time spent starting tablet: real 0.010s user 0.009s sys 0.000s
I20240502 21:28:20.104856 31522 tablet_bootstrap.cc:492] T f6db5ed2663f4019a13adeaae12c4e44 P a27039d23a9041e4831027412c5416c9: No bootstrap required, opened a new log
I20240502 21:28:20.105476 31522 ts_tablet_manager.cc:1367] T f6db5ed2663f4019a13adeaae12c4e44 P a27039d23a9041e4831027412c5416c9: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.004s
I20240502 21:28:20.106382 31522 raft_consensus.cc:348] T f6db5ed2663f4019a13adeaae12c4e44 P a27039d23a9041e4831027412c5416c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a27039d23a9041e4831027412c5416c9" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 42697 } }
I20240502 21:28:20.106529 31522 raft_consensus.cc:374] T f6db5ed2663f4019a13adeaae12c4e44 P a27039d23a9041e4831027412c5416c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:20.106618 31522 raft_consensus.cc:729] T f6db5ed2663f4019a13adeaae12c4e44 P a27039d23a9041e4831027412c5416c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a27039d23a9041e4831027412c5416c9, State: Initialized, Role: FOLLOWER
I20240502 21:28:20.106833 31522 consensus_queue.cc:260] T f6db5ed2663f4019a13adeaae12c4e44 P a27039d23a9041e4831027412c5416c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a27039d23a9041e4831027412c5416c9" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 42697 } }
I20240502 21:28:20.106988 31522 raft_consensus.cc:388] T f6db5ed2663f4019a13adeaae12c4e44 P a27039d23a9041e4831027412c5416c9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 21:28:20.107061 31522 raft_consensus.cc:482] T f6db5ed2663f4019a13adeaae12c4e44 P a27039d23a9041e4831027412c5416c9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 21:28:20.107160 31522 raft_consensus.cc:3037] T f6db5ed2663f4019a13adeaae12c4e44 P a27039d23a9041e4831027412c5416c9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:20.109030 31522 raft_consensus.cc:504] T f6db5ed2663f4019a13adeaae12c4e44 P a27039d23a9041e4831027412c5416c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a27039d23a9041e4831027412c5416c9" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 42697 } }
I20240502 21:28:20.109262 31522 leader_election.cc:302] T f6db5ed2663f4019a13adeaae12c4e44 P a27039d23a9041e4831027412c5416c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a27039d23a9041e4831027412c5416c9; no voters:
I20240502 21:28:20.109654 31522 leader_election.cc:288] T f6db5ed2663f4019a13adeaae12c4e44 P a27039d23a9041e4831027412c5416c9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 21:28:20.110481 31522 ts_tablet_manager.cc:1398] T f6db5ed2663f4019a13adeaae12c4e44 P a27039d23a9041e4831027412c5416c9: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20240502 21:28:20.110702 31522 tablet_bootstrap.cc:492] T 87102d031b724bce8dd11d4f2a5c2e1a P a27039d23a9041e4831027412c5416c9: Bootstrap starting.
I20240502 21:28:20.109819 31530 raft_consensus.cc:2781] T f6db5ed2663f4019a13adeaae12c4e44 P a27039d23a9041e4831027412c5416c9 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:28:20.111140 31530 raft_consensus.cc:686] T f6db5ed2663f4019a13adeaae12c4e44 P a27039d23a9041e4831027412c5416c9 [term 1 LEADER]: Becoming Leader. State: Replica: a27039d23a9041e4831027412c5416c9, State: Running, Role: LEADER
I20240502 21:28:20.111445 31530 consensus_queue.cc:237] T f6db5ed2663f4019a13adeaae12c4e44 P a27039d23a9041e4831027412c5416c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a27039d23a9041e4831027412c5416c9" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 42697 } }
I20240502 21:28:20.111994 31522 tablet_bootstrap.cc:654] T 87102d031b724bce8dd11d4f2a5c2e1a P a27039d23a9041e4831027412c5416c9: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:20.113770 30779 catalog_manager.cc:5484] T f6db5ed2663f4019a13adeaae12c4e44 P a27039d23a9041e4831027412c5416c9 reported cstate change: term changed from 0 to 1, leader changed from <none> to a27039d23a9041e4831027412c5416c9 (127.15.255.130). New cstate: current_term: 1 leader_uuid: "a27039d23a9041e4831027412c5416c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a27039d23a9041e4831027412c5416c9" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 42697 } health_report { overall_health: HEALTHY } } }
I20240502 21:28:20.114670 31522 tablet_bootstrap.cc:492] T 87102d031b724bce8dd11d4f2a5c2e1a P a27039d23a9041e4831027412c5416c9: No bootstrap required, opened a new log
I20240502 21:28:20.114805 31522 ts_tablet_manager.cc:1367] T 87102d031b724bce8dd11d4f2a5c2e1a P a27039d23a9041e4831027412c5416c9: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.001s
I20240502 21:28:20.115464 31522 raft_consensus.cc:348] T 87102d031b724bce8dd11d4f2a5c2e1a P a27039d23a9041e4831027412c5416c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a27039d23a9041e4831027412c5416c9" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 42697 } }
I20240502 21:28:20.115641 31522 raft_consensus.cc:374] T 87102d031b724bce8dd11d4f2a5c2e1a P a27039d23a9041e4831027412c5416c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:20.115761 31522 raft_consensus.cc:729] T 87102d031b724bce8dd11d4f2a5c2e1a P a27039d23a9041e4831027412c5416c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a27039d23a9041e4831027412c5416c9, State: Initialized, Role: FOLLOWER
I20240502 21:28:20.116021 31522 consensus_queue.cc:260] T 87102d031b724bce8dd11d4f2a5c2e1a P a27039d23a9041e4831027412c5416c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a27039d23a9041e4831027412c5416c9" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 42697 } }
I20240502 21:28:20.116150 31522 raft_consensus.cc:388] T 87102d031b724bce8dd11d4f2a5c2e1a P a27039d23a9041e4831027412c5416c9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 21:28:20.116248 31522 raft_consensus.cc:482] T 87102d031b724bce8dd11d4f2a5c2e1a P a27039d23a9041e4831027412c5416c9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 21:28:20.116403 31522 raft_consensus.cc:3037] T 87102d031b724bce8dd11d4f2a5c2e1a P a27039d23a9041e4831027412c5416c9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:20.118451 31522 raft_consensus.cc:504] T 87102d031b724bce8dd11d4f2a5c2e1a P a27039d23a9041e4831027412c5416c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a27039d23a9041e4831027412c5416c9" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 42697 } }
I20240502 21:28:20.118628 31522 leader_election.cc:302] T 87102d031b724bce8dd11d4f2a5c2e1a P a27039d23a9041e4831027412c5416c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a27039d23a9041e4831027412c5416c9; no voters:
I20240502 21:28:20.118798 31522 leader_election.cc:288] T 87102d031b724bce8dd11d4f2a5c2e1a P a27039d23a9041e4831027412c5416c9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 21:28:20.118844 31530 raft_consensus.cc:2781] T 87102d031b724bce8dd11d4f2a5c2e1a P a27039d23a9041e4831027412c5416c9 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:28:20.119236 31522 ts_tablet_manager.cc:1398] T 87102d031b724bce8dd11d4f2a5c2e1a P a27039d23a9041e4831027412c5416c9: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240502 21:28:20.119307 31530 raft_consensus.cc:686] T 87102d031b724bce8dd11d4f2a5c2e1a P a27039d23a9041e4831027412c5416c9 [term 1 LEADER]: Becoming Leader. State: Replica: a27039d23a9041e4831027412c5416c9, State: Running, Role: LEADER
I20240502 21:28:20.119493 31530 consensus_queue.cc:237] T 87102d031b724bce8dd11d4f2a5c2e1a P a27039d23a9041e4831027412c5416c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a27039d23a9041e4831027412c5416c9" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 42697 } }
I20240502 21:28:20.121215 30779 catalog_manager.cc:5484] T 87102d031b724bce8dd11d4f2a5c2e1a P a27039d23a9041e4831027412c5416c9 reported cstate change: term changed from 0 to 1, leader changed from <none> to a27039d23a9041e4831027412c5416c9 (127.15.255.130). New cstate: current_term: 1 leader_uuid: "a27039d23a9041e4831027412c5416c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a27039d23a9041e4831027412c5416c9" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 42697 } health_report { overall_health: HEALTHY } } }
I20240502 21:28:21.426483 16382 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvNciQZ/build/asan/bin/kudu
/tmp/dist-test-taskvNciQZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.255.131:37625
--local_ip_for_outbound_sockets=127.15.255.131
--tserver_master_addrs=127.15.255.190:35921
--webserver_port=32787
--webserver_interface=127.15.255.131
--builtin_ntp_servers=127.15.255.148:38143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240502 21:28:21.742108 31561 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:28:21.742376 31561 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:28:21.742532 31561 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:28:21.749893 31561 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 21:28:21.750013 31561 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:28:21.750154 31561 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.255.131
I20240502 21:28:21.758642 31561 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.255.148:38143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.255.131:37625
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.15.255.131
--webserver_port=32787
--tserver_master_addrs=127.15.255.190:35921
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.255.131
--log_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:28:21.759756 31561 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:28:21.761387 31561 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:28:21.772300 31566 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:28:21.772836 31569 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:28:21.773348 31567 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:28:21.773898 31561 server_base.cc:1034] running on GCE node
I20240502 21:28:21.774403 31561 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:28:21.775187 31561 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:28:21.776376 31561 hybrid_clock.cc:648] HybridClock initialized: now 1714685301776330 us; error 31 us; skew 500 ppm
I20240502 21:28:21.776690 31561 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:28:21.778643 31561 webserver.cc:460] Webserver started at http://127.15.255.131:32787/ using document root <none> and password file <none>
I20240502 21:28:21.779268 31561 fs_manager.cc:359] Metadata directory not provided
I20240502 21:28:21.779402 31561 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:28:21.784224 31561 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20240502 21:28:21.787282 31575 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:28:21.788566 31561 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240502 21:28:21.788710 31561 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "bbfd9ca2d42547dca3f678bd6ed2964e"
format_stamp: "Formatted at 2024-05-02 21:28:12 on dist-test-slave-jdfx"
I20240502 21:28:21.789440 31561 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:28:21.816220 31561 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:28:21.816931 31561 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:28:21.817198 31561 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:28:21.818017 31561 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:28:21.819382 31561 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:28:21.819460 31561 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:28:21.819571 31561 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:28:21.819618 31561 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 21:28:21.847822 31561 rpc_server.cc:307] RPC server started. Bound to: 127.15.255.131:37625
I20240502 21:28:21.847864 31689 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.255.131:37625 every 8 connection(s)
I20240502 21:28:21.849195 31561 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240502 21:28:21.855614 16382 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvNciQZ/build/asan/bin/kudu as pid 31561
I20240502 21:28:21.856447 16382 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvNciQZ/build/asan/bin/kudu
/tmp/dist-test-taskvNciQZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.255.132:45045
--local_ip_for_outbound_sockets=127.15.255.132
--tserver_master_addrs=127.15.255.190:35921
--webserver_port=39927
--webserver_interface=127.15.255.132
--builtin_ntp_servers=127.15.255.148:38143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 21:28:21.861739 31690 heartbeater.cc:347] Connected to a master server at 127.15.255.190:35921
I20240502 21:28:21.861984 31690 heartbeater.cc:464] Registering TS with master...
I20240502 21:28:21.862709 31690 heartbeater.cc:511] Master 127.15.255.190:35921 requested a full tablet report, sending...
I20240502 21:28:21.864223 30779 ts_manager.cc:194] Re-registered known tserver with Master: bbfd9ca2d42547dca3f678bd6ed2964e (127.15.255.131:37625)
I20240502 21:28:21.865262 30779 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.255.131:54703
W20240502 21:28:22.181468 31694 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:28:22.181743 31694 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:28:22.181882 31694 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:28:22.189469 31694 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 21:28:22.189606 31694 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:28:22.189759 31694 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.255.132
I20240502 21:28:22.198621 31694 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.255.148:38143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.255.132:45045
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.15.255.132
--webserver_port=39927
--tserver_master_addrs=127.15.255.190:35921
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.255.132
--log_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:28:22.199848 31694 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:28:22.201617 31694 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:28:22.213079 31703 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:28:22.213575 31700 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:28:22.214067 31701 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:28:22.214689 31694 server_base.cc:1034] running on GCE node
I20240502 21:28:22.215199 31694 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:28:22.216029 31694 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:28:22.217221 31694 hybrid_clock.cc:648] HybridClock initialized: now 1714685302217170 us; error 47 us; skew 500 ppm
I20240502 21:28:22.217525 31694 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:28:22.223420 31694 webserver.cc:460] Webserver started at http://127.15.255.132:39927/ using document root <none> and password file <none>
I20240502 21:28:22.224066 31694 fs_manager.cc:359] Metadata directory not provided
I20240502 21:28:22.224200 31694 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:28:22.229265 31694 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20240502 21:28:22.231846 31709 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:28:22.233018 31694 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240502 21:28:22.233165 31694 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "17faa5ec5e6a42c9846441cf2da4eb4d"
format_stamp: "Formatted at 2024-05-02 21:28:12 on dist-test-slave-jdfx"
I20240502 21:28:22.233896 31694 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:28:22.255496 31694 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:28:22.256139 31694 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:28:22.256383 31694 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:28:22.257213 31694 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:28:22.258859 31716 ts_tablet_manager.cc:522] Loading tablet metadata (0/1 complete)
I20240502 21:28:22.263372 31694 ts_tablet_manager.cc:559] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20240502 21:28:22.263463 31694 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20240502 21:28:22.263576 31694 ts_tablet_manager.cc:574] Registering tablets (0/1 complete)
I20240502 21:28:22.266214 31694 ts_tablet_manager.cc:590] Registered 1 tablets
I20240502 21:28:22.266299 31694 ts_tablet_manager.cc:569] Time spent register tablets: real 0.003s user 0.000s sys 0.001s
I20240502 21:28:22.266607 31716 tablet_bootstrap.cc:492] T 6c5cb1507e5a470abaa5de5ec29b99d7 P 17faa5ec5e6a42c9846441cf2da4eb4d: Bootstrap starting.
I20240502 21:28:22.285293 31716 log.cc:824] T 6c5cb1507e5a470abaa5de5ec29b99d7 P 17faa5ec5e6a42c9846441cf2da4eb4d: Log is configured to *not* fsync() on all Append() calls
I20240502 21:28:22.288699 31716 tablet_bootstrap.cc:492] T 6c5cb1507e5a470abaa5de5ec29b99d7 P 17faa5ec5e6a42c9846441cf2da4eb4d: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:28:22.289435 31716 tablet_bootstrap.cc:492] T 6c5cb1507e5a470abaa5de5ec29b99d7 P 17faa5ec5e6a42c9846441cf2da4eb4d: Bootstrap complete.
I20240502 21:28:22.290796 31716 ts_tablet_manager.cc:1367] T 6c5cb1507e5a470abaa5de5ec29b99d7 P 17faa5ec5e6a42c9846441cf2da4eb4d: Time spent bootstrapping tablet: real 0.024s user 0.018s sys 0.004s
I20240502 21:28:22.300026 31716 raft_consensus.cc:348] T 6c5cb1507e5a470abaa5de5ec29b99d7 P 17faa5ec5e6a42c9846441cf2da4eb4d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17faa5ec5e6a42c9846441cf2da4eb4d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 45045 } }
I20240502 21:28:22.301223 31716 raft_consensus.cc:729] T 6c5cb1507e5a470abaa5de5ec29b99d7 P 17faa5ec5e6a42c9846441cf2da4eb4d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17faa5ec5e6a42c9846441cf2da4eb4d, State: Initialized, Role: FOLLOWER
I20240502 21:28:22.302153 31716 consensus_queue.cc:260] T 6c5cb1507e5a470abaa5de5ec29b99d7 P 17faa5ec5e6a42c9846441cf2da4eb4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17faa5ec5e6a42c9846441cf2da4eb4d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 45045 } }
I20240502 21:28:22.302537 31716 raft_consensus.cc:388] T 6c5cb1507e5a470abaa5de5ec29b99d7 P 17faa5ec5e6a42c9846441cf2da4eb4d [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 21:28:22.302735 31716 raft_consensus.cc:482] T 6c5cb1507e5a470abaa5de5ec29b99d7 P 17faa5ec5e6a42c9846441cf2da4eb4d [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 21:28:22.302974 31716 raft_consensus.cc:3037] T 6c5cb1507e5a470abaa5de5ec29b99d7 P 17faa5ec5e6a42c9846441cf2da4eb4d [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:28:22.307097 31694 rpc_server.cc:307] RPC server started. Bound to: 127.15.255.132:45045
I20240502 21:28:22.307173 31823 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.255.132:45045 every 8 connection(s)
I20240502 21:28:22.308617 31694 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240502 21:28:22.308540 31716 raft_consensus.cc:504] T 6c5cb1507e5a470abaa5de5ec29b99d7 P 17faa5ec5e6a42c9846441cf2da4eb4d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17faa5ec5e6a42c9846441cf2da4eb4d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 45045 } }
I20240502 21:28:22.309057 31716 leader_election.cc:302] T 6c5cb1507e5a470abaa5de5ec29b99d7 P 17faa5ec5e6a42c9846441cf2da4eb4d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 17faa5ec5e6a42c9846441cf2da4eb4d; no voters:
I20240502 21:28:22.310204 31716 leader_election.cc:288] T 6c5cb1507e5a470abaa5de5ec29b99d7 P 17faa5ec5e6a42c9846441cf2da4eb4d [CANDIDATE]: Term 2 election: Requested vote from peers
I20240502 21:28:22.310370 31826 raft_consensus.cc:2781] T 6c5cb1507e5a470abaa5de5ec29b99d7 P 17faa5ec5e6a42c9846441cf2da4eb4d [term 2 FOLLOWER]: Leader election won for term 2
I20240502 21:28:22.311734 31716 ts_tablet_manager.cc:1398] T 6c5cb1507e5a470abaa5de5ec29b99d7 P 17faa5ec5e6a42c9846441cf2da4eb4d: Time spent starting tablet: real 0.021s user 0.018s sys 0.004s
I20240502 21:28:22.313261 31826 raft_consensus.cc:686] T 6c5cb1507e5a470abaa5de5ec29b99d7 P 17faa5ec5e6a42c9846441cf2da4eb4d [term 2 LEADER]: Becoming Leader. State: Replica: 17faa5ec5e6a42c9846441cf2da4eb4d, State: Running, Role: LEADER
I20240502 21:28:22.313740 31826 consensus_queue.cc:237] T 6c5cb1507e5a470abaa5de5ec29b99d7 P 17faa5ec5e6a42c9846441cf2da4eb4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17faa5ec5e6a42c9846441cf2da4eb4d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 45045 } }
I20240502 21:28:22.317370 16382 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvNciQZ/build/asan/bin/kudu as pid 31694
I20240502 21:28:22.320010 31824 heartbeater.cc:347] Connected to a master server at 127.15.255.190:35921
I20240502 21:28:22.320250 31824 heartbeater.cc:464] Registering TS with master...
I20240502 21:28:22.320986 31824 heartbeater.cc:511] Master 127.15.255.190:35921 requested a full tablet report, sending...
I20240502 21:28:22.323180 30779 ts_manager.cc:194] Re-registered known tserver with Master: 17faa5ec5e6a42c9846441cf2da4eb4d (127.15.255.132:45045)
I20240502 21:28:22.323575 30779 catalog_manager.cc:5484] T 6c5cb1507e5a470abaa5de5ec29b99d7 P 17faa5ec5e6a42c9846441cf2da4eb4d reported cstate change: term changed from 1 to 2. New cstate: current_term: 2 leader_uuid: "17faa5ec5e6a42c9846441cf2da4eb4d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17faa5ec5e6a42c9846441cf2da4eb4d" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 45045 } health_report { overall_health: HEALTHY } } }
I20240502 21:28:22.328173 30779 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.255.132:39889
I20240502 21:28:22.330263 31824 heartbeater.cc:503] Master 127.15.255.190:35921 was elected leader, sending a full tablet report...
I20240502 21:28:22.759274 30903 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:28:22.795794 31035 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:28:22.799446 31758 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:28:22.803694 31623 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:28:22.867406 31690 heartbeater.cc:503] Master 127.15.255.190:35921 was elected leader, sending a full tablet report...
I20240502 21:28:22.974195 31758 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:28:22.978518 30903 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:28:22.980441 31623 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:28:22.980651 31035 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:28:23.230773 31758 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:28:23.232764 30903 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:28:23.238039 31035 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 21:28:23.239500 31623 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
d381c63cae664ab9942526dbc30cfefa | 127.15.255.190:35921 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714685120732928-16382-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.15.255.148:38143 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+---------+----------+----------------+-----------------
17faa5ec5e6a42c9846441cf2da4eb4d | 127.15.255.132:45045 | HEALTHY | <none> | 1 | 0
2bff96c137244df6b6beef103aecd997 | 127.15.255.129:44455 | HEALTHY | <none> | 6 | 0
a27039d23a9041e4831027412c5416c9 | 127.15.255.130:42697 | HEALTHY | <none> | 6 | 0
bbfd9ca2d42547dca3f678bd6ed2964e | 127.15.255.131:37625 | HEALTHY | <none> | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 4
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 4 server(s) checked
local_ip_for_outbound_sockets | 127.15.255.129 | experimental | 127.15.255.129:44455
local_ip_for_outbound_sockets | 127.15.255.130 | experimental | 127.15.255.130:42697
local_ip_for_outbound_sockets | 127.15.255.131 | experimental | 127.15.255.131:37625
local_ip_for_outbound_sockets | 127.15.255.132 | experimental | 127.15.255.132:45045
never_fsync | true | unsafe,advanced | all 4 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 4 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 4 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 4 server(s) checked
server_dump_info_format | pb | hidden | all 4 server(s) checked
server_dump_info_path | /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.15.255.129:44455
server_dump_info_path | /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.15.255.130:42697
server_dump_info_path | /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.15.255.131:37625
server_dump_info_path | /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove_5.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.15.255.132:45045
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.15.255.148:38143 | all 4 server(s) checked
time_source | builtin | all 4 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 5 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------------------+----+---------+---------------+---------+------------+------------------+-------------
TestTable | 1 | HEALTHY | 1 | 1 | 0 | 0 | 0
rebalance_test_table_0 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_1 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_2 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_3 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 1
Median | 6
Third Quartile | 6
Maximum | 6
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 4
Tables | 5
Tablets | 13
Replicas | 13
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20240502 21:28:23.287897 16382 log_verifier.cc:126] Checking tablet 19573c9e33484d14b7f5b3b866597593
I20240502 21:28:23.295320 16382 log_verifier.cc:177] Verified matching terms for 1 ops in tablet 19573c9e33484d14b7f5b3b866597593
I20240502 21:28:23.295389 16382 log_verifier.cc:126] Checking tablet 21b9b0408e6f4ffaafaebec4b02c3c83
I20240502 21:28:23.302546 16382 log_verifier.cc:177] Verified matching terms for 1 ops in tablet 21b9b0408e6f4ffaafaebec4b02c3c83
I20240502 21:28:23.302615 16382 log_verifier.cc:126] Checking tablet 493201e6661b4ba1b3c50bab507496a1
I20240502 21:28:23.310837 16382 log_verifier.cc:177] Verified matching terms for 1 ops in tablet 493201e6661b4ba1b3c50bab507496a1
I20240502 21:28:23.310921 16382 log_verifier.cc:126] Checking tablet 6566df65606643b584597fa2fc5a307d
I20240502 21:28:23.320636 16382 log_verifier.cc:177] Verified matching terms for 1 ops in tablet 6566df65606643b584597fa2fc5a307d
I20240502 21:28:23.320713 16382 log_verifier.cc:126] Checking tablet 6c5cb1507e5a470abaa5de5ec29b99d7
I20240502 21:28:23.330157 16382 log_verifier.cc:177] Verified matching terms for 2 ops in tablet 6c5cb1507e5a470abaa5de5ec29b99d7
I20240502 21:28:23.330227 16382 log_verifier.cc:126] Checking tablet 87102d031b724bce8dd11d4f2a5c2e1a
I20240502 21:28:23.337352 16382 log_verifier.cc:177] Verified matching terms for 1 ops in tablet 87102d031b724bce8dd11d4f2a5c2e1a
I20240502 21:28:23.337417 16382 log_verifier.cc:126] Checking tablet 886418f21f1e4eef95d1a0cd69417204
I20240502 21:28:23.344576 16382 log_verifier.cc:177] Verified matching terms for 1 ops in tablet 886418f21f1e4eef95d1a0cd69417204
I20240502 21:28:23.344656 16382 log_verifier.cc:126] Checking tablet 9c53de279b95409ca718939cfd90165b
I20240502 21:28:23.352041 16382 log_verifier.cc:177] Verified matching terms for 2 ops in tablet 9c53de279b95409ca718939cfd90165b
I20240502 21:28:23.352105 16382 log_verifier.cc:126] Checking tablet 9fbb7fe776a541cfb3a383b5e7ab611e
I20240502 21:28:23.359225 16382 log_verifier.cc:177] Verified matching terms for 2 ops in tablet 9fbb7fe776a541cfb3a383b5e7ab611e
I20240502 21:28:23.359287 16382 log_verifier.cc:126] Checking tablet ad16c12ce187416caabf8074b9499f3b
I20240502 21:28:23.366514 16382 log_verifier.cc:177] Verified matching terms for 2 ops in tablet ad16c12ce187416caabf8074b9499f3b
I20240502 21:28:23.366576 16382 log_verifier.cc:126] Checking tablet b71736f0c4c245a29cca4198dc0dfbf8
I20240502 21:28:23.373466 16382 log_verifier.cc:177] Verified matching terms for 1 ops in tablet b71736f0c4c245a29cca4198dc0dfbf8
I20240502 21:28:23.373526 16382 log_verifier.cc:126] Checking tablet c450518f0a1b4c6889538b31d86e7e29
I20240502 21:28:23.380378 16382 log_verifier.cc:177] Verified matching terms for 1 ops in tablet c450518f0a1b4c6889538b31d86e7e29
I20240502 21:28:23.380470 16382 log_verifier.cc:126] Checking tablet f6db5ed2663f4019a13adeaae12c4e44
I20240502 21:28:23.387735 16382 log_verifier.cc:177] Verified matching terms for 2 ops in tablet f6db5ed2663f4019a13adeaae12c4e44
I20240502 21:28:23.388749 16382 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvNciQZ/build/asan/bin/kudu pid 30841
I20240502 21:28:23.389925 30962 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 21:28:23.482812 16382 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvNciQZ/build/asan/bin/kudu with pid 30841
I20240502 21:28:23.499902 16382 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvNciQZ/build/asan/bin/kudu pid 30972
I20240502 21:28:23.500902 31095 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 21:28:23.599306 16382 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvNciQZ/build/asan/bin/kudu with pid 30972
I20240502 21:28:23.616056 16382 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvNciQZ/build/asan/bin/kudu pid 31561
I20240502 21:28:23.622182 31685 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 21:28:23.705202 16382 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvNciQZ/build/asan/bin/kudu with pid 31561
I20240502 21:28:23.721946 16382 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvNciQZ/build/asan/bin/kudu pid 31694
I20240502 21:28:23.727536 31819 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 21:28:23.814384 16382 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvNciQZ/build/asan/bin/kudu with pid 31694
I20240502 21:28:23.830988 16382 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvNciQZ/build/asan/bin/kudu pid 30750
I20240502 21:28:23.832021 30811 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 21:28:23.945441 16382 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvNciQZ/build/asan/bin/kudu with pid 30750
2024-05-02T21:28:23Z chronyd exiting
[ OK ] RebalancerAndSingleReplicaTablets.SingleReplicasStayOrMove/5 (13662 ms)
[----------] 1 test from RebalancerAndSingleReplicaTablets (13662 ms total)
[----------] 2 tests from LocationAwareRebalancingParamTest
[ RUN ] LocationAwareRebalancingParamTest.Rebalance/idx7_rf2_t3_l_1_2_1
I20240502 21:28:24.005790 16382 test_util.cc:268] Using random seed: 1215456378
I20240502 21:28:24.007820 16382 ts_itest-base.cc:112] Starting cluster with:
I20240502 21:28:24.007898 16382 ts_itest-base.cc:113] --------------
I20240502 21:28:24.007962 16382 ts_itest-base.cc:114] 4 tablet servers
I20240502 21:28:24.008015 16382 ts_itest-base.cc:115] 2 replicas per TS
I20240502 21:28:24.008076 16382 ts_itest-base.cc:116] --------------
2024-05-02T21:28:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-02T21:28:24Z Disabled control of system clock
I20240502 21:28:24.061350 16382 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvNciQZ/build/asan/bin/kudu
/tmp/dist-test-taskvNciQZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714685120732928-16382-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714685120732928-16382-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714685120732928-16382-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714685120732928-16382-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.255.190:34507
--webserver_interface=127.15.255.190
--webserver_port=0
--builtin_ntp_servers=127.15.255.148:45063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.255.190:34507
--location_mapping_cmd=/tmp/dist-test-taskvNciQZ/build/asan/bin/testdata/assign-location.py --state_store=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714685120732928-16382-0/raft_consensus-itest-cluster/location-assignment.state --map /A:1 --map /B:2 --map /C:1
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=true
--enable_range_replica_placement=false
--master_client_location_assignment_enabled=false
--allow_unsafe_replication_factor with env {}
W20240502 21:28:24.387540 31950 flags.cc:425] Enabled unsafe flag: --allow_unsafe_replication_factor=true
W20240502 21:28:24.387847 31950 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:28:24.387933 31950 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:28:24.388010 31950 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:28:24.395551 31950 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 21:28:24.395658 31950 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240502 21:28:24.395733 31950 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:28:24.395785 31950 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240502 21:28:24.395849 31950 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240502 21:28:24.404784 31950 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.255.148:45063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714685120732928-16382-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714685120732928-16382-0/raft_consensus-itest-cluster/master-0/wal
--allow_unsafe_replication_factor=true
--enable_range_replica_placement=false
--location_mapping_cmd=/tmp/dist-test-taskvNciQZ/build/asan/bin/testdata/assign-location.py --state_store=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714685120732928-16382-0/raft_consensus-itest-cluster/location-assignment.state --map /A:1 --map /B:2 --map /C:1
--ipki_ca_key_size=768
--master_addresses=127.15.255.190:34507
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.255.190:34507
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714685120732928-16382-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.15.255.190
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714685120732928-16382-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:28:24.405951 31950 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:28:24.407714 31950 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:28:24.418256 31955 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:28:24.418747 31956 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:28:24.419375 31958 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:28:24.419965 31950 server_base.cc:1034] running on GCE node
I20240502 21:28:24.420667 31950 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:28:24.421880 31950 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:28:24.423084 31950 hybrid_clock.cc:648] HybridClock initialized: now 1714685304423025 us; error 41 us; skew 500 ppm
I20240502 21:28:24.423372 31950 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:28:24.425468 31950 webserver.cc:460] Webserver started at http://127.15.255.190:45089/ using document root <none> and password file <none>
I20240502 21:28:24.426100 31950 fs_manager.cc:359] Metadata directory not provided
I20240502 21:28:24.426229 31950 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:28:24.426501 31950 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:28:24.428925 31950 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714685120732928-16382-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "f08edf5b9b4548e8be1d8e3471cb83a5"
format_stamp: "Formatted at 2024-05-02 21:28:24 on dist-test-slave-jdfx"
I20240502 21:28:24.429637 31950 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714685120732928-16382-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "f08edf5b9b4548e8be1d8e3471cb83a5"
format_stamp: "Formatted at 2024-05-02 21:28:24 on dist-test-slave-jdfx"
I20240502 21:28:24.434286 31950 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20240502 21:28:24.437372 31964 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:28:24.438529 31950 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240502 21:28:24.438678 31950 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714685120732928-16382-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714685120732928-16382-0/raft_consensus-itest-cluster/master-0/wal
uuid: "f08edf5b9b4548e8be1d8e3471cb83a5"
format_stamp: "Formatted at 2024-05-02 21:28:24 on dist-test-slave-jdfx"
I20240502 21:28:24.438835 31950 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714685120732928-16382-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714685120732928-16382-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714685120732928-16382-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:28:24.467352 31950 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:28:24.468076 31950 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:28:24.468340 31950 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:28:24.484644 31950 rpc_server.cc:307] RPC server started. Bound to: 127.15.255.190:34507
I20240502 21:28:24.484683 32017 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.255.190:34507 every 8 connection(s)
I20240502 21:28:24.485908 31950 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714685120732928-16382-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240502 21:28:24.489248 32018 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:24.495738 16382 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvNciQZ/build/asan/bin/kudu as pid 31950
I20240502 21:28:24.495951 16382 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714685120732928-16382-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240502 21:28:24.498090 32018 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f08edf5b9b4548e8be1d8e3471cb83a5: Bootstrap starting.
I20240502 21:28:24.501175 32018 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f08edf5b9b4548e8be1d8e3471cb83a5: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:24.502147 32018 log.cc:824] T 00000000000000000000000000000000 P f08edf5b9b4548e8be1d8e3471cb83a5: Log is configured to *not* fsync() on all Append() calls
I20240502 21:28:24.504451 32018 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f08edf5b9b4548e8be1d8e3471cb83a5: No bootstrap required, opened a new log
I20240502 21:28:24.511364 32018 raft_consensus.cc:348] T 00000000000000000000000000000000 P f08edf5b9b4548e8be1d8e3471cb83a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f08edf5b9b4548e8be1d8e3471cb83a5" member_type: VOTER last_known_addr { host: "127.15.255.190" port: 34507 } }
I20240502 21:28:24.511718 32018 raft_consensus.cc:374] T 00000000000000000000000000000000 P f08edf5b9b4548e8be1d8e3471cb83a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:24.511807 32018 raft_consensus.cc:729] T 00000000000000000000000000000000 P f08edf5b9b4548e8be1d8e3471cb83a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f08edf5b9b4548e8be1d8e3471cb83a5, State: Initialized, Role: FOLLOWER
I20240502 21:28:24.512360 32018 consensus_queue.cc:260] T 00000000000000000000000000000000 P f08edf5b9b4548e8be1d8e3471cb83a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f08edf5b9b4548e8be1d8e3471cb83a5" member_type: VOTER last_known_addr { host: "127.15.255.190" port: 34507 } }
I20240502 21:28:24.512596 32018 raft_consensus.cc:388] T 00000000000000000000000000000000 P f08edf5b9b4548e8be1d8e3471cb83a5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 21:28:24.512712 32018 raft_consensus.cc:482] T 00000000000000000000000000000000 P f08edf5b9b4548e8be1d8e3471cb83a5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 21:28:24.512847 32018 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f08edf5b9b4548e8be1d8e3471cb83a5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:24.514730 32018 raft_consensus.cc:504] T 00000000000000000000000000000000 P f08edf5b9b4548e8be1d8e3471cb83a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f08edf5b9b4548e8be1d8e3471cb83a5" member_type: VOTER last_known_addr { host: "127.15.255.190" port: 34507 } }
I20240502 21:28:24.515142 32018 leader_election.cc:302] T 00000000000000000000000000000000 P f08edf5b9b4548e8be1d8e3471cb83a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f08edf5b9b4548e8be1d8e3471cb83a5; no voters:
I20240502 21:28:24.515655 32018 leader_election.cc:288] T 00000000000000000000000000000000 P f08edf5b9b4548e8be1d8e3471cb83a5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 21:28:24.515758 32023 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f08edf5b9b4548e8be1d8e3471cb83a5 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:28:24.516322 32023 raft_consensus.cc:686] T 00000000000000000000000000000000 P f08edf5b9b4548e8be1d8e3471cb83a5 [term 1 LEADER]: Becoming Leader. State: Replica: f08edf5b9b4548e8be1d8e3471cb83a5, State: Running, Role: LEADER
I20240502 21:28:24.516772 32023 consensus_queue.cc:237] T 00000000000000000000000000000000 P f08edf5b9b4548e8be1d8e3471cb83a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f08edf5b9b4548e8be1d8e3471cb83a5" member_type: VOTER last_known_addr { host: "127.15.255.190" port: 34507 } }
I20240502 21:28:24.517318 32018 sys_catalog.cc:564] T 00000000000000000000000000000000 P f08edf5b9b4548e8be1d8e3471cb83a5 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 21:28:24.519344 32024 sys_catalog.cc:455] T 00000000000000000000000000000000 P f08edf5b9b4548e8be1d8e3471cb83a5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f08edf5b9b4548e8be1d8e3471cb83a5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f08edf5b9b4548e8be1d8e3471cb83a5" member_type: VOTER last_known_addr { host: "127.15.255.190" port: 34507 } } }
I20240502 21:28:24.519495 32025 sys_catalog.cc:455] T 00000000000000000000000000000000 P f08edf5b9b4548e8be1d8e3471cb83a5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f08edf5b9b4548e8be1d8e3471cb83a5. Latest consensus state: current_term: 1 leader_uuid: "f08edf5b9b4548e8be1d8e3471cb83a5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f08edf5b9b4548e8be1d8e3471cb83a5" member_type: VOTER last_known_addr { host: "127.15.255.190" port: 34507 } } }
I20240502 21:28:24.519685 32024 sys_catalog.cc:458] T 00000000000000000000000000000000 P f08edf5b9b4548e8be1d8e3471cb83a5 [sys.catalog]: This master's current role is: LEADER
I20240502 21:28:24.519742 32025 sys_catalog.cc:458] T 00000000000000000000000000000000 P f08edf5b9b4548e8be1d8e3471cb83a5 [sys.catalog]: This master's current role is: LEADER
I20240502 21:28:24.521503 32029 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 21:28:24.525779 32029 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 21:28:24.533124 32029 catalog_manager.cc:1347] Generated new cluster ID: 183d066f18cf45e0885b523e06fd03b3
I20240502 21:28:24.533232 32029 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 21:28:24.549757 32029 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 21:28:24.550815 32029 catalog_manager.cc:1504] Loading token signing keys...
I20240502 21:28:24.559662 32029 catalog_manager.cc:5857] T 00000000000000000000000000000000 P f08edf5b9b4548e8be1d8e3471cb83a5: Generated new TSK 0
I20240502 21:28:24.560319 32029 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 21:28:24.574083 16382 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvNciQZ/build/asan/bin/kudu
/tmp/dist-test-taskvNciQZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.255.129:0
--local_ip_for_outbound_sockets=127.15.255.129
--webserver_interface=127.15.255.129
--webserver_port=0
--tserver_master_addrs=127.15.255.190:34507
--builtin_ntp_servers=127.15.255.148:45063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240502 21:28:24.896668 32042 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:28:24.896916 32042 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:28:24.897042 32042 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:28:24.904147 32042 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 21:28:24.904264 32042 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:28:24.904410 32042 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.255.129
I20240502 21:28:24.913008 32042 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.255.148:45063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.255.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.15.255.129
--webserver_port=0
--tserver_master_addrs=127.15.255.190:34507
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.255.129
--log_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:28:24.914127 32042 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:28:24.915724 32042 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:28:24.927183 32048 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:28:24.927568 32047 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:28:24.928246 32050 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:28:24.928905 32042 server_base.cc:1034] running on GCE node
I20240502 21:28:24.929420 32042 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:28:24.930238 32042 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:28:24.931437 32042 hybrid_clock.cc:648] HybridClock initialized: now 1714685304931390 us; error 37 us; skew 500 ppm
I20240502 21:28:24.931728 32042 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:28:24.933753 32042 webserver.cc:460] Webserver started at http://127.15.255.129:34089/ using document root <none> and password file <none>
I20240502 21:28:24.934418 32042 fs_manager.cc:359] Metadata directory not provided
I20240502 21:28:24.934525 32042 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:28:24.934803 32042 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:28:24.937271 32042 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "67a81d55b1bd4d2e9b587c6477d624a2"
format_stamp: "Formatted at 2024-05-02 21:28:24 on dist-test-slave-jdfx"
I20240502 21:28:24.937985 32042 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "67a81d55b1bd4d2e9b587c6477d624a2"
format_stamp: "Formatted at 2024-05-02 21:28:24 on dist-test-slave-jdfx"
I20240502 21:28:24.942620 32042 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20240502 21:28:24.945686 32056 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:28:24.946822 32042 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240502 21:28:24.946977 32042 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "67a81d55b1bd4d2e9b587c6477d624a2"
format_stamp: "Formatted at 2024-05-02 21:28:24 on dist-test-slave-jdfx"
I20240502 21:28:24.947170 32042 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:28:24.970003 32042 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:28:24.970742 32042 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:28:24.970993 32042 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:28:24.971848 32042 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:28:24.973253 32042 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:28:24.973345 32042 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:28:24.973448 32042 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:28:24.973516 32042 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 21:28:25.002205 32042 rpc_server.cc:307] RPC server started. Bound to: 127.15.255.129:40197
I20240502 21:28:25.002264 32170 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.255.129:40197 every 8 connection(s)
I20240502 21:28:25.003407 32042 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240502 21:28:25.003902 16382 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvNciQZ/build/asan/bin/kudu as pid 32042
I20240502 21:28:25.004148 16382 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240502 21:28:25.007952 16382 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvNciQZ/build/asan/bin/kudu
/tmp/dist-test-taskvNciQZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.255.130:0
--local_ip_for_outbound_sockets=127.15.255.130
--webserver_interface=127.15.255.130
--webserver_port=0
--tserver_master_addrs=127.15.255.190:34507
--builtin_ntp_servers=127.15.255.148:45063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 21:28:25.016363 32171 heartbeater.cc:347] Connected to a master server at 127.15.255.190:34507
I20240502 21:28:25.016652 32171 heartbeater.cc:464] Registering TS with master...
I20240502 21:28:25.017475 32171 heartbeater.cc:511] Master 127.15.255.190:34507 requested a full tablet report, sending...
I20240502 21:28:25.062840 31982 ts_manager.cc:194] Registered new tserver with Master: 67a81d55b1bd4d2e9b587c6477d624a2 (127.15.255.129:40197)
I20240502 21:28:25.064867 31982 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.255.129:35377
W20240502 21:28:25.336602 32175 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:28:25.336885 32175 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:28:25.337030 32175 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:28:25.344560 32175 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 21:28:25.344698 32175 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:28:25.344854 32175 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.255.130
I20240502 21:28:25.353821 32175 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.255.148:45063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.255.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.15.255.130
--webserver_port=0
--tserver_master_addrs=127.15.255.190:34507
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.255.130
--log_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:28:25.355046 32175 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:28:25.356766 32175 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:28:25.368088 32181 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:28:25.368670 32182 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:28:25.369174 32184 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:28:25.369670 32175 server_base.cc:1034] running on GCE node
I20240502 21:28:25.370182 32175 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:28:25.371001 32175 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:28:25.372193 32175 hybrid_clock.cc:648] HybridClock initialized: now 1714685305372154 us; error 29 us; skew 500 ppm
I20240502 21:28:25.372514 32175 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:28:25.374516 32175 webserver.cc:460] Webserver started at http://127.15.255.130:42967/ using document root <none> and password file <none>
I20240502 21:28:25.375162 32175 fs_manager.cc:359] Metadata directory not provided
I20240502 21:28:25.375267 32175 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:28:25.375545 32175 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:28:25.377919 32175 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "c9ab11aee0734f048a27a267e9a34ee4"
format_stamp: "Formatted at 2024-05-02 21:28:25 on dist-test-slave-jdfx"
I20240502 21:28:25.378608 32175 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "c9ab11aee0734f048a27a267e9a34ee4"
format_stamp: "Formatted at 2024-05-02 21:28:25 on dist-test-slave-jdfx"
I20240502 21:28:25.383090 32175 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20240502 21:28:25.386032 32190 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:28:25.387127 32175 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240502 21:28:25.387277 32175 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "c9ab11aee0734f048a27a267e9a34ee4"
format_stamp: "Formatted at 2024-05-02 21:28:25 on dist-test-slave-jdfx"
I20240502 21:28:25.387451 32175 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:28:25.401644 32175 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:28:25.402326 32175 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:28:25.402571 32175 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:28:25.403385 32175 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:28:25.404737 32175 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:28:25.404831 32175 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:28:25.404929 32175 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:28:25.404996 32175 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 21:28:25.433233 32175 rpc_server.cc:307] RPC server started. Bound to: 127.15.255.130:34399
I20240502 21:28:25.433288 32303 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.255.130:34399 every 8 connection(s)
I20240502 21:28:25.434377 32175 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240502 21:28:25.442899 16382 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvNciQZ/build/asan/bin/kudu as pid 32175
I20240502 21:28:25.443102 16382 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240502 21:28:25.445865 32304 heartbeater.cc:347] Connected to a master server at 127.15.255.190:34507
I20240502 21:28:25.446161 32304 heartbeater.cc:464] Registering TS with master...
I20240502 21:28:25.446635 16382 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvNciQZ/build/asan/bin/kudu
/tmp/dist-test-taskvNciQZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.255.131:0
--local_ip_for_outbound_sockets=127.15.255.131
--webserver_interface=127.15.255.131
--webserver_port=0
--tserver_master_addrs=127.15.255.190:34507
--builtin_ntp_servers=127.15.255.148:45063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 21:28:25.447068 32304 heartbeater.cc:511] Master 127.15.255.190:34507 requested a full tablet report, sending...
I20240502 21:28:25.496672 31981 ts_manager.cc:194] Registered new tserver with Master: c9ab11aee0734f048a27a267e9a34ee4 (127.15.255.130:34399)
I20240502 21:28:25.498039 31981 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.255.130:34101
W20240502 21:28:25.765769 32308 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:28:25.766009 32308 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:28:25.766132 32308 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:28:25.773306 32308 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 21:28:25.773427 32308 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:28:25.773572 32308 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.255.131
I20240502 21:28:25.782090 32308 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.255.148:45063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.255.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.15.255.131
--webserver_port=0
--tserver_master_addrs=127.15.255.190:34507
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.255.131
--log_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:28:25.783238 32308 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:28:25.784843 32308 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:28:25.796114 32313 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:28:25.796619 32316 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:28:25.797196 32314 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:28:25.797756 32308 server_base.cc:1034] running on GCE node
I20240502 21:28:25.798251 32308 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:28:25.799077 32308 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:28:25.800271 32308 hybrid_clock.cc:648] HybridClock initialized: now 1714685305800223 us; error 38 us; skew 500 ppm
I20240502 21:28:25.800591 32308 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:28:25.802642 32308 webserver.cc:460] Webserver started at http://127.15.255.131:42885/ using document root <none> and password file <none>
I20240502 21:28:25.803308 32308 fs_manager.cc:359] Metadata directory not provided
I20240502 21:28:25.803416 32308 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:28:25.803692 32308 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:28:25.806210 32308 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "c38cec78fdf74d05b3c75f21eeb8612c"
format_stamp: "Formatted at 2024-05-02 21:28:25 on dist-test-slave-jdfx"
I20240502 21:28:25.806896 32308 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "c38cec78fdf74d05b3c75f21eeb8612c"
format_stamp: "Formatted at 2024-05-02 21:28:25 on dist-test-slave-jdfx"
I20240502 21:28:25.811511 32308 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20240502 21:28:25.814560 32322 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:28:25.815742 32308 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20240502 21:28:25.815882 32308 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "c38cec78fdf74d05b3c75f21eeb8612c"
format_stamp: "Formatted at 2024-05-02 21:28:25 on dist-test-slave-jdfx"
I20240502 21:28:25.816061 32308 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:28:25.844185 32308 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:28:25.844931 32308 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:28:25.845188 32308 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:28:25.846057 32308 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:28:25.847476 32308 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:28:25.847566 32308 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:28:25.847669 32308 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:28:25.847738 32308 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 21:28:25.877034 32308 rpc_server.cc:307] RPC server started. Bound to: 127.15.255.131:45205
I20240502 21:28:25.877128 32435 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.255.131:45205 every 8 connection(s)
I20240502 21:28:25.878206 32308 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240502 21:28:25.887298 16382 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvNciQZ/build/asan/bin/kudu as pid 32308
I20240502 21:28:25.887511 16382 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240502 21:28:25.890240 32436 heartbeater.cc:347] Connected to a master server at 127.15.255.190:34507
I20240502 21:28:25.890508 32436 heartbeater.cc:464] Registering TS with master...
I20240502 21:28:25.891242 32436 heartbeater.cc:511] Master 127.15.255.190:34507 requested a full tablet report, sending...
I20240502 21:28:25.891245 16382 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvNciQZ/build/asan/bin/kudu
/tmp/dist-test-taskvNciQZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.255.132:0
--local_ip_for_outbound_sockets=127.15.255.132
--webserver_interface=127.15.255.132
--webserver_port=0
--tserver_master_addrs=127.15.255.190:34507
--builtin_ntp_servers=127.15.255.148:45063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 21:28:25.934692 31981 ts_manager.cc:194] Registered new tserver with Master: c38cec78fdf74d05b3c75f21eeb8612c (127.15.255.131:45205)
I20240502 21:28:25.935973 31981 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.255.131:47891
I20240502 21:28:26.068485 32171 heartbeater.cc:503] Master 127.15.255.190:34507 was elected leader, sending a full tablet report...
W20240502 21:28:26.217568 32440 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:28:26.217805 32440 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:28:26.217892 32440 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:28:26.225138 32440 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 21:28:26.225247 32440 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:28:26.225391 32440 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.255.132
I20240502 21:28:26.234113 32440 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.255.148:45063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.255.132:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.15.255.132
--webserver_port=0
--tserver_master_addrs=127.15.255.190:34507
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.255.132
--log_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:28:26.235215 32440 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:28:26.236830 32440 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:28:26.248090 32445 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:28:26.248522 32448 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:28:26.249209 32446 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:28:26.249820 32440 server_base.cc:1034] running on GCE node
I20240502 21:28:26.250321 32440 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:28:26.251179 32440 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:28:26.252364 32440 hybrid_clock.cc:648] HybridClock initialized: now 1714685306252330 us; error 32 us; skew 500 ppm
I20240502 21:28:26.252665 32440 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:28:26.254590 32440 webserver.cc:460] Webserver started at http://127.15.255.132:40661/ using document root <none> and password file <none>
I20240502 21:28:26.255198 32440 fs_manager.cc:359] Metadata directory not provided
I20240502 21:28:26.255286 32440 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:28:26.255570 32440 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:28:26.257930 32440 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "e0a4e8376dae403e9b812a429b8960af"
format_stamp: "Formatted at 2024-05-02 21:28:26 on dist-test-slave-jdfx"
I20240502 21:28:26.258595 32440 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "e0a4e8376dae403e9b812a429b8960af"
format_stamp: "Formatted at 2024-05-02 21:28:26 on dist-test-slave-jdfx"
I20240502 21:28:26.263142 32440 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20240502 21:28:26.266115 32454 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:28:26.267231 32440 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240502 21:28:26.267364 32440 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "e0a4e8376dae403e9b812a429b8960af"
format_stamp: "Formatted at 2024-05-02 21:28:26 on dist-test-slave-jdfx"
I20240502 21:28:26.267532 32440 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:28:26.286969 32440 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:28:26.287703 32440 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:28:26.287978 32440 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:28:26.288945 32440 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:28:26.290252 32440 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:28:26.290336 32440 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:28:26.290436 32440 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:28:26.290498 32440 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 21:28:26.319660 32440 rpc_server.cc:307] RPC server started. Bound to: 127.15.255.132:42105
I20240502 21:28:26.319736 32567 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.255.132:42105 every 8 connection(s)
I20240502 21:28:26.320858 32440 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240502 21:28:26.321854 16382 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvNciQZ/build/asan/bin/kudu as pid 32440
I20240502 21:28:26.322031 16382 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240502 21:28:26.332818 32568 heartbeater.cc:347] Connected to a master server at 127.15.255.190:34507
I20240502 21:28:26.333091 32568 heartbeater.cc:464] Registering TS with master...
I20240502 21:28:26.333803 32568 heartbeater.cc:511] Master 127.15.255.190:34507 requested a full tablet report, sending...
I20240502 21:28:26.374994 31981 ts_manager.cc:194] Registered new tserver with Master: e0a4e8376dae403e9b812a429b8960af (127.15.255.132:42105)
I20240502 21:28:26.375972 31981 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.255.132:60755
I20240502 21:28:26.378338 16382 external_mini_cluster.cc:930] 4 TS(s) registered with all masters
I20240502 21:28:26.393678 16382 external_mini_cluster.cc:930] 4 TS(s) registered with all masters
I20240502 21:28:26.393839 16382 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvNciQZ/build/asan/bin/kudu pid 32175
I20240502 21:28:26.401155 32299 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 21:28:26.479620 16382 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvNciQZ/build/asan/bin/kudu with pid 32175
I20240502 21:28:26.498265 16382 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvNciQZ/build/asan/bin/kudu pid 32308
I20240502 21:28:26.506145 32431 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 21:28:26.583705 16382 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvNciQZ/build/asan/bin/kudu with pid 32308
I20240502 21:28:27.378736 32568 heartbeater.cc:503] Master 127.15.255.190:34507 was elected leader, sending a full tablet report...
I20240502 21:28:30.350519 31981 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49812:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
W20240502 21:28:30.357214 31981 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table rebalance_test_table_0 in case of a server failure: 3 tablet servers would be needed, 2 are available. Consider bringing up more tablet servers.
I20240502 21:28:30.383738 32103 tablet_service.cc:1467] Processing CreateTablet for tablet d7c55e3c68744428a3bc7f2a1fe50c1f (DEFAULT_TABLE table=rebalance_test_table_0 [id=7b825a608b1348f09ea4d42250b107e2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:30.383684 32104 tablet_service.cc:1467] Processing CreateTablet for tablet 1707bbe6fcd44b09ab452215e3997199 (DEFAULT_TABLE table=rebalance_test_table_0 [id=7b825a608b1348f09ea4d42250b107e2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:30.384187 32105 tablet_service.cc:1467] Processing CreateTablet for tablet 2531188c066d4c33976b53333caed52c (DEFAULT_TABLE table=rebalance_test_table_0 [id=7b825a608b1348f09ea4d42250b107e2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:30.385447 32103 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d7c55e3c68744428a3bc7f2a1fe50c1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:30.385033 32503 tablet_service.cc:1467] Processing CreateTablet for tablet 2531188c066d4c33976b53333caed52c (DEFAULT_TABLE table=rebalance_test_table_0 [id=7b825a608b1348f09ea4d42250b107e2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:30.386183 32105 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2531188c066d4c33976b53333caed52c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:30.386770 32503 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2531188c066d4c33976b53333caed52c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:30.387437 32104 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1707bbe6fcd44b09ab452215e3997199. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:30.385670 32501 tablet_service.cc:1467] Processing CreateTablet for tablet d7c55e3c68744428a3bc7f2a1fe50c1f (DEFAULT_TABLE table=rebalance_test_table_0 [id=7b825a608b1348f09ea4d42250b107e2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:30.385859 32502 tablet_service.cc:1467] Processing CreateTablet for tablet 1707bbe6fcd44b09ab452215e3997199 (DEFAULT_TABLE table=rebalance_test_table_0 [id=7b825a608b1348f09ea4d42250b107e2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:30.388341 32501 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d7c55e3c68744428a3bc7f2a1fe50c1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:30.388710 32502 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1707bbe6fcd44b09ab452215e3997199. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:30.403574 32590 tablet_bootstrap.cc:492] T 1707bbe6fcd44b09ab452215e3997199 P 67a81d55b1bd4d2e9b587c6477d624a2: Bootstrap starting.
I20240502 21:28:30.407194 32590 tablet_bootstrap.cc:654] T 1707bbe6fcd44b09ab452215e3997199 P 67a81d55b1bd4d2e9b587c6477d624a2: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:30.407773 32591 tablet_bootstrap.cc:492] T 2531188c066d4c33976b53333caed52c P e0a4e8376dae403e9b812a429b8960af: Bootstrap starting.
I20240502 21:28:30.408604 32590 log.cc:824] T 1707bbe6fcd44b09ab452215e3997199 P 67a81d55b1bd4d2e9b587c6477d624a2: Log is configured to *not* fsync() on all Append() calls
I20240502 21:28:30.411159 32591 tablet_bootstrap.cc:654] T 2531188c066d4c33976b53333caed52c P e0a4e8376dae403e9b812a429b8960af: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:30.411389 32590 tablet_bootstrap.cc:492] T 1707bbe6fcd44b09ab452215e3997199 P 67a81d55b1bd4d2e9b587c6477d624a2: No bootstrap required, opened a new log
I20240502 21:28:30.411664 32590 ts_tablet_manager.cc:1367] T 1707bbe6fcd44b09ab452215e3997199 P 67a81d55b1bd4d2e9b587c6477d624a2: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20240502 21:28:30.412382 32591 log.cc:824] T 2531188c066d4c33976b53333caed52c P e0a4e8376dae403e9b812a429b8960af: Log is configured to *not* fsync() on all Append() calls
I20240502 21:28:30.414919 32591 tablet_bootstrap.cc:492] T 2531188c066d4c33976b53333caed52c P e0a4e8376dae403e9b812a429b8960af: No bootstrap required, opened a new log
I20240502 21:28:30.415179 32591 ts_tablet_manager.cc:1367] T 2531188c066d4c33976b53333caed52c P e0a4e8376dae403e9b812a429b8960af: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20240502 21:28:30.420461 32590 raft_consensus.cc:348] T 1707bbe6fcd44b09ab452215e3997199 P 67a81d55b1bd4d2e9b587c6477d624a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a81d55b1bd4d2e9b587c6477d624a2" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40197 } } peers { permanent_uuid: "e0a4e8376dae403e9b812a429b8960af" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 42105 } }
I20240502 21:28:30.420872 32590 raft_consensus.cc:374] T 1707bbe6fcd44b09ab452215e3997199 P 67a81d55b1bd4d2e9b587c6477d624a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:30.420981 32590 raft_consensus.cc:729] T 1707bbe6fcd44b09ab452215e3997199 P 67a81d55b1bd4d2e9b587c6477d624a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67a81d55b1bd4d2e9b587c6477d624a2, State: Initialized, Role: FOLLOWER
I20240502 21:28:30.421698 32590 consensus_queue.cc:260] T 1707bbe6fcd44b09ab452215e3997199 P 67a81d55b1bd4d2e9b587c6477d624a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a81d55b1bd4d2e9b587c6477d624a2" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40197 } } peers { permanent_uuid: "e0a4e8376dae403e9b812a429b8960af" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 42105 } }
I20240502 21:28:30.423359 32590 ts_tablet_manager.cc:1398] T 1707bbe6fcd44b09ab452215e3997199 P 67a81d55b1bd4d2e9b587c6477d624a2: Time spent starting tablet: real 0.012s user 0.012s sys 0.001s
I20240502 21:28:30.423799 32590 tablet_bootstrap.cc:492] T d7c55e3c68744428a3bc7f2a1fe50c1f P 67a81d55b1bd4d2e9b587c6477d624a2: Bootstrap starting.
I20240502 21:28:30.423837 32591 raft_consensus.cc:348] T 2531188c066d4c33976b53333caed52c P e0a4e8376dae403e9b812a429b8960af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a81d55b1bd4d2e9b587c6477d624a2" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40197 } } peers { permanent_uuid: "e0a4e8376dae403e9b812a429b8960af" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 42105 } }
I20240502 21:28:30.424304 32591 raft_consensus.cc:374] T 2531188c066d4c33976b53333caed52c P e0a4e8376dae403e9b812a429b8960af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:30.424510 32591 raft_consensus.cc:729] T 2531188c066d4c33976b53333caed52c P e0a4e8376dae403e9b812a429b8960af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0a4e8376dae403e9b812a429b8960af, State: Initialized, Role: FOLLOWER
I20240502 21:28:30.425475 32590 tablet_bootstrap.cc:654] T d7c55e3c68744428a3bc7f2a1fe50c1f P 67a81d55b1bd4d2e9b587c6477d624a2: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:30.425696 32591 consensus_queue.cc:260] T 2531188c066d4c33976b53333caed52c P e0a4e8376dae403e9b812a429b8960af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a81d55b1bd4d2e9b587c6477d624a2" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40197 } } peers { permanent_uuid: "e0a4e8376dae403e9b812a429b8960af" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 42105 } }
I20240502 21:28:30.427630 32590 tablet_bootstrap.cc:492] T d7c55e3c68744428a3bc7f2a1fe50c1f P 67a81d55b1bd4d2e9b587c6477d624a2: No bootstrap required, opened a new log
I20240502 21:28:30.427750 32590 ts_tablet_manager.cc:1367] T d7c55e3c68744428a3bc7f2a1fe50c1f P 67a81d55b1bd4d2e9b587c6477d624a2: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240502 21:28:30.428555 32590 raft_consensus.cc:348] T d7c55e3c68744428a3bc7f2a1fe50c1f P 67a81d55b1bd4d2e9b587c6477d624a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0a4e8376dae403e9b812a429b8960af" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 42105 } } peers { permanent_uuid: "67a81d55b1bd4d2e9b587c6477d624a2" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40197 } }
I20240502 21:28:30.428735 32590 raft_consensus.cc:374] T d7c55e3c68744428a3bc7f2a1fe50c1f P 67a81d55b1bd4d2e9b587c6477d624a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:30.428881 32590 raft_consensus.cc:729] T d7c55e3c68744428a3bc7f2a1fe50c1f P 67a81d55b1bd4d2e9b587c6477d624a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67a81d55b1bd4d2e9b587c6477d624a2, State: Initialized, Role: FOLLOWER
I20240502 21:28:30.428951 32591 ts_tablet_manager.cc:1398] T 2531188c066d4c33976b53333caed52c P e0a4e8376dae403e9b812a429b8960af: Time spent starting tablet: real 0.014s user 0.006s sys 0.007s
I20240502 21:28:30.429129 32590 consensus_queue.cc:260] T d7c55e3c68744428a3bc7f2a1fe50c1f P 67a81d55b1bd4d2e9b587c6477d624a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0a4e8376dae403e9b812a429b8960af" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 42105 } } peers { permanent_uuid: "67a81d55b1bd4d2e9b587c6477d624a2" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40197 } }
I20240502 21:28:30.429415 32591 tablet_bootstrap.cc:492] T d7c55e3c68744428a3bc7f2a1fe50c1f P e0a4e8376dae403e9b812a429b8960af: Bootstrap starting.
I20240502 21:28:30.429651 32590 ts_tablet_manager.cc:1398] T d7c55e3c68744428a3bc7f2a1fe50c1f P 67a81d55b1bd4d2e9b587c6477d624a2: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 21:28:30.429891 32590 tablet_bootstrap.cc:492] T 2531188c066d4c33976b53333caed52c P 67a81d55b1bd4d2e9b587c6477d624a2: Bootstrap starting.
I20240502 21:28:30.431131 32591 tablet_bootstrap.cc:654] T d7c55e3c68744428a3bc7f2a1fe50c1f P e0a4e8376dae403e9b812a429b8960af: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:30.431321 32590 tablet_bootstrap.cc:654] T 2531188c066d4c33976b53333caed52c P 67a81d55b1bd4d2e9b587c6477d624a2: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:30.433215 32590 tablet_bootstrap.cc:492] T 2531188c066d4c33976b53333caed52c P 67a81d55b1bd4d2e9b587c6477d624a2: No bootstrap required, opened a new log
I20240502 21:28:30.433388 32590 ts_tablet_manager.cc:1367] T 2531188c066d4c33976b53333caed52c P 67a81d55b1bd4d2e9b587c6477d624a2: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.002s
I20240502 21:28:30.433925 32590 raft_consensus.cc:348] T 2531188c066d4c33976b53333caed52c P 67a81d55b1bd4d2e9b587c6477d624a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a81d55b1bd4d2e9b587c6477d624a2" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40197 } } peers { permanent_uuid: "e0a4e8376dae403e9b812a429b8960af" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 42105 } }
I20240502 21:28:30.434084 32590 raft_consensus.cc:374] T 2531188c066d4c33976b53333caed52c P 67a81d55b1bd4d2e9b587c6477d624a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:30.434145 32590 raft_consensus.cc:729] T 2531188c066d4c33976b53333caed52c P 67a81d55b1bd4d2e9b587c6477d624a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67a81d55b1bd4d2e9b587c6477d624a2, State: Initialized, Role: FOLLOWER
I20240502 21:28:30.434376 32590 consensus_queue.cc:260] T 2531188c066d4c33976b53333caed52c P 67a81d55b1bd4d2e9b587c6477d624a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a81d55b1bd4d2e9b587c6477d624a2" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40197 } } peers { permanent_uuid: "e0a4e8376dae403e9b812a429b8960af" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 42105 } }
I20240502 21:28:30.434793 32590 ts_tablet_manager.cc:1398] T 2531188c066d4c33976b53333caed52c P 67a81d55b1bd4d2e9b587c6477d624a2: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20240502 21:28:30.437595 32591 tablet_bootstrap.cc:492] T d7c55e3c68744428a3bc7f2a1fe50c1f P e0a4e8376dae403e9b812a429b8960af: No bootstrap required, opened a new log
I20240502 21:28:30.437748 32591 ts_tablet_manager.cc:1367] T d7c55e3c68744428a3bc7f2a1fe50c1f P e0a4e8376dae403e9b812a429b8960af: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.005s
I20240502 21:28:30.438549 32591 raft_consensus.cc:348] T d7c55e3c68744428a3bc7f2a1fe50c1f P e0a4e8376dae403e9b812a429b8960af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0a4e8376dae403e9b812a429b8960af" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 42105 } } peers { permanent_uuid: "67a81d55b1bd4d2e9b587c6477d624a2" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40197 } }
I20240502 21:28:30.438688 32591 raft_consensus.cc:374] T d7c55e3c68744428a3bc7f2a1fe50c1f P e0a4e8376dae403e9b812a429b8960af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:30.438781 32591 raft_consensus.cc:729] T d7c55e3c68744428a3bc7f2a1fe50c1f P e0a4e8376dae403e9b812a429b8960af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0a4e8376dae403e9b812a429b8960af, State: Initialized, Role: FOLLOWER
I20240502 21:28:30.438987 32591 consensus_queue.cc:260] T d7c55e3c68744428a3bc7f2a1fe50c1f P e0a4e8376dae403e9b812a429b8960af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0a4e8376dae403e9b812a429b8960af" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 42105 } } peers { permanent_uuid: "67a81d55b1bd4d2e9b587c6477d624a2" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40197 } }
I20240502 21:28:30.439430 32591 ts_tablet_manager.cc:1398] T d7c55e3c68744428a3bc7f2a1fe50c1f P e0a4e8376dae403e9b812a429b8960af: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 21:28:30.439699 32591 tablet_bootstrap.cc:492] T 1707bbe6fcd44b09ab452215e3997199 P e0a4e8376dae403e9b812a429b8960af: Bootstrap starting.
I20240502 21:28:30.440953 32591 tablet_bootstrap.cc:654] T 1707bbe6fcd44b09ab452215e3997199 P e0a4e8376dae403e9b812a429b8960af: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:30.442829 32591 tablet_bootstrap.cc:492] T 1707bbe6fcd44b09ab452215e3997199 P e0a4e8376dae403e9b812a429b8960af: No bootstrap required, opened a new log
I20240502 21:28:30.442970 32591 ts_tablet_manager.cc:1367] T 1707bbe6fcd44b09ab452215e3997199 P e0a4e8376dae403e9b812a429b8960af: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 21:28:30.443563 32591 raft_consensus.cc:348] T 1707bbe6fcd44b09ab452215e3997199 P e0a4e8376dae403e9b812a429b8960af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a81d55b1bd4d2e9b587c6477d624a2" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40197 } } peers { permanent_uuid: "e0a4e8376dae403e9b812a429b8960af" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 42105 } }
I20240502 21:28:30.443729 32591 raft_consensus.cc:374] T 1707bbe6fcd44b09ab452215e3997199 P e0a4e8376dae403e9b812a429b8960af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:30.443789 32591 raft_consensus.cc:729] T 1707bbe6fcd44b09ab452215e3997199 P e0a4e8376dae403e9b812a429b8960af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0a4e8376dae403e9b812a429b8960af, State: Initialized, Role: FOLLOWER
I20240502 21:28:30.443987 32591 consensus_queue.cc:260] T 1707bbe6fcd44b09ab452215e3997199 P e0a4e8376dae403e9b812a429b8960af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a81d55b1bd4d2e9b587c6477d624a2" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40197 } } peers { permanent_uuid: "e0a4e8376dae403e9b812a429b8960af" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 42105 } }
I20240502 21:28:30.444463 32595 raft_consensus.cc:482] T 2531188c066d4c33976b53333caed52c P e0a4e8376dae403e9b812a429b8960af [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:28:30.444593 32591 ts_tablet_manager.cc:1398] T 1707bbe6fcd44b09ab452215e3997199 P e0a4e8376dae403e9b812a429b8960af: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 21:28:30.444684 32595 raft_consensus.cc:504] T 2531188c066d4c33976b53333caed52c P e0a4e8376dae403e9b812a429b8960af [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a81d55b1bd4d2e9b587c6477d624a2" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40197 } } peers { permanent_uuid: "e0a4e8376dae403e9b812a429b8960af" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 42105 } }
I20240502 21:28:30.446221 32595 leader_election.cc:288] T 2531188c066d4c33976b53333caed52c P e0a4e8376dae403e9b812a429b8960af [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 67a81d55b1bd4d2e9b587c6477d624a2 (127.15.255.129:40197)
I20240502 21:28:30.453023 32126 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2531188c066d4c33976b53333caed52c" candidate_uuid: "e0a4e8376dae403e9b812a429b8960af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67a81d55b1bd4d2e9b587c6477d624a2" is_pre_election: true
I20240502 21:28:30.453465 32126 raft_consensus.cc:2454] T 2531188c066d4c33976b53333caed52c P 67a81d55b1bd4d2e9b587c6477d624a2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0a4e8376dae403e9b812a429b8960af in term 0.
I20240502 21:28:30.453958 32455 leader_election.cc:302] T 2531188c066d4c33976b53333caed52c P e0a4e8376dae403e9b812a429b8960af [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 67a81d55b1bd4d2e9b587c6477d624a2, e0a4e8376dae403e9b812a429b8960af; no voters:
I20240502 21:28:30.454265 32600 raft_consensus.cc:2781] T 2531188c066d4c33976b53333caed52c P e0a4e8376dae403e9b812a429b8960af [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:28:30.454387 32600 raft_consensus.cc:482] T 2531188c066d4c33976b53333caed52c P e0a4e8376dae403e9b812a429b8960af [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:28:30.454515 32600 raft_consensus.cc:3037] T 2531188c066d4c33976b53333caed52c P e0a4e8376dae403e9b812a429b8960af [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:30.456560 32600 raft_consensus.cc:504] T 2531188c066d4c33976b53333caed52c P e0a4e8376dae403e9b812a429b8960af [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a81d55b1bd4d2e9b587c6477d624a2" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40197 } } peers { permanent_uuid: "e0a4e8376dae403e9b812a429b8960af" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 42105 } }
I20240502 21:28:30.457037 32600 leader_election.cc:288] T 2531188c066d4c33976b53333caed52c P e0a4e8376dae403e9b812a429b8960af [CANDIDATE]: Term 1 election: Requested vote from peers 67a81d55b1bd4d2e9b587c6477d624a2 (127.15.255.129:40197)
I20240502 21:28:30.457444 32126 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2531188c066d4c33976b53333caed52c" candidate_uuid: "e0a4e8376dae403e9b812a429b8960af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67a81d55b1bd4d2e9b587c6477d624a2"
I20240502 21:28:30.457594 32126 raft_consensus.cc:3037] T 2531188c066d4c33976b53333caed52c P 67a81d55b1bd4d2e9b587c6477d624a2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:30.459573 32126 raft_consensus.cc:2454] T 2531188c066d4c33976b53333caed52c P 67a81d55b1bd4d2e9b587c6477d624a2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0a4e8376dae403e9b812a429b8960af in term 1.
I20240502 21:28:30.459892 32455 leader_election.cc:302] T 2531188c066d4c33976b53333caed52c P e0a4e8376dae403e9b812a429b8960af [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 67a81d55b1bd4d2e9b587c6477d624a2, e0a4e8376dae403e9b812a429b8960af; no voters:
I20240502 21:28:30.460119 32600 raft_consensus.cc:2781] T 2531188c066d4c33976b53333caed52c P e0a4e8376dae403e9b812a429b8960af [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:28:30.460271 32600 raft_consensus.cc:686] T 2531188c066d4c33976b53333caed52c P e0a4e8376dae403e9b812a429b8960af [term 1 LEADER]: Becoming Leader. State: Replica: e0a4e8376dae403e9b812a429b8960af, State: Running, Role: LEADER
I20240502 21:28:30.460640 32600 consensus_queue.cc:237] T 2531188c066d4c33976b53333caed52c P e0a4e8376dae403e9b812a429b8960af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a81d55b1bd4d2e9b587c6477d624a2" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40197 } } peers { permanent_uuid: "e0a4e8376dae403e9b812a429b8960af" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 42105 } }
I20240502 21:28:30.464588 31981 catalog_manager.cc:5484] T 2531188c066d4c33976b53333caed52c P e0a4e8376dae403e9b812a429b8960af reported cstate change: term changed from 0 to 1, leader changed from <none> to e0a4e8376dae403e9b812a429b8960af (127.15.255.132). New cstate: current_term: 1 leader_uuid: "e0a4e8376dae403e9b812a429b8960af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a81d55b1bd4d2e9b587c6477d624a2" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40197 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e0a4e8376dae403e9b812a429b8960af" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 42105 } health_report { overall_health: HEALTHY } } }
I20240502 21:28:30.479447 32600 raft_consensus.cc:482] T d7c55e3c68744428a3bc7f2a1fe50c1f P e0a4e8376dae403e9b812a429b8960af [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:28:30.479574 32600 raft_consensus.cc:504] T d7c55e3c68744428a3bc7f2a1fe50c1f P e0a4e8376dae403e9b812a429b8960af [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0a4e8376dae403e9b812a429b8960af" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 42105 } } peers { permanent_uuid: "67a81d55b1bd4d2e9b587c6477d624a2" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40197 } }
I20240502 21:28:30.479998 32600 leader_election.cc:288] T d7c55e3c68744428a3bc7f2a1fe50c1f P e0a4e8376dae403e9b812a429b8960af [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 67a81d55b1bd4d2e9b587c6477d624a2 (127.15.255.129:40197)
I20240502 21:28:30.480441 32126 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7c55e3c68744428a3bc7f2a1fe50c1f" candidate_uuid: "e0a4e8376dae403e9b812a429b8960af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67a81d55b1bd4d2e9b587c6477d624a2" is_pre_election: true
I20240502 21:28:30.480638 32126 raft_consensus.cc:2454] T d7c55e3c68744428a3bc7f2a1fe50c1f P 67a81d55b1bd4d2e9b587c6477d624a2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0a4e8376dae403e9b812a429b8960af in term 0.
I20240502 21:28:30.480932 32455 leader_election.cc:302] T d7c55e3c68744428a3bc7f2a1fe50c1f P e0a4e8376dae403e9b812a429b8960af [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 67a81d55b1bd4d2e9b587c6477d624a2, e0a4e8376dae403e9b812a429b8960af; no voters:
I20240502 21:28:30.481180 32600 raft_consensus.cc:2781] T d7c55e3c68744428a3bc7f2a1fe50c1f P e0a4e8376dae403e9b812a429b8960af [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:28:30.481292 32600 raft_consensus.cc:482] T d7c55e3c68744428a3bc7f2a1fe50c1f P e0a4e8376dae403e9b812a429b8960af [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:28:30.481371 32600 raft_consensus.cc:3037] T d7c55e3c68744428a3bc7f2a1fe50c1f P e0a4e8376dae403e9b812a429b8960af [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:30.483145 32600 raft_consensus.cc:504] T d7c55e3c68744428a3bc7f2a1fe50c1f P e0a4e8376dae403e9b812a429b8960af [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0a4e8376dae403e9b812a429b8960af" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 42105 } } peers { permanent_uuid: "67a81d55b1bd4d2e9b587c6477d624a2" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40197 } }
I20240502 21:28:30.483562 32600 leader_election.cc:288] T d7c55e3c68744428a3bc7f2a1fe50c1f P e0a4e8376dae403e9b812a429b8960af [CANDIDATE]: Term 1 election: Requested vote from peers 67a81d55b1bd4d2e9b587c6477d624a2 (127.15.255.129:40197)
I20240502 21:28:30.483973 32126 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7c55e3c68744428a3bc7f2a1fe50c1f" candidate_uuid: "e0a4e8376dae403e9b812a429b8960af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67a81d55b1bd4d2e9b587c6477d624a2"
I20240502 21:28:30.484153 32126 raft_consensus.cc:3037] T d7c55e3c68744428a3bc7f2a1fe50c1f P 67a81d55b1bd4d2e9b587c6477d624a2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:30.486670 32126 raft_consensus.cc:2454] T d7c55e3c68744428a3bc7f2a1fe50c1f P 67a81d55b1bd4d2e9b587c6477d624a2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0a4e8376dae403e9b812a429b8960af in term 1.
I20240502 21:28:30.487036 32455 leader_election.cc:302] T d7c55e3c68744428a3bc7f2a1fe50c1f P e0a4e8376dae403e9b812a429b8960af [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 67a81d55b1bd4d2e9b587c6477d624a2, e0a4e8376dae403e9b812a429b8960af; no voters:
I20240502 21:28:30.487368 32600 raft_consensus.cc:2781] T d7c55e3c68744428a3bc7f2a1fe50c1f P e0a4e8376dae403e9b812a429b8960af [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:28:30.487501 32600 raft_consensus.cc:686] T d7c55e3c68744428a3bc7f2a1fe50c1f P e0a4e8376dae403e9b812a429b8960af [term 1 LEADER]: Becoming Leader. State: Replica: e0a4e8376dae403e9b812a429b8960af, State: Running, Role: LEADER
I20240502 21:28:30.487741 32600 consensus_queue.cc:237] T d7c55e3c68744428a3bc7f2a1fe50c1f P e0a4e8376dae403e9b812a429b8960af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0a4e8376dae403e9b812a429b8960af" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 42105 } } peers { permanent_uuid: "67a81d55b1bd4d2e9b587c6477d624a2" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40197 } }
I20240502 21:28:30.489531 31981 catalog_manager.cc:5484] T d7c55e3c68744428a3bc7f2a1fe50c1f P e0a4e8376dae403e9b812a429b8960af reported cstate change: term changed from 0 to 1, leader changed from <none> to e0a4e8376dae403e9b812a429b8960af (127.15.255.132). New cstate: current_term: 1 leader_uuid: "e0a4e8376dae403e9b812a429b8960af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0a4e8376dae403e9b812a429b8960af" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 42105 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "67a81d55b1bd4d2e9b587c6477d624a2" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40197 } health_report { overall_health: UNKNOWN } } }
W20240502 21:28:30.508143 32172 tablet.cc:2369] T 1707bbe6fcd44b09ab452215e3997199 P 67a81d55b1bd4d2e9b587c6477d624a2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240502 21:28:30.574745 32569 tablet.cc:2369] T 1707bbe6fcd44b09ab452215e3997199 P e0a4e8376dae403e9b812a429b8960af: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 21:28:30.669588 32600 raft_consensus.cc:482] T 1707bbe6fcd44b09ab452215e3997199 P e0a4e8376dae403e9b812a429b8960af [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:28:30.669818 32600 raft_consensus.cc:504] T 1707bbe6fcd44b09ab452215e3997199 P e0a4e8376dae403e9b812a429b8960af [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a81d55b1bd4d2e9b587c6477d624a2" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40197 } } peers { permanent_uuid: "e0a4e8376dae403e9b812a429b8960af" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 42105 } }
I20240502 21:28:30.670469 32600 leader_election.cc:288] T 1707bbe6fcd44b09ab452215e3997199 P e0a4e8376dae403e9b812a429b8960af [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 67a81d55b1bd4d2e9b587c6477d624a2 (127.15.255.129:40197)
I20240502 21:28:30.671020 32126 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1707bbe6fcd44b09ab452215e3997199" candidate_uuid: "e0a4e8376dae403e9b812a429b8960af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67a81d55b1bd4d2e9b587c6477d624a2" is_pre_election: true
I20240502 21:28:30.671250 32126 raft_consensus.cc:2454] T 1707bbe6fcd44b09ab452215e3997199 P 67a81d55b1bd4d2e9b587c6477d624a2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0a4e8376dae403e9b812a429b8960af in term 0.
I20240502 21:28:30.671674 32455 leader_election.cc:302] T 1707bbe6fcd44b09ab452215e3997199 P e0a4e8376dae403e9b812a429b8960af [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 67a81d55b1bd4d2e9b587c6477d624a2, e0a4e8376dae403e9b812a429b8960af; no voters:
I20240502 21:28:30.671990 32600 raft_consensus.cc:2781] T 1707bbe6fcd44b09ab452215e3997199 P e0a4e8376dae403e9b812a429b8960af [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:28:30.672102 32600 raft_consensus.cc:482] T 1707bbe6fcd44b09ab452215e3997199 P e0a4e8376dae403e9b812a429b8960af [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:28:30.672214 32600 raft_consensus.cc:3037] T 1707bbe6fcd44b09ab452215e3997199 P e0a4e8376dae403e9b812a429b8960af [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:30.675068 32600 raft_consensus.cc:504] T 1707bbe6fcd44b09ab452215e3997199 P e0a4e8376dae403e9b812a429b8960af [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a81d55b1bd4d2e9b587c6477d624a2" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40197 } } peers { permanent_uuid: "e0a4e8376dae403e9b812a429b8960af" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 42105 } }
I20240502 21:28:30.675601 32600 leader_election.cc:288] T 1707bbe6fcd44b09ab452215e3997199 P e0a4e8376dae403e9b812a429b8960af [CANDIDATE]: Term 1 election: Requested vote from peers 67a81d55b1bd4d2e9b587c6477d624a2 (127.15.255.129:40197)
I20240502 21:28:30.676018 32126 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1707bbe6fcd44b09ab452215e3997199" candidate_uuid: "e0a4e8376dae403e9b812a429b8960af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67a81d55b1bd4d2e9b587c6477d624a2"
I20240502 21:28:30.676187 32126 raft_consensus.cc:3037] T 1707bbe6fcd44b09ab452215e3997199 P 67a81d55b1bd4d2e9b587c6477d624a2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:30.677999 32126 raft_consensus.cc:2454] T 1707bbe6fcd44b09ab452215e3997199 P 67a81d55b1bd4d2e9b587c6477d624a2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0a4e8376dae403e9b812a429b8960af in term 1.
I20240502 21:28:30.678364 32455 leader_election.cc:302] T 1707bbe6fcd44b09ab452215e3997199 P e0a4e8376dae403e9b812a429b8960af [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 67a81d55b1bd4d2e9b587c6477d624a2, e0a4e8376dae403e9b812a429b8960af; no voters:
I20240502 21:28:30.678603 32600 raft_consensus.cc:2781] T 1707bbe6fcd44b09ab452215e3997199 P e0a4e8376dae403e9b812a429b8960af [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:28:30.678787 32600 raft_consensus.cc:686] T 1707bbe6fcd44b09ab452215e3997199 P e0a4e8376dae403e9b812a429b8960af [term 1 LEADER]: Becoming Leader. State: Replica: e0a4e8376dae403e9b812a429b8960af, State: Running, Role: LEADER
I20240502 21:28:30.679080 32600 consensus_queue.cc:237] T 1707bbe6fcd44b09ab452215e3997199 P e0a4e8376dae403e9b812a429b8960af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a81d55b1bd4d2e9b587c6477d624a2" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40197 } } peers { permanent_uuid: "e0a4e8376dae403e9b812a429b8960af" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 42105 } }
I20240502 21:28:30.681968 31981 catalog_manager.cc:5484] T 1707bbe6fcd44b09ab452215e3997199 P e0a4e8376dae403e9b812a429b8960af reported cstate change: term changed from 0 to 1, leader changed from <none> to e0a4e8376dae403e9b812a429b8960af (127.15.255.132). New cstate: current_term: 1 leader_uuid: "e0a4e8376dae403e9b812a429b8960af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a81d55b1bd4d2e9b587c6477d624a2" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40197 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e0a4e8376dae403e9b812a429b8960af" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 42105 } health_report { overall_health: HEALTHY } } }
I20240502 21:28:30.886292 32600 consensus_queue.cc:1035] T 2531188c066d4c33976b53333caed52c P e0a4e8376dae403e9b812a429b8960af [LEADER]: Connected to new peer: Peer: permanent_uuid: "67a81d55b1bd4d2e9b587c6477d624a2" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 21:28:31.061074 32605 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:28:31.061352 32605 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240502 21:28:31.063479 32595 consensus_queue.cc:1035] T d7c55e3c68744428a3bc7f2a1fe50c1f P e0a4e8376dae403e9b812a429b8960af [LEADER]: Connected to new peer: Peer: permanent_uuid: "67a81d55b1bd4d2e9b587c6477d624a2" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 21:28:31.070297 32605 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_0'
I20240502 21:28:31.131172 32608 consensus_queue.cc:1035] T 1707bbe6fcd44b09ab452215e3997199 P e0a4e8376dae403e9b812a429b8960af [LEADER]: Connected to new peer: Peer: permanent_uuid: "67a81d55b1bd4d2e9b587c6477d624a2" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 132.485 ms
time per row: 0.0662423 ms
I20240502 21:28:31.332594 31981 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49812:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
W20240502 21:28:31.333407 31981 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table rebalance_test_table_1 in case of a server failure: 3 tablet servers would be needed, 2 are available. Consider bringing up more tablet servers.
I20240502 21:28:31.341749 32104 tablet_service.cc:1467] Processing CreateTablet for tablet fae3d984510a4250bc7812d4387c513d (DEFAULT_TABLE table=rebalance_test_table_1 [id=ec8886b062014ed78b9bf3e9a156193a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:31.341766 32503 tablet_service.cc:1467] Processing CreateTablet for tablet fae3d984510a4250bc7812d4387c513d (DEFAULT_TABLE table=rebalance_test_table_1 [id=ec8886b062014ed78b9bf3e9a156193a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:31.342207 32104 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fae3d984510a4250bc7812d4387c513d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:31.342217 32503 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fae3d984510a4250bc7812d4387c513d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:31.342906 32105 tablet_service.cc:1467] Processing CreateTablet for tablet 39d0ac5ec0cd4a33b947de380b5469be (DEFAULT_TABLE table=rebalance_test_table_1 [id=ec8886b062014ed78b9bf3e9a156193a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:31.343317 32105 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39d0ac5ec0cd4a33b947de380b5469be. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:31.347765 32103 tablet_service.cc:1467] Processing CreateTablet for tablet 0bf5f9db2efe49029fa1da7399ab5373 (DEFAULT_TABLE table=rebalance_test_table_1 [id=ec8886b062014ed78b9bf3e9a156193a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:31.348116 32103 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0bf5f9db2efe49029fa1da7399ab5373. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:31.350118 32654 tablet_bootstrap.cc:492] T fae3d984510a4250bc7812d4387c513d P e0a4e8376dae403e9b812a429b8960af: Bootstrap starting.
I20240502 21:28:31.351286 32501 tablet_service.cc:1467] Processing CreateTablet for tablet 39d0ac5ec0cd4a33b947de380b5469be (DEFAULT_TABLE table=rebalance_test_table_1 [id=ec8886b062014ed78b9bf3e9a156193a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:31.351667 32501 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39d0ac5ec0cd4a33b947de380b5469be. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:31.351725 32654 tablet_bootstrap.cc:654] T fae3d984510a4250bc7812d4387c513d P e0a4e8376dae403e9b812a429b8960af: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:31.356920 32655 tablet_bootstrap.cc:492] T fae3d984510a4250bc7812d4387c513d P 67a81d55b1bd4d2e9b587c6477d624a2: Bootstrap starting.
I20240502 21:28:31.358192 32503 tablet_service.cc:1467] Processing CreateTablet for tablet 0bf5f9db2efe49029fa1da7399ab5373 (DEFAULT_TABLE table=rebalance_test_table_1 [id=ec8886b062014ed78b9bf3e9a156193a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:31.358549 32503 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0bf5f9db2efe49029fa1da7399ab5373. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:31.359019 32655 tablet_bootstrap.cc:654] T fae3d984510a4250bc7812d4387c513d P 67a81d55b1bd4d2e9b587c6477d624a2: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:31.361771 32655 tablet_bootstrap.cc:492] T fae3d984510a4250bc7812d4387c513d P 67a81d55b1bd4d2e9b587c6477d624a2: No bootstrap required, opened a new log
I20240502 21:28:31.361909 32655 ts_tablet_manager.cc:1367] T fae3d984510a4250bc7812d4387c513d P 67a81d55b1bd4d2e9b587c6477d624a2: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20240502 21:28:31.361956 32654 tablet_bootstrap.cc:492] T fae3d984510a4250bc7812d4387c513d P e0a4e8376dae403e9b812a429b8960af: No bootstrap required, opened a new log
I20240502 21:28:31.362094 32654 ts_tablet_manager.cc:1367] T fae3d984510a4250bc7812d4387c513d P e0a4e8376dae403e9b812a429b8960af: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.008s
I20240502 21:28:31.362666 32655 raft_consensus.cc:348] T fae3d984510a4250bc7812d4387c513d P 67a81d55b1bd4d2e9b587c6477d624a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0a4e8376dae403e9b812a429b8960af" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 42105 } } peers { permanent_uuid: "67a81d55b1bd4d2e9b587c6477d624a2" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40197 } }
I20240502 21:28:31.362859 32655 raft_consensus.cc:374] T fae3d984510a4250bc7812d4387c513d P 67a81d55b1bd4d2e9b587c6477d624a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:31.362949 32655 raft_consensus.cc:729] T fae3d984510a4250bc7812d4387c513d P 67a81d55b1bd4d2e9b587c6477d624a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67a81d55b1bd4d2e9b587c6477d624a2, State: Initialized, Role: FOLLOWER
I20240502 21:28:31.362915 32654 raft_consensus.cc:348] T fae3d984510a4250bc7812d4387c513d P e0a4e8376dae403e9b812a429b8960af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0a4e8376dae403e9b812a429b8960af" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 42105 } } peers { permanent_uuid: "67a81d55b1bd4d2e9b587c6477d624a2" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40197 } }
I20240502 21:28:31.363101 32654 raft_consensus.cc:374] T fae3d984510a4250bc7812d4387c513d P e0a4e8376dae403e9b812a429b8960af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:31.363209 32654 raft_consensus.cc:729] T fae3d984510a4250bc7812d4387c513d P e0a4e8376dae403e9b812a429b8960af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0a4e8376dae403e9b812a429b8960af, State: Initialized, Role: FOLLOWER
I20240502 21:28:31.363237 32655 consensus_queue.cc:260] T fae3d984510a4250bc7812d4387c513d P 67a81d55b1bd4d2e9b587c6477d624a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0a4e8376dae403e9b812a429b8960af" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 42105 } } peers { permanent_uuid: "67a81d55b1bd4d2e9b587c6477d624a2" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40197 } }
I20240502 21:28:31.363421 32654 consensus_queue.cc:260] T fae3d984510a4250bc7812d4387c513d P e0a4e8376dae403e9b812a429b8960af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0a4e8376dae403e9b812a429b8960af" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 42105 } } peers { permanent_uuid: "67a81d55b1bd4d2e9b587c6477d624a2" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40197 } }
I20240502 21:28:31.363782 32655 ts_tablet_manager.cc:1398] T fae3d984510a4250bc7812d4387c513d P 67a81d55b1bd4d2e9b587c6477d624a2: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 21:28:31.364095 32655 tablet_bootstrap.cc:492] T 39d0ac5ec0cd4a33b947de380b5469be P 67a81d55b1bd4d2e9b587c6477d624a2: Bootstrap starting.
I20240502 21:28:31.365029 32654 ts_tablet_manager.cc:1398] T fae3d984510a4250bc7812d4387c513d P e0a4e8376dae403e9b812a429b8960af: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20240502 21:28:31.365326 32654 tablet_bootstrap.cc:492] T 39d0ac5ec0cd4a33b947de380b5469be P e0a4e8376dae403e9b812a429b8960af: Bootstrap starting.
I20240502 21:28:31.365578 32655 tablet_bootstrap.cc:654] T 39d0ac5ec0cd4a33b947de380b5469be P 67a81d55b1bd4d2e9b587c6477d624a2: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:31.366714 32654 tablet_bootstrap.cc:654] T 39d0ac5ec0cd4a33b947de380b5469be P e0a4e8376dae403e9b812a429b8960af: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:31.367445 32655 tablet_bootstrap.cc:492] T 39d0ac5ec0cd4a33b947de380b5469be P 67a81d55b1bd4d2e9b587c6477d624a2: No bootstrap required, opened a new log
I20240502 21:28:31.367682 32655 ts_tablet_manager.cc:1367] T 39d0ac5ec0cd4a33b947de380b5469be P 67a81d55b1bd4d2e9b587c6477d624a2: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240502 21:28:31.368508 32655 raft_consensus.cc:348] T 39d0ac5ec0cd4a33b947de380b5469be P 67a81d55b1bd4d2e9b587c6477d624a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0a4e8376dae403e9b812a429b8960af" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 42105 } } peers { permanent_uuid: "67a81d55b1bd4d2e9b587c6477d624a2" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40197 } }
I20240502 21:28:31.368777 32655 raft_consensus.cc:374] T 39d0ac5ec0cd4a33b947de380b5469be P 67a81d55b1bd4d2e9b587c6477d624a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:31.368858 32655 raft_consensus.cc:729] T 39d0ac5ec0cd4a33b947de380b5469be P 67a81d55b1bd4d2e9b587c6477d624a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67a81d55b1bd4d2e9b587c6477d624a2, State: Initialized, Role: FOLLOWER
I20240502 21:28:31.369109 32655 consensus_queue.cc:260] T 39d0ac5ec0cd4a33b947de380b5469be P 67a81d55b1bd4d2e9b587c6477d624a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0a4e8376dae403e9b812a429b8960af" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 42105 } } peers { permanent_uuid: "67a81d55b1bd4d2e9b587c6477d624a2" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40197 } }
I20240502 21:28:31.369735 32655 ts_tablet_manager.cc:1398] T 39d0ac5ec0cd4a33b947de380b5469be P 67a81d55b1bd4d2e9b587c6477d624a2: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 21:28:31.370040 32655 tablet_bootstrap.cc:492] T 0bf5f9db2efe49029fa1da7399ab5373 P 67a81d55b1bd4d2e9b587c6477d624a2: Bootstrap starting.
I20240502 21:28:31.371587 32655 tablet_bootstrap.cc:654] T 0bf5f9db2efe49029fa1da7399ab5373 P 67a81d55b1bd4d2e9b587c6477d624a2: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:31.373378 32655 tablet_bootstrap.cc:492] T 0bf5f9db2efe49029fa1da7399ab5373 P 67a81d55b1bd4d2e9b587c6477d624a2: No bootstrap required, opened a new log
I20240502 21:28:31.373479 32655 ts_tablet_manager.cc:1367] T 0bf5f9db2efe49029fa1da7399ab5373 P 67a81d55b1bd4d2e9b587c6477d624a2: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240502 21:28:31.373562 32654 tablet_bootstrap.cc:492] T 39d0ac5ec0cd4a33b947de380b5469be P e0a4e8376dae403e9b812a429b8960af: No bootstrap required, opened a new log
I20240502 21:28:31.373697 32654 ts_tablet_manager.cc:1367] T 39d0ac5ec0cd4a33b947de380b5469be P e0a4e8376dae403e9b812a429b8960af: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.004s
I20240502 21:28:31.374002 32655 raft_consensus.cc:348] T 0bf5f9db2efe49029fa1da7399ab5373 P 67a81d55b1bd4d2e9b587c6477d624a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a81d55b1bd4d2e9b587c6477d624a2" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40197 } } peers { permanent_uuid: "e0a4e8376dae403e9b812a429b8960af" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 42105 } }
I20240502 21:28:31.374135 32655 raft_consensus.cc:374] T 0bf5f9db2efe49029fa1da7399ab5373 P 67a81d55b1bd4d2e9b587c6477d624a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:31.374215 32655 raft_consensus.cc:729] T 0bf5f9db2efe49029fa1da7399ab5373 P 67a81d55b1bd4d2e9b587c6477d624a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67a81d55b1bd4d2e9b587c6477d624a2, State: Initialized, Role: FOLLOWER
I20240502 21:28:31.374249 32654 raft_consensus.cc:348] T 39d0ac5ec0cd4a33b947de380b5469be P e0a4e8376dae403e9b812a429b8960af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0a4e8376dae403e9b812a429b8960af" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 42105 } } peers { permanent_uuid: "67a81d55b1bd4d2e9b587c6477d624a2" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40197 } }
I20240502 21:28:31.374445 32654 raft_consensus.cc:374] T 39d0ac5ec0cd4a33b947de380b5469be P e0a4e8376dae403e9b812a429b8960af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:31.374395 32655 consensus_queue.cc:260] T 0bf5f9db2efe49029fa1da7399ab5373 P 67a81d55b1bd4d2e9b587c6477d624a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a81d55b1bd4d2e9b587c6477d624a2" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40197 } } peers { permanent_uuid: "e0a4e8376dae403e9b812a429b8960af" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 42105 } }
I20240502 21:28:31.374517 32654 raft_consensus.cc:729] T 39d0ac5ec0cd4a33b947de380b5469be P e0a4e8376dae403e9b812a429b8960af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0a4e8376dae403e9b812a429b8960af, State: Initialized, Role: FOLLOWER
I20240502 21:28:31.374795 32655 ts_tablet_manager.cc:1398] T 0bf5f9db2efe49029fa1da7399ab5373 P 67a81d55b1bd4d2e9b587c6477d624a2: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 21:28:31.374732 32654 consensus_queue.cc:260] T 39d0ac5ec0cd4a33b947de380b5469be P e0a4e8376dae403e9b812a429b8960af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0a4e8376dae403e9b812a429b8960af" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 42105 } } peers { permanent_uuid: "67a81d55b1bd4d2e9b587c6477d624a2" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40197 } }
I20240502 21:28:31.375656 32654 ts_tablet_manager.cc:1398] T 39d0ac5ec0cd4a33b947de380b5469be P e0a4e8376dae403e9b812a429b8960af: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 21:28:31.375927 32654 tablet_bootstrap.cc:492] T 0bf5f9db2efe49029fa1da7399ab5373 P e0a4e8376dae403e9b812a429b8960af: Bootstrap starting.
I20240502 21:28:31.377441 32654 tablet_bootstrap.cc:654] T 0bf5f9db2efe49029fa1da7399ab5373 P e0a4e8376dae403e9b812a429b8960af: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:31.379213 32654 tablet_bootstrap.cc:492] T 0bf5f9db2efe49029fa1da7399ab5373 P e0a4e8376dae403e9b812a429b8960af: No bootstrap required, opened a new log
I20240502 21:28:31.379336 32654 ts_tablet_manager.cc:1367] T 0bf5f9db2efe49029fa1da7399ab5373 P e0a4e8376dae403e9b812a429b8960af: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.003s
I20240502 21:28:31.379825 32654 raft_consensus.cc:348] T 0bf5f9db2efe49029fa1da7399ab5373 P e0a4e8376dae403e9b812a429b8960af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a81d55b1bd4d2e9b587c6477d624a2" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40197 } } peers { permanent_uuid: "e0a4e8376dae403e9b812a429b8960af" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 42105 } }
I20240502 21:28:31.379961 32654 raft_consensus.cc:374] T 0bf5f9db2efe49029fa1da7399ab5373 P e0a4e8376dae403e9b812a429b8960af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:31.380045 32654 raft_consensus.cc:729] T 0bf5f9db2efe49029fa1da7399ab5373 P e0a4e8376dae403e9b812a429b8960af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0a4e8376dae403e9b812a429b8960af, State: Initialized, Role: FOLLOWER
I20240502 21:28:31.380219 32654 consensus_queue.cc:260] T 0bf5f9db2efe49029fa1da7399ab5373 P e0a4e8376dae403e9b812a429b8960af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a81d55b1bd4d2e9b587c6477d624a2" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40197 } } peers { permanent_uuid: "e0a4e8376dae403e9b812a429b8960af" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 42105 } }
I20240502 21:28:31.380635 32654 ts_tablet_manager.cc:1398] T 0bf5f9db2efe49029fa1da7399ab5373 P e0a4e8376dae403e9b812a429b8960af: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240502 21:28:31.407706 32594 raft_consensus.cc:482] T fae3d984510a4250bc7812d4387c513d P 67a81d55b1bd4d2e9b587c6477d624a2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:28:31.407848 32594 raft_consensus.cc:504] T fae3d984510a4250bc7812d4387c513d P 67a81d55b1bd4d2e9b587c6477d624a2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0a4e8376dae403e9b812a429b8960af" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 42105 } } peers { permanent_uuid: "67a81d55b1bd4d2e9b587c6477d624a2" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40197 } }
I20240502 21:28:31.409312 32594 leader_election.cc:288] T fae3d984510a4250bc7812d4387c513d P 67a81d55b1bd4d2e9b587c6477d624a2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e0a4e8376dae403e9b812a429b8960af (127.15.255.132:42105)
I20240502 21:28:31.414754 32523 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fae3d984510a4250bc7812d4387c513d" candidate_uuid: "67a81d55b1bd4d2e9b587c6477d624a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0a4e8376dae403e9b812a429b8960af" is_pre_election: true
I20240502 21:28:31.415079 32523 raft_consensus.cc:2454] T fae3d984510a4250bc7812d4387c513d P e0a4e8376dae403e9b812a429b8960af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67a81d55b1bd4d2e9b587c6477d624a2 in term 0.
I20240502 21:28:31.415532 32059 leader_election.cc:302] T fae3d984510a4250bc7812d4387c513d P 67a81d55b1bd4d2e9b587c6477d624a2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 67a81d55b1bd4d2e9b587c6477d624a2, e0a4e8376dae403e9b812a429b8960af; no voters:
I20240502 21:28:31.415920 32594 raft_consensus.cc:2781] T fae3d984510a4250bc7812d4387c513d P 67a81d55b1bd4d2e9b587c6477d624a2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:28:31.416031 32594 raft_consensus.cc:482] T fae3d984510a4250bc7812d4387c513d P 67a81d55b1bd4d2e9b587c6477d624a2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:28:31.416133 32594 raft_consensus.cc:3037] T fae3d984510a4250bc7812d4387c513d P 67a81d55b1bd4d2e9b587c6477d624a2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:31.418134 32594 raft_consensus.cc:504] T fae3d984510a4250bc7812d4387c513d P 67a81d55b1bd4d2e9b587c6477d624a2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0a4e8376dae403e9b812a429b8960af" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 42105 } } peers { permanent_uuid: "67a81d55b1bd4d2e9b587c6477d624a2" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40197 } }
I20240502 21:28:31.418543 32594 leader_election.cc:288] T fae3d984510a4250bc7812d4387c513d P 67a81d55b1bd4d2e9b587c6477d624a2 [CANDIDATE]: Term 1 election: Requested vote from peers e0a4e8376dae403e9b812a429b8960af (127.15.255.132:42105)
I20240502 21:28:31.418964 32523 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fae3d984510a4250bc7812d4387c513d" candidate_uuid: "67a81d55b1bd4d2e9b587c6477d624a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0a4e8376dae403e9b812a429b8960af"
I20240502 21:28:31.419116 32523 raft_consensus.cc:3037] T fae3d984510a4250bc7812d4387c513d P e0a4e8376dae403e9b812a429b8960af [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:31.421025 32523 raft_consensus.cc:2454] T fae3d984510a4250bc7812d4387c513d P e0a4e8376dae403e9b812a429b8960af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67a81d55b1bd4d2e9b587c6477d624a2 in term 1.
I20240502 21:28:31.421339 32059 leader_election.cc:302] T fae3d984510a4250bc7812d4387c513d P 67a81d55b1bd4d2e9b587c6477d624a2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 67a81d55b1bd4d2e9b587c6477d624a2, e0a4e8376dae403e9b812a429b8960af; no voters:
I20240502 21:28:31.421595 32594 raft_consensus.cc:2781] T fae3d984510a4250bc7812d4387c513d P 67a81d55b1bd4d2e9b587c6477d624a2 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:28:31.421969 32594 raft_consensus.cc:686] T fae3d984510a4250bc7812d4387c513d P 67a81d55b1bd4d2e9b587c6477d624a2 [term 1 LEADER]: Becoming Leader. State: Replica: 67a81d55b1bd4d2e9b587c6477d624a2, State: Running, Role: LEADER
I20240502 21:28:31.422401 32594 consensus_queue.cc:237] T fae3d984510a4250bc7812d4387c513d P 67a81d55b1bd4d2e9b587c6477d624a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0a4e8376dae403e9b812a429b8960af" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 42105 } } peers { permanent_uuid: "67a81d55b1bd4d2e9b587c6477d624a2" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40197 } }
I20240502 21:28:31.425124 31981 catalog_manager.cc:5484] T fae3d984510a4250bc7812d4387c513d P 67a81d55b1bd4d2e9b587c6477d624a2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 67a81d55b1bd4d2e9b587c6477d624a2 (127.15.255.129). New cstate: current_term: 1 leader_uuid: "67a81d55b1bd4d2e9b587c6477d624a2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0a4e8376dae403e9b812a429b8960af" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 42105 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "67a81d55b1bd4d2e9b587c6477d624a2" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40197 } health_report { overall_health: HEALTHY } } }
I20240502 21:28:31.441877 32594 raft_consensus.cc:482] T 0bf5f9db2efe49029fa1da7399ab5373 P 67a81d55b1bd4d2e9b587c6477d624a2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:28:31.442023 32594 raft_consensus.cc:504] T 0bf5f9db2efe49029fa1da7399ab5373 P 67a81d55b1bd4d2e9b587c6477d624a2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a81d55b1bd4d2e9b587c6477d624a2" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40197 } } peers { permanent_uuid: "e0a4e8376dae403e9b812a429b8960af" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 42105 } }
I20240502 21:28:31.442420 32594 leader_election.cc:288] T 0bf5f9db2efe49029fa1da7399ab5373 P 67a81d55b1bd4d2e9b587c6477d624a2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e0a4e8376dae403e9b812a429b8960af (127.15.255.132:42105)
I20240502 21:28:31.442840 32523 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bf5f9db2efe49029fa1da7399ab5373" candidate_uuid: "67a81d55b1bd4d2e9b587c6477d624a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0a4e8376dae403e9b812a429b8960af" is_pre_election: true
I20240502 21:28:31.443032 32523 raft_consensus.cc:2454] T 0bf5f9db2efe49029fa1da7399ab5373 P e0a4e8376dae403e9b812a429b8960af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67a81d55b1bd4d2e9b587c6477d624a2 in term 0.
I20240502 21:28:31.443377 32059 leader_election.cc:302] T 0bf5f9db2efe49029fa1da7399ab5373 P 67a81d55b1bd4d2e9b587c6477d624a2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 67a81d55b1bd4d2e9b587c6477d624a2, e0a4e8376dae403e9b812a429b8960af; no voters:
I20240502 21:28:31.443643 32594 raft_consensus.cc:2781] T 0bf5f9db2efe49029fa1da7399ab5373 P 67a81d55b1bd4d2e9b587c6477d624a2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:28:31.443728 32594 raft_consensus.cc:482] T 0bf5f9db2efe49029fa1da7399ab5373 P 67a81d55b1bd4d2e9b587c6477d624a2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:28:31.443805 32594 raft_consensus.cc:3037] T 0bf5f9db2efe49029fa1da7399ab5373 P 67a81d55b1bd4d2e9b587c6477d624a2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:31.445533 32594 raft_consensus.cc:504] T 0bf5f9db2efe49029fa1da7399ab5373 P 67a81d55b1bd4d2e9b587c6477d624a2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a81d55b1bd4d2e9b587c6477d624a2" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40197 } } peers { permanent_uuid: "e0a4e8376dae403e9b812a429b8960af" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 42105 } }
I20240502 21:28:31.445946 32594 leader_election.cc:288] T 0bf5f9db2efe49029fa1da7399ab5373 P 67a81d55b1bd4d2e9b587c6477d624a2 [CANDIDATE]: Term 1 election: Requested vote from peers e0a4e8376dae403e9b812a429b8960af (127.15.255.132:42105)
I20240502 21:28:31.446367 32523 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bf5f9db2efe49029fa1da7399ab5373" candidate_uuid: "67a81d55b1bd4d2e9b587c6477d624a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0a4e8376dae403e9b812a429b8960af"
I20240502 21:28:31.446537 32523 raft_consensus.cc:3037] T 0bf5f9db2efe49029fa1da7399ab5373 P e0a4e8376dae403e9b812a429b8960af [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:31.448206 32523 raft_consensus.cc:2454] T 0bf5f9db2efe49029fa1da7399ab5373 P e0a4e8376dae403e9b812a429b8960af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67a81d55b1bd4d2e9b587c6477d624a2 in term 1.
I20240502 21:28:31.448539 32059 leader_election.cc:302] T 0bf5f9db2efe49029fa1da7399ab5373 P 67a81d55b1bd4d2e9b587c6477d624a2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 67a81d55b1bd4d2e9b587c6477d624a2, e0a4e8376dae403e9b812a429b8960af; no voters:
I20240502 21:28:31.448767 32594 raft_consensus.cc:2781] T 0bf5f9db2efe49029fa1da7399ab5373 P 67a81d55b1bd4d2e9b587c6477d624a2 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:28:31.448913 32594 raft_consensus.cc:686] T 0bf5f9db2efe49029fa1da7399ab5373 P 67a81d55b1bd4d2e9b587c6477d624a2 [term 1 LEADER]: Becoming Leader. State: Replica: 67a81d55b1bd4d2e9b587c6477d624a2, State: Running, Role: LEADER
I20240502 21:28:31.449133 32594 consensus_queue.cc:237] T 0bf5f9db2efe49029fa1da7399ab5373 P 67a81d55b1bd4d2e9b587c6477d624a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a81d55b1bd4d2e9b587c6477d624a2" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40197 } } peers { permanent_uuid: "e0a4e8376dae403e9b812a429b8960af" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 42105 } }
I20240502 21:28:31.451248 31981 catalog_manager.cc:5484] T 0bf5f9db2efe49029fa1da7399ab5373 P 67a81d55b1bd4d2e9b587c6477d624a2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 67a81d55b1bd4d2e9b587c6477d624a2 (127.15.255.129). New cstate: current_term: 1 leader_uuid: "67a81d55b1bd4d2e9b587c6477d624a2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a81d55b1bd4d2e9b587c6477d624a2" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40197 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e0a4e8376dae403e9b812a429b8960af" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 42105 } health_report { overall_health: UNKNOWN } } }
I20240502 21:28:31.468782 32608 raft_consensus.cc:482] T 39d0ac5ec0cd4a33b947de380b5469be P e0a4e8376dae403e9b812a429b8960af [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:28:31.468937 32608 raft_consensus.cc:504] T 39d0ac5ec0cd4a33b947de380b5469be P e0a4e8376dae403e9b812a429b8960af [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0a4e8376dae403e9b812a429b8960af" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 42105 } } peers { permanent_uuid: "67a81d55b1bd4d2e9b587c6477d624a2" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40197 } }
I20240502 21:28:31.469362 32608 leader_election.cc:288] T 39d0ac5ec0cd4a33b947de380b5469be P e0a4e8376dae403e9b812a429b8960af [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 67a81d55b1bd4d2e9b587c6477d624a2 (127.15.255.129:40197)
I20240502 21:28:31.469835 32124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39d0ac5ec0cd4a33b947de380b5469be" candidate_uuid: "e0a4e8376dae403e9b812a429b8960af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67a81d55b1bd4d2e9b587c6477d624a2" is_pre_election: true
I20240502 21:28:31.470006 32124 raft_consensus.cc:2454] T 39d0ac5ec0cd4a33b947de380b5469be P 67a81d55b1bd4d2e9b587c6477d624a2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0a4e8376dae403e9b812a429b8960af in term 0.
I20240502 21:28:31.470402 32455 leader_election.cc:302] T 39d0ac5ec0cd4a33b947de380b5469be P e0a4e8376dae403e9b812a429b8960af [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 67a81d55b1bd4d2e9b587c6477d624a2, e0a4e8376dae403e9b812a429b8960af; no voters:
I20240502 21:28:31.470659 32608 raft_consensus.cc:2781] T 39d0ac5ec0cd4a33b947de380b5469be P e0a4e8376dae403e9b812a429b8960af [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:28:31.470741 32608 raft_consensus.cc:482] T 39d0ac5ec0cd4a33b947de380b5469be P e0a4e8376dae403e9b812a429b8960af [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:28:31.470839 32608 raft_consensus.cc:3037] T 39d0ac5ec0cd4a33b947de380b5469be P e0a4e8376dae403e9b812a429b8960af [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:31.472648 32608 raft_consensus.cc:504] T 39d0ac5ec0cd4a33b947de380b5469be P e0a4e8376dae403e9b812a429b8960af [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0a4e8376dae403e9b812a429b8960af" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 42105 } } peers { permanent_uuid: "67a81d55b1bd4d2e9b587c6477d624a2" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40197 } }
I20240502 21:28:31.473047 32608 leader_election.cc:288] T 39d0ac5ec0cd4a33b947de380b5469be P e0a4e8376dae403e9b812a429b8960af [CANDIDATE]: Term 1 election: Requested vote from peers 67a81d55b1bd4d2e9b587c6477d624a2 (127.15.255.129:40197)
I20240502 21:28:31.473425 32124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39d0ac5ec0cd4a33b947de380b5469be" candidate_uuid: "e0a4e8376dae403e9b812a429b8960af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67a81d55b1bd4d2e9b587c6477d624a2"
I20240502 21:28:31.473573 32124 raft_consensus.cc:3037] T 39d0ac5ec0cd4a33b947de380b5469be P 67a81d55b1bd4d2e9b587c6477d624a2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:31.475309 32124 raft_consensus.cc:2454] T 39d0ac5ec0cd4a33b947de380b5469be P 67a81d55b1bd4d2e9b587c6477d624a2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0a4e8376dae403e9b812a429b8960af in term 1.
I20240502 21:28:31.475636 32455 leader_election.cc:302] T 39d0ac5ec0cd4a33b947de380b5469be P e0a4e8376dae403e9b812a429b8960af [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 67a81d55b1bd4d2e9b587c6477d624a2, e0a4e8376dae403e9b812a429b8960af; no voters:
I20240502 21:28:31.475890 32608 raft_consensus.cc:2781] T 39d0ac5ec0cd4a33b947de380b5469be P e0a4e8376dae403e9b812a429b8960af [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:28:31.476040 32608 raft_consensus.cc:686] T 39d0ac5ec0cd4a33b947de380b5469be P e0a4e8376dae403e9b812a429b8960af [term 1 LEADER]: Becoming Leader. State: Replica: e0a4e8376dae403e9b812a429b8960af, State: Running, Role: LEADER
I20240502 21:28:31.476245 32608 consensus_queue.cc:237] T 39d0ac5ec0cd4a33b947de380b5469be P e0a4e8376dae403e9b812a429b8960af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0a4e8376dae403e9b812a429b8960af" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 42105 } } peers { permanent_uuid: "67a81d55b1bd4d2e9b587c6477d624a2" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40197 } }
I20240502 21:28:31.478274 31981 catalog_manager.cc:5484] T 39d0ac5ec0cd4a33b947de380b5469be P e0a4e8376dae403e9b812a429b8960af reported cstate change: term changed from 0 to 1, leader changed from <none> to e0a4e8376dae403e9b812a429b8960af (127.15.255.132). New cstate: current_term: 1 leader_uuid: "e0a4e8376dae403e9b812a429b8960af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0a4e8376dae403e9b812a429b8960af" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 42105 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "67a81d55b1bd4d2e9b587c6477d624a2" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40197 } health_report { overall_health: UNKNOWN } } }
W20240502 21:28:31.823869 32668 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:28:31.824129 32668 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:28:31.831290 32668 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'
I20240502 21:28:31.906946 32594 consensus_queue.cc:1035] T 0bf5f9db2efe49029fa1da7399ab5373 P 67a81d55b1bd4d2e9b587c6477d624a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0a4e8376dae403e9b812a429b8960af" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 42105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:28:31.969852 32523 raft_consensus.cc:1261] T fae3d984510a4250bc7812d4387c513d P e0a4e8376dae403e9b812a429b8960af [term 1 FOLLOWER]: Refusing update from remote peer 67a81d55b1bd4d2e9b587c6477d624a2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:28:31.970621 32594 consensus_queue.cc:1035] T fae3d984510a4250bc7812d4387c513d P 67a81d55b1bd4d2e9b587c6477d624a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0a4e8376dae403e9b812a429b8960af" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 42105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:28:31.976944 32124 raft_consensus.cc:1261] T 39d0ac5ec0cd4a33b947de380b5469be P 67a81d55b1bd4d2e9b587c6477d624a2 [term 1 FOLLOWER]: Refusing update from remote peer e0a4e8376dae403e9b812a429b8960af: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 21:28:31.977727 32608 consensus_queue.cc:1035] T 39d0ac5ec0cd4a33b947de380b5469be P e0a4e8376dae403e9b812a429b8960af [LEADER]: Connected to new peer: Peer: permanent_uuid: "67a81d55b1bd4d2e9b587c6477d624a2" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 197.49 ms
time per row: 0.0987449 ms
I20240502 21:28:32.163163 31981 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49812:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
W20240502 21:28:32.163924 31981 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table rebalance_test_table_2 in case of a server failure: 3 tablet servers would be needed, 2 are available. Consider bringing up more tablet servers.
I20240502 21:28:32.171821 32103 tablet_service.cc:1467] Processing CreateTablet for tablet 040ebecbd0d549a09a05e72534c78e4b (DEFAULT_TABLE table=rebalance_test_table_2 [id=b49ef2928b8449da9a95493239d775d2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:32.172353 32103 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 040ebecbd0d549a09a05e72534c78e4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:32.172299 32503 tablet_service.cc:1467] Processing CreateTablet for tablet 040ebecbd0d549a09a05e72534c78e4b (DEFAULT_TABLE table=rebalance_test_table_2 [id=b49ef2928b8449da9a95493239d775d2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:32.172441 32501 tablet_service.cc:1467] Processing CreateTablet for tablet 5b4ff662667a442cabfee1a1818d8dca (DEFAULT_TABLE table=rebalance_test_table_2 [id=b49ef2928b8449da9a95493239d775d2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:32.172853 32503 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 040ebecbd0d549a09a05e72534c78e4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:32.173200 32105 tablet_service.cc:1467] Processing CreateTablet for tablet 4ea23c1a76784237a5619fefa55335fc (DEFAULT_TABLE table=rebalance_test_table_2 [id=b49ef2928b8449da9a95493239d775d2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:32.173265 32104 tablet_service.cc:1467] Processing CreateTablet for tablet 5b4ff662667a442cabfee1a1818d8dca (DEFAULT_TABLE table=rebalance_test_table_2 [id=b49ef2928b8449da9a95493239d775d2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:32.173583 32105 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ea23c1a76784237a5619fefa55335fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:32.173993 32104 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b4ff662667a442cabfee1a1818d8dca. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:32.178951 32502 tablet_service.cc:1467] Processing CreateTablet for tablet 4ea23c1a76784237a5619fefa55335fc (DEFAULT_TABLE table=rebalance_test_table_2 [id=b49ef2928b8449da9a95493239d775d2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:32.179288 32502 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ea23c1a76784237a5619fefa55335fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:32.180543 32501 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b4ff662667a442cabfee1a1818d8dca. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:32.180737 32719 tablet_bootstrap.cc:492] T 040ebecbd0d549a09a05e72534c78e4b P 67a81d55b1bd4d2e9b587c6477d624a2: Bootstrap starting.
I20240502 21:28:32.182523 32719 tablet_bootstrap.cc:654] T 040ebecbd0d549a09a05e72534c78e4b P 67a81d55b1bd4d2e9b587c6477d624a2: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:32.187420 32719 tablet_bootstrap.cc:492] T 040ebecbd0d549a09a05e72534c78e4b P 67a81d55b1bd4d2e9b587c6477d624a2: No bootstrap required, opened a new log
I20240502 21:28:32.187536 32719 ts_tablet_manager.cc:1367] T 040ebecbd0d549a09a05e72534c78e4b P 67a81d55b1bd4d2e9b587c6477d624a2: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20240502 21:28:32.187757 32718 tablet_bootstrap.cc:492] T 040ebecbd0d549a09a05e72534c78e4b P e0a4e8376dae403e9b812a429b8960af: Bootstrap starting.
I20240502 21:28:32.188323 32719 raft_consensus.cc:348] T 040ebecbd0d549a09a05e72534c78e4b P 67a81d55b1bd4d2e9b587c6477d624a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a81d55b1bd4d2e9b587c6477d624a2" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40197 } } peers { permanent_uuid: "e0a4e8376dae403e9b812a429b8960af" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 42105 } }
I20240502 21:28:32.188560 32719 raft_consensus.cc:374] T 040ebecbd0d549a09a05e72534c78e4b P 67a81d55b1bd4d2e9b587c6477d624a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:32.188654 32719 raft_consensus.cc:729] T 040ebecbd0d549a09a05e72534c78e4b P 67a81d55b1bd4d2e9b587c6477d624a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67a81d55b1bd4d2e9b587c6477d624a2, State: Initialized, Role: FOLLOWER
I20240502 21:28:32.188874 32719 consensus_queue.cc:260] T 040ebecbd0d549a09a05e72534c78e4b P 67a81d55b1bd4d2e9b587c6477d624a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a81d55b1bd4d2e9b587c6477d624a2" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40197 } } peers { permanent_uuid: "e0a4e8376dae403e9b812a429b8960af" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 42105 } }
I20240502 21:28:32.189348 32718 tablet_bootstrap.cc:654] T 040ebecbd0d549a09a05e72534c78e4b P e0a4e8376dae403e9b812a429b8960af: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:32.189447 32719 ts_tablet_manager.cc:1398] T 040ebecbd0d549a09a05e72534c78e4b P 67a81d55b1bd4d2e9b587c6477d624a2: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 21:28:32.189726 32719 tablet_bootstrap.cc:492] T 4ea23c1a76784237a5619fefa55335fc P 67a81d55b1bd4d2e9b587c6477d624a2: Bootstrap starting.
I20240502 21:28:32.191120 32719 tablet_bootstrap.cc:654] T 4ea23c1a76784237a5619fefa55335fc P 67a81d55b1bd4d2e9b587c6477d624a2: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:32.191327 32718 tablet_bootstrap.cc:492] T 040ebecbd0d549a09a05e72534c78e4b P e0a4e8376dae403e9b812a429b8960af: No bootstrap required, opened a new log
I20240502 21:28:32.191449 32718 ts_tablet_manager.cc:1367] T 040ebecbd0d549a09a05e72534c78e4b P e0a4e8376dae403e9b812a429b8960af: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240502 21:28:32.192339 32718 raft_consensus.cc:348] T 040ebecbd0d549a09a05e72534c78e4b P e0a4e8376dae403e9b812a429b8960af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a81d55b1bd4d2e9b587c6477d624a2" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40197 } } peers { permanent_uuid: "e0a4e8376dae403e9b812a429b8960af" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 42105 } }
I20240502 21:28:32.192569 32718 raft_consensus.cc:374] T 040ebecbd0d549a09a05e72534c78e4b P e0a4e8376dae403e9b812a429b8960af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:32.192652 32718 raft_consensus.cc:729] T 040ebecbd0d549a09a05e72534c78e4b P e0a4e8376dae403e9b812a429b8960af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0a4e8376dae403e9b812a429b8960af, State: Initialized, Role: FOLLOWER
I20240502 21:28:32.192852 32718 consensus_queue.cc:260] T 040ebecbd0d549a09a05e72534c78e4b P e0a4e8376dae403e9b812a429b8960af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a81d55b1bd4d2e9b587c6477d624a2" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40197 } } peers { permanent_uuid: "e0a4e8376dae403e9b812a429b8960af" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 42105 } }
I20240502 21:28:32.193394 32718 ts_tablet_manager.cc:1398] T 040ebecbd0d549a09a05e72534c78e4b P e0a4e8376dae403e9b812a429b8960af: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:28:32.193646 32718 tablet_bootstrap.cc:492] T 4ea23c1a76784237a5619fefa55335fc P e0a4e8376dae403e9b812a429b8960af: Bootstrap starting.
I20240502 21:28:32.195093 32718 tablet_bootstrap.cc:654] T 4ea23c1a76784237a5619fefa55335fc P e0a4e8376dae403e9b812a429b8960af: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:32.195328 32719 tablet_bootstrap.cc:492] T 4ea23c1a76784237a5619fefa55335fc P 67a81d55b1bd4d2e9b587c6477d624a2: No bootstrap required, opened a new log
I20240502 21:28:32.195545 32719 ts_tablet_manager.cc:1367] T 4ea23c1a76784237a5619fefa55335fc P 67a81d55b1bd4d2e9b587c6477d624a2: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20240502 21:28:32.196345 32719 raft_consensus.cc:348] T 4ea23c1a76784237a5619fefa55335fc P 67a81d55b1bd4d2e9b587c6477d624a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0a4e8376dae403e9b812a429b8960af" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 42105 } } peers { permanent_uuid: "67a81d55b1bd4d2e9b587c6477d624a2" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40197 } }
I20240502 21:28:32.196578 32719 raft_consensus.cc:374] T 4ea23c1a76784237a5619fefa55335fc P 67a81d55b1bd4d2e9b587c6477d624a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:32.196651 32719 raft_consensus.cc:729] T 4ea23c1a76784237a5619fefa55335fc P 67a81d55b1bd4d2e9b587c6477d624a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67a81d55b1bd4d2e9b587c6477d624a2, State: Initialized, Role: FOLLOWER
I20240502 21:28:32.196862 32719 consensus_queue.cc:260] T 4ea23c1a76784237a5619fefa55335fc P 67a81d55b1bd4d2e9b587c6477d624a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0a4e8376dae403e9b812a429b8960af" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 42105 } } peers { permanent_uuid: "67a81d55b1bd4d2e9b587c6477d624a2" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40197 } }
I20240502 21:28:32.197363 32718 tablet_bootstrap.cc:492] T 4ea23c1a76784237a5619fefa55335fc P e0a4e8376dae403e9b812a429b8960af: No bootstrap required, opened a new log
I20240502 21:28:32.197371 32719 ts_tablet_manager.cc:1398] T 4ea23c1a76784237a5619fefa55335fc P 67a81d55b1bd4d2e9b587c6477d624a2: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 21:28:32.197515 32718 ts_tablet_manager.cc:1367] T 4ea23c1a76784237a5619fefa55335fc P e0a4e8376dae403e9b812a429b8960af: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240502 21:28:32.197639 32719 tablet_bootstrap.cc:492] T 5b4ff662667a442cabfee1a1818d8dca P 67a81d55b1bd4d2e9b587c6477d624a2: Bootstrap starting.
I20240502 21:28:32.198189 32718 raft_consensus.cc:348] T 4ea23c1a76784237a5619fefa55335fc P e0a4e8376dae403e9b812a429b8960af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0a4e8376dae403e9b812a429b8960af" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 42105 } } peers { permanent_uuid: "67a81d55b1bd4d2e9b587c6477d624a2" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40197 } }
I20240502 21:28:32.198379 32718 raft_consensus.cc:374] T 4ea23c1a76784237a5619fefa55335fc P e0a4e8376dae403e9b812a429b8960af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:32.198457 32718 raft_consensus.cc:729] T 4ea23c1a76784237a5619fefa55335fc P e0a4e8376dae403e9b812a429b8960af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0a4e8376dae403e9b812a429b8960af, State: Initialized, Role: FOLLOWER
I20240502 21:28:32.198691 32718 consensus_queue.cc:260] T 4ea23c1a76784237a5619fefa55335fc P e0a4e8376dae403e9b812a429b8960af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0a4e8376dae403e9b812a429b8960af" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 42105 } } peers { permanent_uuid: "67a81d55b1bd4d2e9b587c6477d624a2" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40197 } }
I20240502 21:28:32.199033 32719 tablet_bootstrap.cc:654] T 5b4ff662667a442cabfee1a1818d8dca P 67a81d55b1bd4d2e9b587c6477d624a2: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:32.199201 32718 ts_tablet_manager.cc:1398] T 4ea23c1a76784237a5619fefa55335fc P e0a4e8376dae403e9b812a429b8960af: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:28:32.199489 32718 tablet_bootstrap.cc:492] T 5b4ff662667a442cabfee1a1818d8dca P e0a4e8376dae403e9b812a429b8960af: Bootstrap starting.
I20240502 21:28:32.200933 32718 tablet_bootstrap.cc:654] T 5b4ff662667a442cabfee1a1818d8dca P e0a4e8376dae403e9b812a429b8960af: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:32.201274 32719 tablet_bootstrap.cc:492] T 5b4ff662667a442cabfee1a1818d8dca P 67a81d55b1bd4d2e9b587c6477d624a2: No bootstrap required, opened a new log
I20240502 21:28:32.201618 32719 ts_tablet_manager.cc:1367] T 5b4ff662667a442cabfee1a1818d8dca P 67a81d55b1bd4d2e9b587c6477d624a2: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240502 21:28:32.202383 32719 raft_consensus.cc:348] T 5b4ff662667a442cabfee1a1818d8dca P 67a81d55b1bd4d2e9b587c6477d624a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0a4e8376dae403e9b812a429b8960af" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 42105 } } peers { permanent_uuid: "67a81d55b1bd4d2e9b587c6477d624a2" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40197 } }
I20240502 21:28:32.202556 32719 raft_consensus.cc:374] T 5b4ff662667a442cabfee1a1818d8dca P 67a81d55b1bd4d2e9b587c6477d624a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:32.202656 32719 raft_consensus.cc:729] T 5b4ff662667a442cabfee1a1818d8dca P 67a81d55b1bd4d2e9b587c6477d624a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67a81d55b1bd4d2e9b587c6477d624a2, State: Initialized, Role: FOLLOWER
I20240502 21:28:32.202873 32719 consensus_queue.cc:260] T 5b4ff662667a442cabfee1a1818d8dca P 67a81d55b1bd4d2e9b587c6477d624a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0a4e8376dae403e9b812a429b8960af" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 42105 } } peers { permanent_uuid: "67a81d55b1bd4d2e9b587c6477d624a2" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40197 } }
I20240502 21:28:32.203153 32718 tablet_bootstrap.cc:492] T 5b4ff662667a442cabfee1a1818d8dca P e0a4e8376dae403e9b812a429b8960af: No bootstrap required, opened a new log
I20240502 21:28:32.203307 32718 ts_tablet_manager.cc:1367] T 5b4ff662667a442cabfee1a1818d8dca P e0a4e8376dae403e9b812a429b8960af: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240502 21:28:32.203464 32719 ts_tablet_manager.cc:1398] T 5b4ff662667a442cabfee1a1818d8dca P 67a81d55b1bd4d2e9b587c6477d624a2: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 21:28:32.203923 32718 raft_consensus.cc:348] T 5b4ff662667a442cabfee1a1818d8dca P e0a4e8376dae403e9b812a429b8960af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0a4e8376dae403e9b812a429b8960af" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 42105 } } peers { permanent_uuid: "67a81d55b1bd4d2e9b587c6477d624a2" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40197 } }
I20240502 21:28:32.204066 32718 raft_consensus.cc:374] T 5b4ff662667a442cabfee1a1818d8dca P e0a4e8376dae403e9b812a429b8960af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:32.204156 32718 raft_consensus.cc:729] T 5b4ff662667a442cabfee1a1818d8dca P e0a4e8376dae403e9b812a429b8960af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0a4e8376dae403e9b812a429b8960af, State: Initialized, Role: FOLLOWER
I20240502 21:28:32.204345 32718 consensus_queue.cc:260] T 5b4ff662667a442cabfee1a1818d8dca P e0a4e8376dae403e9b812a429b8960af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0a4e8376dae403e9b812a429b8960af" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 42105 } } peers { permanent_uuid: "67a81d55b1bd4d2e9b587c6477d624a2" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40197 } }
I20240502 21:28:32.204849 32718 ts_tablet_manager.cc:1398] T 5b4ff662667a442cabfee1a1818d8dca P e0a4e8376dae403e9b812a429b8960af: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 21:28:32.219827 32608 raft_consensus.cc:482] T 4ea23c1a76784237a5619fefa55335fc P e0a4e8376dae403e9b812a429b8960af [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:28:32.219980 32608 raft_consensus.cc:504] T 4ea23c1a76784237a5619fefa55335fc P e0a4e8376dae403e9b812a429b8960af [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0a4e8376dae403e9b812a429b8960af" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 42105 } } peers { permanent_uuid: "67a81d55b1bd4d2e9b587c6477d624a2" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40197 } }
I20240502 21:28:32.220378 32608 leader_election.cc:288] T 4ea23c1a76784237a5619fefa55335fc P e0a4e8376dae403e9b812a429b8960af [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 67a81d55b1bd4d2e9b587c6477d624a2 (127.15.255.129:40197)
I20240502 21:28:32.220809 32124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ea23c1a76784237a5619fefa55335fc" candidate_uuid: "e0a4e8376dae403e9b812a429b8960af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67a81d55b1bd4d2e9b587c6477d624a2" is_pre_election: true
I20240502 21:28:32.220980 32124 raft_consensus.cc:2454] T 4ea23c1a76784237a5619fefa55335fc P 67a81d55b1bd4d2e9b587c6477d624a2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0a4e8376dae403e9b812a429b8960af in term 0.
I20240502 21:28:32.221321 32455 leader_election.cc:302] T 4ea23c1a76784237a5619fefa55335fc P e0a4e8376dae403e9b812a429b8960af [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 67a81d55b1bd4d2e9b587c6477d624a2, e0a4e8376dae403e9b812a429b8960af; no voters:
I20240502 21:28:32.221575 32608 raft_consensus.cc:2781] T 4ea23c1a76784237a5619fefa55335fc P e0a4e8376dae403e9b812a429b8960af [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:28:32.221654 32608 raft_consensus.cc:482] T 4ea23c1a76784237a5619fefa55335fc P e0a4e8376dae403e9b812a429b8960af [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:28:32.221748 32608 raft_consensus.cc:3037] T 4ea23c1a76784237a5619fefa55335fc P e0a4e8376dae403e9b812a429b8960af [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:32.223448 32608 raft_consensus.cc:504] T 4ea23c1a76784237a5619fefa55335fc P e0a4e8376dae403e9b812a429b8960af [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0a4e8376dae403e9b812a429b8960af" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 42105 } } peers { permanent_uuid: "67a81d55b1bd4d2e9b587c6477d624a2" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40197 } }
I20240502 21:28:32.223839 32608 leader_election.cc:288] T 4ea23c1a76784237a5619fefa55335fc P e0a4e8376dae403e9b812a429b8960af [CANDIDATE]: Term 1 election: Requested vote from peers 67a81d55b1bd4d2e9b587c6477d624a2 (127.15.255.129:40197)
I20240502 21:28:32.224220 32124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ea23c1a76784237a5619fefa55335fc" candidate_uuid: "e0a4e8376dae403e9b812a429b8960af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67a81d55b1bd4d2e9b587c6477d624a2"
I20240502 21:28:32.224376 32124 raft_consensus.cc:3037] T 4ea23c1a76784237a5619fefa55335fc P 67a81d55b1bd4d2e9b587c6477d624a2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:32.226097 32124 raft_consensus.cc:2454] T 4ea23c1a76784237a5619fefa55335fc P 67a81d55b1bd4d2e9b587c6477d624a2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0a4e8376dae403e9b812a429b8960af in term 1.
I20240502 21:28:32.226449 32455 leader_election.cc:302] T 4ea23c1a76784237a5619fefa55335fc P e0a4e8376dae403e9b812a429b8960af [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 67a81d55b1bd4d2e9b587c6477d624a2, e0a4e8376dae403e9b812a429b8960af; no voters:
I20240502 21:28:32.226675 32608 raft_consensus.cc:2781] T 4ea23c1a76784237a5619fefa55335fc P e0a4e8376dae403e9b812a429b8960af [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:28:32.226814 32608 raft_consensus.cc:686] T 4ea23c1a76784237a5619fefa55335fc P e0a4e8376dae403e9b812a429b8960af [term 1 LEADER]: Becoming Leader. State: Replica: e0a4e8376dae403e9b812a429b8960af, State: Running, Role: LEADER
I20240502 21:28:32.227023 32608 consensus_queue.cc:237] T 4ea23c1a76784237a5619fefa55335fc P e0a4e8376dae403e9b812a429b8960af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0a4e8376dae403e9b812a429b8960af" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 42105 } } peers { permanent_uuid: "67a81d55b1bd4d2e9b587c6477d624a2" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40197 } }
I20240502 21:28:32.229136 31980 catalog_manager.cc:5484] T 4ea23c1a76784237a5619fefa55335fc P e0a4e8376dae403e9b812a429b8960af reported cstate change: term changed from 0 to 1, leader changed from <none> to e0a4e8376dae403e9b812a429b8960af (127.15.255.132). New cstate: current_term: 1 leader_uuid: "e0a4e8376dae403e9b812a429b8960af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0a4e8376dae403e9b812a429b8960af" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 42105 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "67a81d55b1bd4d2e9b587c6477d624a2" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40197 } health_report { overall_health: UNKNOWN } } }
I20240502 21:28:32.433611 32706 raft_consensus.cc:482] T 040ebecbd0d549a09a05e72534c78e4b P 67a81d55b1bd4d2e9b587c6477d624a2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:28:32.433773 32706 raft_consensus.cc:504] T 040ebecbd0d549a09a05e72534c78e4b P 67a81d55b1bd4d2e9b587c6477d624a2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a81d55b1bd4d2e9b587c6477d624a2" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40197 } } peers { permanent_uuid: "e0a4e8376dae403e9b812a429b8960af" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 42105 } }
I20240502 21:28:32.434315 32706 leader_election.cc:288] T 040ebecbd0d549a09a05e72534c78e4b P 67a81d55b1bd4d2e9b587c6477d624a2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e0a4e8376dae403e9b812a429b8960af (127.15.255.132:42105)
I20240502 21:28:32.434815 32523 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "040ebecbd0d549a09a05e72534c78e4b" candidate_uuid: "67a81d55b1bd4d2e9b587c6477d624a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0a4e8376dae403e9b812a429b8960af" is_pre_election: true
I20240502 21:28:32.435031 32523 raft_consensus.cc:2454] T 040ebecbd0d549a09a05e72534c78e4b P e0a4e8376dae403e9b812a429b8960af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67a81d55b1bd4d2e9b587c6477d624a2 in term 0.
I20240502 21:28:32.435441 32059 leader_election.cc:302] T 040ebecbd0d549a09a05e72534c78e4b P 67a81d55b1bd4d2e9b587c6477d624a2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 67a81d55b1bd4d2e9b587c6477d624a2, e0a4e8376dae403e9b812a429b8960af; no voters:
I20240502 21:28:32.435708 32706 raft_consensus.cc:2781] T 040ebecbd0d549a09a05e72534c78e4b P 67a81d55b1bd4d2e9b587c6477d624a2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:28:32.435819 32706 raft_consensus.cc:482] T 040ebecbd0d549a09a05e72534c78e4b P 67a81d55b1bd4d2e9b587c6477d624a2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:28:32.435896 32706 raft_consensus.cc:3037] T 040ebecbd0d549a09a05e72534c78e4b P 67a81d55b1bd4d2e9b587c6477d624a2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:32.437754 32706 raft_consensus.cc:504] T 040ebecbd0d549a09a05e72534c78e4b P 67a81d55b1bd4d2e9b587c6477d624a2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a81d55b1bd4d2e9b587c6477d624a2" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40197 } } peers { permanent_uuid: "e0a4e8376dae403e9b812a429b8960af" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 42105 } }
I20240502 21:28:32.438150 32706 leader_election.cc:288] T 040ebecbd0d549a09a05e72534c78e4b P 67a81d55b1bd4d2e9b587c6477d624a2 [CANDIDATE]: Term 1 election: Requested vote from peers e0a4e8376dae403e9b812a429b8960af (127.15.255.132:42105)
I20240502 21:28:32.438586 32523 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "040ebecbd0d549a09a05e72534c78e4b" candidate_uuid: "67a81d55b1bd4d2e9b587c6477d624a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0a4e8376dae403e9b812a429b8960af"
I20240502 21:28:32.438772 32523 raft_consensus.cc:3037] T 040ebecbd0d549a09a05e72534c78e4b P e0a4e8376dae403e9b812a429b8960af [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:32.440533 32523 raft_consensus.cc:2454] T 040ebecbd0d549a09a05e72534c78e4b P e0a4e8376dae403e9b812a429b8960af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67a81d55b1bd4d2e9b587c6477d624a2 in term 1.
I20240502 21:28:32.440843 32059 leader_election.cc:302] T 040ebecbd0d549a09a05e72534c78e4b P 67a81d55b1bd4d2e9b587c6477d624a2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 67a81d55b1bd4d2e9b587c6477d624a2, e0a4e8376dae403e9b812a429b8960af; no voters:
I20240502 21:28:32.441085 32706 raft_consensus.cc:2781] T 040ebecbd0d549a09a05e72534c78e4b P 67a81d55b1bd4d2e9b587c6477d624a2 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:28:32.441224 32706 raft_consensus.cc:686] T 040ebecbd0d549a09a05e72534c78e4b P 67a81d55b1bd4d2e9b587c6477d624a2 [term 1 LEADER]: Becoming Leader. State: Replica: 67a81d55b1bd4d2e9b587c6477d624a2, State: Running, Role: LEADER
I20240502 21:28:32.441473 32706 consensus_queue.cc:237] T 040ebecbd0d549a09a05e72534c78e4b P 67a81d55b1bd4d2e9b587c6477d624a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a81d55b1bd4d2e9b587c6477d624a2" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40197 } } peers { permanent_uuid: "e0a4e8376dae403e9b812a429b8960af" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 42105 } }
I20240502 21:28:32.443764 31980 catalog_manager.cc:5484] T 040ebecbd0d549a09a05e72534c78e4b P 67a81d55b1bd4d2e9b587c6477d624a2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 67a81d55b1bd4d2e9b587c6477d624a2 (127.15.255.129). New cstate: current_term: 1 leader_uuid: "67a81d55b1bd4d2e9b587c6477d624a2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "67a81d55b1bd4d2e9b587c6477d624a2" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40197 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e0a4e8376dae403e9b812a429b8960af" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 42105 } health_report { overall_health: UNKNOWN } } }
I20240502 21:28:32.532184 32706 raft_consensus.cc:482] T 5b4ff662667a442cabfee1a1818d8dca P 67a81d55b1bd4d2e9b587c6477d624a2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:28:32.532366 32706 raft_consensus.cc:504] T 5b4ff662667a442cabfee1a1818d8dca P 67a81d55b1bd4d2e9b587c6477d624a2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0a4e8376dae403e9b812a429b8960af" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 42105 } } peers { permanent_uuid: "67a81d55b1bd4d2e9b587c6477d624a2" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40197 } }
I20240502 21:28:32.532862 32706 leader_election.cc:288] T 5b4ff662667a442cabfee1a1818d8dca P 67a81d55b1bd4d2e9b587c6477d624a2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e0a4e8376dae403e9b812a429b8960af (127.15.255.132:42105)
I20240502 21:28:32.533375 32523 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b4ff662667a442cabfee1a1818d8dca" candidate_uuid: "67a81d55b1bd4d2e9b587c6477d624a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0a4e8376dae403e9b812a429b8960af" is_pre_election: true
I20240502 21:28:32.533566 32523 raft_consensus.cc:2454] T 5b4ff662667a442cabfee1a1818d8dca P e0a4e8376dae403e9b812a429b8960af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67a81d55b1bd4d2e9b587c6477d624a2 in term 0.
I20240502 21:28:32.533953 32059 leader_election.cc:302] T 5b4ff662667a442cabfee1a1818d8dca P 67a81d55b1bd4d2e9b587c6477d624a2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 67a81d55b1bd4d2e9b587c6477d624a2, e0a4e8376dae403e9b812a429b8960af; no voters:
I20240502 21:28:32.534201 32706 raft_consensus.cc:2781] T 5b4ff662667a442cabfee1a1818d8dca P 67a81d55b1bd4d2e9b587c6477d624a2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:28:32.534291 32706 raft_consensus.cc:482] T 5b4ff662667a442cabfee1a1818d8dca P 67a81d55b1bd4d2e9b587c6477d624a2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:28:32.534404 32706 raft_consensus.cc:3037] T 5b4ff662667a442cabfee1a1818d8dca P 67a81d55b1bd4d2e9b587c6477d624a2 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:32.536412 32706 raft_consensus.cc:504] T 5b4ff662667a442cabfee1a1818d8dca P 67a81d55b1bd4d2e9b587c6477d624a2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0a4e8376dae403e9b812a429b8960af" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 42105 } } peers { permanent_uuid: "67a81d55b1bd4d2e9b587c6477d624a2" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40197 } }
I20240502 21:28:32.536880 32706 leader_election.cc:288] T 5b4ff662667a442cabfee1a1818d8dca P 67a81d55b1bd4d2e9b587c6477d624a2 [CANDIDATE]: Term 1 election: Requested vote from peers e0a4e8376dae403e9b812a429b8960af (127.15.255.132:42105)
I20240502 21:28:32.537283 32522 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b4ff662667a442cabfee1a1818d8dca" candidate_uuid: "67a81d55b1bd4d2e9b587c6477d624a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0a4e8376dae403e9b812a429b8960af"
I20240502 21:28:32.537444 32522 raft_consensus.cc:3037] T 5b4ff662667a442cabfee1a1818d8dca P e0a4e8376dae403e9b812a429b8960af [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:32.539122 32522 raft_consensus.cc:2454] T 5b4ff662667a442cabfee1a1818d8dca P e0a4e8376dae403e9b812a429b8960af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67a81d55b1bd4d2e9b587c6477d624a2 in term 1.
I20240502 21:28:32.539453 32059 leader_election.cc:302] T 5b4ff662667a442cabfee1a1818d8dca P 67a81d55b1bd4d2e9b587c6477d624a2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 67a81d55b1bd4d2e9b587c6477d624a2, e0a4e8376dae403e9b812a429b8960af; no voters:
I20240502 21:28:32.539746 32706 raft_consensus.cc:2781] T 5b4ff662667a442cabfee1a1818d8dca P 67a81d55b1bd4d2e9b587c6477d624a2 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:28:32.539896 32706 raft_consensus.cc:686] T 5b4ff662667a442cabfee1a1818d8dca P 67a81d55b1bd4d2e9b587c6477d624a2 [term 1 LEADER]: Becoming Leader. State: Replica: 67a81d55b1bd4d2e9b587c6477d624a2, State: Running, Role: LEADER
I20240502 21:28:32.540115 32706 consensus_queue.cc:237] T 5b4ff662667a442cabfee1a1818d8dca P 67a81d55b1bd4d2e9b587c6477d624a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0a4e8376dae403e9b812a429b8960af" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 42105 } } peers { permanent_uuid: "67a81d55b1bd4d2e9b587c6477d624a2" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40197 } }
I20240502 21:28:32.542165 31980 catalog_manager.cc:5484] T 5b4ff662667a442cabfee1a1818d8dca P 67a81d55b1bd4d2e9b587c6477d624a2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 67a81d55b1bd4d2e9b587c6477d624a2 (127.15.255.129). New cstate: current_term: 1 leader_uuid: "67a81d55b1bd4d2e9b587c6477d624a2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0a4e8376dae403e9b812a429b8960af" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 42105 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "67a81d55b1bd4d2e9b587c6477d624a2" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40197 } health_report { overall_health: HEALTHY } } }
I20240502 21:28:32.808665 32608 consensus_queue.cc:1035] T 4ea23c1a76784237a5619fefa55335fc P e0a4e8376dae403e9b812a429b8960af [LEADER]: Connected to new peer: Peer: permanent_uuid: "67a81d55b1bd4d2e9b587c6477d624a2" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:28:32.926970 32706 consensus_queue.cc:1035] T 040ebecbd0d549a09a05e72534c78e4b P 67a81d55b1bd4d2e9b587c6477d624a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0a4e8376dae403e9b812a429b8960af" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 42105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 21:28:32.946496 32730 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:28:32.946784 32730 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:28:32.954123 32730 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_2'
I20240502 21:28:33.049304 32594 consensus_queue.cc:1035] T 5b4ff662667a442cabfee1a1818d8dca P 67a81d55b1bd4d2e9b587c6477d624a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0a4e8376dae403e9b812a429b8960af" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 42105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 159.919 ms
time per row: 0.0799597 ms
I20240502 21:28:33.247014 16382 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvNciQZ/build/asan/bin/kudu
/tmp/dist-test-taskvNciQZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.255.130:34399
--local_ip_for_outbound_sockets=127.15.255.130
--tserver_master_addrs=127.15.255.190:34507
--webserver_port=42967
--webserver_interface=127.15.255.130
--builtin_ntp_servers=127.15.255.148:45063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240502 21:28:33.565960 311 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:28:33.566196 311 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:28:33.566287 311 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:28:33.573388 311 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 21:28:33.573489 311 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:28:33.573597 311 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.255.130
I20240502 21:28:33.581955 311 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.255.148:45063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.255.130:34399
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.15.255.130
--webserver_port=42967
--tserver_master_addrs=127.15.255.190:34507
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.255.130
--log_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:28:33.583112 311 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:28:33.584784 311 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:28:33.596168 316 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:28:33.596685 317 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:28:33.597260 319 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:28:33.597909 311 server_base.cc:1034] running on GCE node
I20240502 21:28:33.598413 311 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:28:33.599221 311 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:28:33.600430 311 hybrid_clock.cc:648] HybridClock initialized: now 1714685313600378 us; error 25 us; skew 500 ppm
I20240502 21:28:33.600718 311 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:28:33.606942 311 webserver.cc:460] Webserver started at http://127.15.255.130:42967/ using document root <none> and password file <none>
I20240502 21:28:33.607653 311 fs_manager.cc:359] Metadata directory not provided
I20240502 21:28:33.607774 311 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:28:33.612604 311 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20240502 21:28:33.614996 325 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:28:33.616176 311 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 21:28:33.616318 311 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "c9ab11aee0734f048a27a267e9a34ee4"
format_stamp: "Formatted at 2024-05-02 21:28:25 on dist-test-slave-jdfx"
I20240502 21:28:33.617103 311 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:28:33.649147 311 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:28:33.649995 311 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:28:33.650261 311 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:28:33.651178 311 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:28:33.652706 311 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:28:33.652782 311 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:28:33.652895 311 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:28:33.652944 311 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 21:28:33.680507 311 rpc_server.cc:307] RPC server started. Bound to: 127.15.255.130:34399
I20240502 21:28:33.680557 439 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.255.130:34399 every 8 connection(s)
I20240502 21:28:33.681835 311 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240502 21:28:33.686316 16382 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvNciQZ/build/asan/bin/kudu as pid 311
I20240502 21:28:33.686909 16382 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvNciQZ/build/asan/bin/kudu
/tmp/dist-test-taskvNciQZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.255.131:45205
--local_ip_for_outbound_sockets=127.15.255.131
--tserver_master_addrs=127.15.255.190:34507
--webserver_port=42885
--webserver_interface=127.15.255.131
--builtin_ntp_servers=127.15.255.148:45063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 21:28:33.693828 440 heartbeater.cc:347] Connected to a master server at 127.15.255.190:34507
I20240502 21:28:33.694096 440 heartbeater.cc:464] Registering TS with master...
I20240502 21:28:33.694820 440 heartbeater.cc:511] Master 127.15.255.190:34507 requested a full tablet report, sending...
I20240502 21:28:33.696331 31980 ts_manager.cc:194] Re-registered known tserver with Master: c9ab11aee0734f048a27a267e9a34ee4 (127.15.255.130:34399)
I20240502 21:28:33.697190 31980 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.255.130:60681
W20240502 21:28:34.004060 444 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:28:34.004392 444 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:28:34.004597 444 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:28:34.011993 444 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 21:28:34.012132 444 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:28:34.012272 444 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.255.131
I20240502 21:28:34.021167 444 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.255.148:45063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.255.131:45205
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.15.255.131
--webserver_port=42885
--tserver_master_addrs=127.15.255.190:34507
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.255.131
--log_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:28:34.022364 444 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:28:34.023978 444 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:28:34.035449 449 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:28:34.035843 450 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:28:34.036475 452 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:28:34.037104 444 server_base.cc:1034] running on GCE node
I20240502 21:28:34.037567 444 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:28:34.038336 444 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:28:34.039515 444 hybrid_clock.cc:648] HybridClock initialized: now 1714685314039471 us; error 36 us; skew 500 ppm
I20240502 21:28:34.039770 444 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:28:34.041712 444 webserver.cc:460] Webserver started at http://127.15.255.131:42885/ using document root <none> and password file <none>
I20240502 21:28:34.042327 444 fs_manager.cc:359] Metadata directory not provided
I20240502 21:28:34.042467 444 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:28:34.047132 444 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20240502 21:28:34.049525 458 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:28:34.050678 444 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240502 21:28:34.050808 444 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "c38cec78fdf74d05b3c75f21eeb8612c"
format_stamp: "Formatted at 2024-05-02 21:28:25 on dist-test-slave-jdfx"
I20240502 21:28:34.051554 444 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:28:34.074474 444 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:28:34.075106 444 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:28:34.075331 444 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:28:34.076104 444 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:28:34.077525 444 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:28:34.077595 444 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:28:34.077683 444 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:28:34.077724 444 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 21:28:34.105823 444 rpc_server.cc:307] RPC server started. Bound to: 127.15.255.131:45205
I20240502 21:28:34.105870 571 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.255.131:45205 every 8 connection(s)
I20240502 21:28:34.107210 444 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240502 21:28:34.107851 16382 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvNciQZ/build/asan/bin/kudu as pid 444
I20240502 21:28:34.119922 572 heartbeater.cc:347] Connected to a master server at 127.15.255.190:34507
I20240502 21:28:34.120187 572 heartbeater.cc:464] Registering TS with master...
I20240502 21:28:34.120966 572 heartbeater.cc:511] Master 127.15.255.190:34507 requested a full tablet report, sending...
I20240502 21:28:34.122359 31980 ts_manager.cc:194] Re-registered known tserver with Master: c38cec78fdf74d05b3c75f21eeb8612c (127.15.255.131:45205)
I20240502 21:28:34.123278 31980 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.255.131:33421
I20240502 21:28:34.564113 32104 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:28:34.569628 506 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:28:34.575125 374 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:28:34.586951 32503 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:28:34.699342 440 heartbeater.cc:503] Master 127.15.255.190:34507 was elected leader, sending a full tablet report...
I20240502 21:28:34.721230 506 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:28:34.721628 32104 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:28:34.735756 374 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:28:34.740005 32503 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:28:34.895025 32104 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:28:34.895757 506 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:28:34.910530 374 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:28:34.917982 32503 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:28:35.051955 32104 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:28:35.052798 374 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:28:35.052820 506 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:28:35.071632 32503 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:28:35.125509 572 heartbeater.cc:503] Master 127.15.255.190:34507 was elected leader, sending a full tablet report...
I20240502 21:28:35.130029 32124 consensus_queue.cc:237] T 5b4ff662667a442cabfee1a1818d8dca P 67a81d55b1bd4d2e9b587c6477d624a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e0a4e8376dae403e9b812a429b8960af" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 42105 } attrs { replace: true } } peers { permanent_uuid: "67a81d55b1bd4d2e9b587c6477d624a2" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40197 } } peers { permanent_uuid: "c38cec78fdf74d05b3c75f21eeb8612c" member_type: NON_VOTER last_known_addr { host: "127.15.255.131" port: 45205 } attrs { promote: true } }
I20240502 21:28:35.132663 32523 raft_consensus.cc:1261] T 5b4ff662667a442cabfee1a1818d8dca P e0a4e8376dae403e9b812a429b8960af [term 1 FOLLOWER]: Refusing update from remote peer 67a81d55b1bd4d2e9b587c6477d624a2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:28:35.133240 708 consensus_queue.cc:1035] T 5b4ff662667a442cabfee1a1818d8dca P 67a81d55b1bd4d2e9b587c6477d624a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0a4e8376dae403e9b812a429b8960af" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 42105 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:28:35.136026 305 raft_consensus.cc:2932] T 5b4ff662667a442cabfee1a1818d8dca P 67a81d55b1bd4d2e9b587c6477d624a2 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c38cec78fdf74d05b3c75f21eeb8612c (127.15.255.131) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e0a4e8376dae403e9b812a429b8960af" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 42105 } attrs { replace: true } } peers { permanent_uuid: "67a81d55b1bd4d2e9b587c6477d624a2" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40197 } } peers { permanent_uuid: "c38cec78fdf74d05b3c75f21eeb8612c" member_type: NON_VOTER last_known_addr { host: "127.15.255.131" port: 45205 } attrs { promote: true } } }
W20240502 21:28:35.138805 32058 consensus_peers.cc:485] T 5b4ff662667a442cabfee1a1818d8dca P 67a81d55b1bd4d2e9b587c6477d624a2 -> Peer c38cec78fdf74d05b3c75f21eeb8612c (127.15.255.131:45205): Couldn't send request to peer c38cec78fdf74d05b3c75f21eeb8612c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5b4ff662667a442cabfee1a1818d8dca. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:28:35.140202 32523 raft_consensus.cc:2932] T 5b4ff662667a442cabfee1a1818d8dca P e0a4e8376dae403e9b812a429b8960af [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c38cec78fdf74d05b3c75f21eeb8612c (127.15.255.131) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e0a4e8376dae403e9b812a429b8960af" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 42105 } attrs { replace: true } } peers { permanent_uuid: "67a81d55b1bd4d2e9b587c6477d624a2" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40197 } } peers { permanent_uuid: "c38cec78fdf74d05b3c75f21eeb8612c" member_type: NON_VOTER last_known_addr { host: "127.15.255.131" port: 45205 } attrs { promote: true } } }
I20240502 21:28:35.142863 31980 catalog_manager.cc:5484] T 5b4ff662667a442cabfee1a1818d8dca P 67a81d55b1bd4d2e9b587c6477d624a2 reported cstate change: config changed from index -1 to 4, NON_VOTER c38cec78fdf74d05b3c75f21eeb8612c (127.15.255.131) added. New cstate: current_term: 1 leader_uuid: "67a81d55b1bd4d2e9b587c6477d624a2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e0a4e8376dae403e9b812a429b8960af" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 42105 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "67a81d55b1bd4d2e9b587c6477d624a2" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40197 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c38cec78fdf74d05b3c75f21eeb8612c" member_type: NON_VOTER last_known_addr { host: "127.15.255.131" port: 45205 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 21:28:35.154155 32523 consensus_queue.cc:237] T 39d0ac5ec0cd4a33b947de380b5469be P e0a4e8376dae403e9b812a429b8960af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e0a4e8376dae403e9b812a429b8960af" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 42105 } } peers { permanent_uuid: "67a81d55b1bd4d2e9b587c6477d624a2" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40197 } attrs { replace: true } } peers { permanent_uuid: "c9ab11aee0734f048a27a267e9a34ee4" member_type: NON_VOTER last_known_addr { host: "127.15.255.130" port: 34399 } attrs { promote: true } }
I20240502 21:28:35.156178 32124 raft_consensus.cc:1261] T 39d0ac5ec0cd4a33b947de380b5469be P 67a81d55b1bd4d2e9b587c6477d624a2 [term 1 FOLLOWER]: Refusing update from remote peer e0a4e8376dae403e9b812a429b8960af: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:28:35.156739 32732 consensus_queue.cc:1035] T 39d0ac5ec0cd4a33b947de380b5469be P e0a4e8376dae403e9b812a429b8960af [LEADER]: Connected to new peer: Peer: permanent_uuid: "67a81d55b1bd4d2e9b587c6477d624a2" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40197 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:28:35.159436 723 raft_consensus.cc:2932] T 39d0ac5ec0cd4a33b947de380b5469be P e0a4e8376dae403e9b812a429b8960af [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c9ab11aee0734f048a27a267e9a34ee4 (127.15.255.130) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e0a4e8376dae403e9b812a429b8960af" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 42105 } } peers { permanent_uuid: "67a81d55b1bd4d2e9b587c6477d624a2" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40197 } attrs { replace: true } } peers { permanent_uuid: "c9ab11aee0734f048a27a267e9a34ee4" member_type: NON_VOTER last_known_addr { host: "127.15.255.130" port: 34399 } attrs { promote: true } } }
I20240502 21:28:35.159967 32124 raft_consensus.cc:2932] T 39d0ac5ec0cd4a33b947de380b5469be P 67a81d55b1bd4d2e9b587c6477d624a2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c9ab11aee0734f048a27a267e9a34ee4 (127.15.255.130) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e0a4e8376dae403e9b812a429b8960af" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 42105 } } peers { permanent_uuid: "67a81d55b1bd4d2e9b587c6477d624a2" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40197 } attrs { replace: true } } peers { permanent_uuid: "c9ab11aee0734f048a27a267e9a34ee4" member_type: NON_VOTER last_known_addr { host: "127.15.255.130" port: 34399 } attrs { promote: true } } }
I20240502 21:28:35.164268 31980 catalog_manager.cc:5484] T 39d0ac5ec0cd4a33b947de380b5469be P e0a4e8376dae403e9b812a429b8960af reported cstate change: config changed from index -1 to 4, NON_VOTER c9ab11aee0734f048a27a267e9a34ee4 (127.15.255.130) added. New cstate: current_term: 1 leader_uuid: "e0a4e8376dae403e9b812a429b8960af" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e0a4e8376dae403e9b812a429b8960af" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 42105 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "67a81d55b1bd4d2e9b587c6477d624a2" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40197 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c9ab11aee0734f048a27a267e9a34ee4" member_type: NON_VOTER last_known_addr { host: "127.15.255.130" port: 34399 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240502 21:28:35.164602 32457 consensus_peers.cc:485] T 39d0ac5ec0cd4a33b947de380b5469be P e0a4e8376dae403e9b812a429b8960af -> Peer c9ab11aee0734f048a27a267e9a34ee4 (127.15.255.130:34399): Couldn't send request to peer c9ab11aee0734f048a27a267e9a34ee4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 39d0ac5ec0cd4a33b947de380b5469be. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:28:35.174945 32523 consensus_queue.cc:237] T d7c55e3c68744428a3bc7f2a1fe50c1f P e0a4e8376dae403e9b812a429b8960af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e0a4e8376dae403e9b812a429b8960af" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 42105 } attrs { replace: true } } peers { permanent_uuid: "67a81d55b1bd4d2e9b587c6477d624a2" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40197 } } peers { permanent_uuid: "c38cec78fdf74d05b3c75f21eeb8612c" member_type: NON_VOTER last_known_addr { host: "127.15.255.131" port: 45205 } attrs { promote: true } }
I20240502 21:28:35.176875 32124 raft_consensus.cc:1261] T d7c55e3c68744428a3bc7f2a1fe50c1f P 67a81d55b1bd4d2e9b587c6477d624a2 [term 1 FOLLOWER]: Refusing update from remote peer e0a4e8376dae403e9b812a429b8960af: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:28:35.177354 32732 consensus_queue.cc:1035] T d7c55e3c68744428a3bc7f2a1fe50c1f P e0a4e8376dae403e9b812a429b8960af [LEADER]: Connected to new peer: Peer: permanent_uuid: "67a81d55b1bd4d2e9b587c6477d624a2" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:28:35.179806 723 raft_consensus.cc:2932] T d7c55e3c68744428a3bc7f2a1fe50c1f P e0a4e8376dae403e9b812a429b8960af [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c38cec78fdf74d05b3c75f21eeb8612c (127.15.255.131) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e0a4e8376dae403e9b812a429b8960af" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 42105 } attrs { replace: true } } peers { permanent_uuid: "67a81d55b1bd4d2e9b587c6477d624a2" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40197 } } peers { permanent_uuid: "c38cec78fdf74d05b3c75f21eeb8612c" member_type: NON_VOTER last_known_addr { host: "127.15.255.131" port: 45205 } attrs { promote: true } } }
I20240502 21:28:35.180444 32124 raft_consensus.cc:2932] T d7c55e3c68744428a3bc7f2a1fe50c1f P 67a81d55b1bd4d2e9b587c6477d624a2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c38cec78fdf74d05b3c75f21eeb8612c (127.15.255.131) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e0a4e8376dae403e9b812a429b8960af" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 42105 } attrs { replace: true } } peers { permanent_uuid: "67a81d55b1bd4d2e9b587c6477d624a2" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40197 } } peers { permanent_uuid: "c38cec78fdf74d05b3c75f21eeb8612c" member_type: NON_VOTER last_known_addr { host: "127.15.255.131" port: 45205 } attrs { promote: true } } }
W20240502 21:28:35.182706 32456 consensus_peers.cc:485] T d7c55e3c68744428a3bc7f2a1fe50c1f P e0a4e8376dae403e9b812a429b8960af -> Peer c38cec78fdf74d05b3c75f21eeb8612c (127.15.255.131:45205): Couldn't send request to peer c38cec78fdf74d05b3c75f21eeb8612c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d7c55e3c68744428a3bc7f2a1fe50c1f. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:28:35.184481 31978 catalog_manager.cc:5484] T d7c55e3c68744428a3bc7f2a1fe50c1f P 67a81d55b1bd4d2e9b587c6477d624a2 reported cstate change: config changed from index -1 to 4, NON_VOTER c38cec78fdf74d05b3c75f21eeb8612c (127.15.255.131) added. New cstate: current_term: 1 leader_uuid: "e0a4e8376dae403e9b812a429b8960af" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e0a4e8376dae403e9b812a429b8960af" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 42105 } attrs { replace: true } } peers { permanent_uuid: "67a81d55b1bd4d2e9b587c6477d624a2" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40197 } } peers { permanent_uuid: "c38cec78fdf74d05b3c75f21eeb8612c" member_type: NON_VOTER last_known_addr { host: "127.15.255.131" port: 45205 } attrs { promote: true } } }
I20240502 21:28:35.200106 32523 consensus_queue.cc:237] T 2531188c066d4c33976b53333caed52c P e0a4e8376dae403e9b812a429b8960af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67a81d55b1bd4d2e9b587c6477d624a2" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40197 } attrs { replace: true } } peers { permanent_uuid: "e0a4e8376dae403e9b812a429b8960af" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 42105 } } peers { permanent_uuid: "c9ab11aee0734f048a27a267e9a34ee4" member_type: NON_VOTER last_known_addr { host: "127.15.255.130" port: 34399 } attrs { promote: true } }
W20240502 21:28:35.202399 32457 consensus_peers.cc:485] T 2531188c066d4c33976b53333caed52c P e0a4e8376dae403e9b812a429b8960af -> Peer c9ab11aee0734f048a27a267e9a34ee4 (127.15.255.130:34399): Couldn't send request to peer c9ab11aee0734f048a27a267e9a34ee4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2531188c066d4c33976b53333caed52c. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:28:35.202669 32124 raft_consensus.cc:1261] T 2531188c066d4c33976b53333caed52c P 67a81d55b1bd4d2e9b587c6477d624a2 [term 1 FOLLOWER]: Refusing update from remote peer e0a4e8376dae403e9b812a429b8960af: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:28:35.203191 733 consensus_queue.cc:1035] T 2531188c066d4c33976b53333caed52c P e0a4e8376dae403e9b812a429b8960af [LEADER]: Connected to new peer: Peer: permanent_uuid: "67a81d55b1bd4d2e9b587c6477d624a2" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40197 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:28:35.205674 734 raft_consensus.cc:2932] T 2531188c066d4c33976b53333caed52c P e0a4e8376dae403e9b812a429b8960af [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c9ab11aee0734f048a27a267e9a34ee4 (127.15.255.130) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67a81d55b1bd4d2e9b587c6477d624a2" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40197 } attrs { replace: true } } peers { permanent_uuid: "e0a4e8376dae403e9b812a429b8960af" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 42105 } } peers { permanent_uuid: "c9ab11aee0734f048a27a267e9a34ee4" member_type: NON_VOTER last_known_addr { host: "127.15.255.130" port: 34399 } attrs { promote: true } } }
I20240502 21:28:35.206249 32124 raft_consensus.cc:2932] T 2531188c066d4c33976b53333caed52c P 67a81d55b1bd4d2e9b587c6477d624a2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c9ab11aee0734f048a27a267e9a34ee4 (127.15.255.130) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67a81d55b1bd4d2e9b587c6477d624a2" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40197 } attrs { replace: true } } peers { permanent_uuid: "e0a4e8376dae403e9b812a429b8960af" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 42105 } } peers { permanent_uuid: "c9ab11aee0734f048a27a267e9a34ee4" member_type: NON_VOTER last_known_addr { host: "127.15.255.130" port: 34399 } attrs { promote: true } } }
I20240502 21:28:35.209955 31980 catalog_manager.cc:5484] T 2531188c066d4c33976b53333caed52c P e0a4e8376dae403e9b812a429b8960af reported cstate change: config changed from index -1 to 4, NON_VOTER c9ab11aee0734f048a27a267e9a34ee4 (127.15.255.130) added. New cstate: current_term: 1 leader_uuid: "e0a4e8376dae403e9b812a429b8960af" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67a81d55b1bd4d2e9b587c6477d624a2" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40197 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e0a4e8376dae403e9b812a429b8960af" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 42105 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c9ab11aee0734f048a27a267e9a34ee4" member_type: NON_VOTER last_known_addr { host: "127.15.255.130" port: 34399 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 21:28:35.220968 32124 consensus_queue.cc:237] T 0bf5f9db2efe49029fa1da7399ab5373 P 67a81d55b1bd4d2e9b587c6477d624a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67a81d55b1bd4d2e9b587c6477d624a2" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40197 } } peers { permanent_uuid: "e0a4e8376dae403e9b812a429b8960af" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 42105 } attrs { replace: true } } peers { permanent_uuid: "c38cec78fdf74d05b3c75f21eeb8612c" member_type: NON_VOTER last_known_addr { host: "127.15.255.131" port: 45205 } attrs { promote: true } }
I20240502 21:28:35.222988 32523 raft_consensus.cc:1261] T 0bf5f9db2efe49029fa1da7399ab5373 P e0a4e8376dae403e9b812a429b8960af [term 1 FOLLOWER]: Refusing update from remote peer 67a81d55b1bd4d2e9b587c6477d624a2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 21:28:35.223205 32058 consensus_peers.cc:485] T 0bf5f9db2efe49029fa1da7399ab5373 P 67a81d55b1bd4d2e9b587c6477d624a2 -> Peer c38cec78fdf74d05b3c75f21eeb8612c (127.15.255.131:45205): Couldn't send request to peer c38cec78fdf74d05b3c75f21eeb8612c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0bf5f9db2efe49029fa1da7399ab5373. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:28:35.223466 708 consensus_queue.cc:1035] T 0bf5f9db2efe49029fa1da7399ab5373 P 67a81d55b1bd4d2e9b587c6477d624a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0a4e8376dae403e9b812a429b8960af" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 42105 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:28:35.225764 711 raft_consensus.cc:2932] T 0bf5f9db2efe49029fa1da7399ab5373 P 67a81d55b1bd4d2e9b587c6477d624a2 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c38cec78fdf74d05b3c75f21eeb8612c (127.15.255.131) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67a81d55b1bd4d2e9b587c6477d624a2" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40197 } } peers { permanent_uuid: "e0a4e8376dae403e9b812a429b8960af" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 42105 } attrs { replace: true } } peers { permanent_uuid: "c38cec78fdf74d05b3c75f21eeb8612c" member_type: NON_VOTER last_known_addr { host: "127.15.255.131" port: 45205 } attrs { promote: true } } }
I20240502 21:28:35.226383 32523 raft_consensus.cc:2932] T 0bf5f9db2efe49029fa1da7399ab5373 P e0a4e8376dae403e9b812a429b8960af [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c38cec78fdf74d05b3c75f21eeb8612c (127.15.255.131) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67a81d55b1bd4d2e9b587c6477d624a2" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40197 } } peers { permanent_uuid: "e0a4e8376dae403e9b812a429b8960af" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 42105 } attrs { replace: true } } peers { permanent_uuid: "c38cec78fdf74d05b3c75f21eeb8612c" member_type: NON_VOTER last_known_addr { host: "127.15.255.131" port: 45205 } attrs { promote: true } } }
I20240502 21:28:35.229882 31978 catalog_manager.cc:5484] T 0bf5f9db2efe49029fa1da7399ab5373 P 67a81d55b1bd4d2e9b587c6477d624a2 reported cstate change: config changed from index -1 to 4, NON_VOTER c38cec78fdf74d05b3c75f21eeb8612c (127.15.255.131) added. New cstate: current_term: 1 leader_uuid: "67a81d55b1bd4d2e9b587c6477d624a2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67a81d55b1bd4d2e9b587c6477d624a2" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40197 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e0a4e8376dae403e9b812a429b8960af" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 42105 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c38cec78fdf74d05b3c75f21eeb8612c" member_type: NON_VOTER last_known_addr { host: "127.15.255.131" port: 45205 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 21:28:35.240710 32523 consensus_queue.cc:237] T 4ea23c1a76784237a5619fefa55335fc P e0a4e8376dae403e9b812a429b8960af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e0a4e8376dae403e9b812a429b8960af" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 42105 } } peers { permanent_uuid: "67a81d55b1bd4d2e9b587c6477d624a2" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40197 } attrs { replace: true } } peers { permanent_uuid: "c9ab11aee0734f048a27a267e9a34ee4" member_type: NON_VOTER last_known_addr { host: "127.15.255.130" port: 34399 } attrs { promote: true } }
I20240502 21:28:35.242570 32124 raft_consensus.cc:1261] T 4ea23c1a76784237a5619fefa55335fc P 67a81d55b1bd4d2e9b587c6477d624a2 [term 1 FOLLOWER]: Refusing update from remote peer e0a4e8376dae403e9b812a429b8960af: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 21:28:35.242961 32457 consensus_peers.cc:485] T 4ea23c1a76784237a5619fefa55335fc P e0a4e8376dae403e9b812a429b8960af -> Peer c9ab11aee0734f048a27a267e9a34ee4 (127.15.255.130:34399): Couldn't send request to peer c9ab11aee0734f048a27a267e9a34ee4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4ea23c1a76784237a5619fefa55335fc. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:28:35.243196 733 consensus_queue.cc:1035] T 4ea23c1a76784237a5619fefa55335fc P e0a4e8376dae403e9b812a429b8960af [LEADER]: Connected to new peer: Peer: permanent_uuid: "67a81d55b1bd4d2e9b587c6477d624a2" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40197 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:28:35.245725 734 raft_consensus.cc:2932] T 4ea23c1a76784237a5619fefa55335fc P e0a4e8376dae403e9b812a429b8960af [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c9ab11aee0734f048a27a267e9a34ee4 (127.15.255.130) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e0a4e8376dae403e9b812a429b8960af" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 42105 } } peers { permanent_uuid: "67a81d55b1bd4d2e9b587c6477d624a2" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40197 } attrs { replace: true } } peers { permanent_uuid: "c9ab11aee0734f048a27a267e9a34ee4" member_type: NON_VOTER last_known_addr { host: "127.15.255.130" port: 34399 } attrs { promote: true } } }
I20240502 21:28:35.246282 32124 raft_consensus.cc:2932] T 4ea23c1a76784237a5619fefa55335fc P 67a81d55b1bd4d2e9b587c6477d624a2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c9ab11aee0734f048a27a267e9a34ee4 (127.15.255.130) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e0a4e8376dae403e9b812a429b8960af" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 42105 } } peers { permanent_uuid: "67a81d55b1bd4d2e9b587c6477d624a2" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40197 } attrs { replace: true } } peers { permanent_uuid: "c9ab11aee0734f048a27a267e9a34ee4" member_type: NON_VOTER last_known_addr { host: "127.15.255.130" port: 34399 } attrs { promote: true } } }
I20240502 21:28:35.250012 31980 catalog_manager.cc:5484] T 4ea23c1a76784237a5619fefa55335fc P 67a81d55b1bd4d2e9b587c6477d624a2 reported cstate change: config changed from index -1 to 4, NON_VOTER c9ab11aee0734f048a27a267e9a34ee4 (127.15.255.130) added. New cstate: current_term: 1 leader_uuid: "e0a4e8376dae403e9b812a429b8960af" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e0a4e8376dae403e9b812a429b8960af" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 42105 } } peers { permanent_uuid: "67a81d55b1bd4d2e9b587c6477d624a2" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40197 } attrs { replace: true } } peers { permanent_uuid: "c9ab11aee0734f048a27a267e9a34ee4" member_type: NON_VOTER last_known_addr { host: "127.15.255.130" port: 34399 } attrs { promote: true } } }
I20240502 21:28:35.539453 811 ts_tablet_manager.cc:902] T 39d0ac5ec0cd4a33b947de380b5469be P c9ab11aee0734f048a27a267e9a34ee4: Initiating tablet copy from peer e0a4e8376dae403e9b812a429b8960af (127.15.255.132:42105)
I20240502 21:28:35.541280 811 tablet_copy_client.cc:308] T 39d0ac5ec0cd4a33b947de380b5469be P c9ab11aee0734f048a27a267e9a34ee4: tablet copy: Beginning tablet copy session from remote peer at address 127.15.255.132:42105
I20240502 21:28:35.548220 32543 tablet_copy_service.cc:140] P e0a4e8376dae403e9b812a429b8960af: Received BeginTabletCopySession request for tablet 39d0ac5ec0cd4a33b947de380b5469be from peer c9ab11aee0734f048a27a267e9a34ee4 ({username='slave'} at 127.15.255.130:60007)
I20240502 21:28:35.548485 32543 tablet_copy_service.cc:161] P e0a4e8376dae403e9b812a429b8960af: Beginning new tablet copy session on tablet 39d0ac5ec0cd4a33b947de380b5469be from peer c9ab11aee0734f048a27a267e9a34ee4 at {username='slave'} at 127.15.255.130:60007: session id = c9ab11aee0734f048a27a267e9a34ee4-39d0ac5ec0cd4a33b947de380b5469be
I20240502 21:28:35.551112 32543 tablet_copy_source_session.cc:205] T 39d0ac5ec0cd4a33b947de380b5469be P e0a4e8376dae403e9b812a429b8960af: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:28:35.555696 811 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39d0ac5ec0cd4a33b947de380b5469be. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:35.566310 811 tablet_copy_client.cc:783] T 39d0ac5ec0cd4a33b947de380b5469be P c9ab11aee0734f048a27a267e9a34ee4: tablet copy: Starting download of 0 data blocks...
I20240502 21:28:35.566625 811 tablet_copy_client.cc:647] T 39d0ac5ec0cd4a33b947de380b5469be P c9ab11aee0734f048a27a267e9a34ee4: tablet copy: Starting download of 1 WAL segments...
I20240502 21:28:35.568728 811 tablet_copy_client.cc:523] T 39d0ac5ec0cd4a33b947de380b5469be P c9ab11aee0734f048a27a267e9a34ee4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:28:35.571555 811 tablet_bootstrap.cc:492] T 39d0ac5ec0cd4a33b947de380b5469be P c9ab11aee0734f048a27a267e9a34ee4: Bootstrap starting.
I20240502 21:28:35.579913 811 log.cc:824] T 39d0ac5ec0cd4a33b947de380b5469be P c9ab11aee0734f048a27a267e9a34ee4: Log is configured to *not* fsync() on all Append() calls
I20240502 21:28:35.583537 826 ts_tablet_manager.cc:902] T d7c55e3c68744428a3bc7f2a1fe50c1f P c38cec78fdf74d05b3c75f21eeb8612c: Initiating tablet copy from peer e0a4e8376dae403e9b812a429b8960af (127.15.255.132:42105)
I20240502 21:28:35.585245 826 tablet_copy_client.cc:308] T d7c55e3c68744428a3bc7f2a1fe50c1f P c38cec78fdf74d05b3c75f21eeb8612c: tablet copy: Beginning tablet copy session from remote peer at address 127.15.255.132:42105
I20240502 21:28:35.591886 32543 tablet_copy_service.cc:140] P e0a4e8376dae403e9b812a429b8960af: Received BeginTabletCopySession request for tablet d7c55e3c68744428a3bc7f2a1fe50c1f from peer c38cec78fdf74d05b3c75f21eeb8612c ({username='slave'} at 127.15.255.131:50157)
I20240502 21:28:35.592032 32543 tablet_copy_service.cc:161] P e0a4e8376dae403e9b812a429b8960af: Beginning new tablet copy session on tablet d7c55e3c68744428a3bc7f2a1fe50c1f from peer c38cec78fdf74d05b3c75f21eeb8612c at {username='slave'} at 127.15.255.131:50157: session id = c38cec78fdf74d05b3c75f21eeb8612c-d7c55e3c68744428a3bc7f2a1fe50c1f
I20240502 21:28:35.593950 32543 tablet_copy_source_session.cc:205] T d7c55e3c68744428a3bc7f2a1fe50c1f P e0a4e8376dae403e9b812a429b8960af: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:28:35.597556 826 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d7c55e3c68744428a3bc7f2a1fe50c1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:35.607121 826 tablet_copy_client.cc:783] T d7c55e3c68744428a3bc7f2a1fe50c1f P c38cec78fdf74d05b3c75f21eeb8612c: tablet copy: Starting download of 0 data blocks...
I20240502 21:28:35.607409 826 tablet_copy_client.cc:647] T d7c55e3c68744428a3bc7f2a1fe50c1f P c38cec78fdf74d05b3c75f21eeb8612c: tablet copy: Starting download of 1 WAL segments...
I20240502 21:28:35.609450 826 tablet_copy_client.cc:523] T d7c55e3c68744428a3bc7f2a1fe50c1f P c38cec78fdf74d05b3c75f21eeb8612c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:28:35.611840 811 tablet_bootstrap.cc:492] T 39d0ac5ec0cd4a33b947de380b5469be P c9ab11aee0734f048a27a267e9a34ee4: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:28:35.612227 811 tablet_bootstrap.cc:492] T 39d0ac5ec0cd4a33b947de380b5469be P c9ab11aee0734f048a27a267e9a34ee4: Bootstrap complete.
I20240502 21:28:35.612413 826 tablet_bootstrap.cc:492] T d7c55e3c68744428a3bc7f2a1fe50c1f P c38cec78fdf74d05b3c75f21eeb8612c: Bootstrap starting.
I20240502 21:28:35.612632 811 ts_tablet_manager.cc:1367] T 39d0ac5ec0cd4a33b947de380b5469be P c9ab11aee0734f048a27a267e9a34ee4: Time spent bootstrapping tablet: real 0.041s user 0.043s sys 0.001s
I20240502 21:28:35.617007 811 raft_consensus.cc:348] T 39d0ac5ec0cd4a33b947de380b5469be P c9ab11aee0734f048a27a267e9a34ee4 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e0a4e8376dae403e9b812a429b8960af" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 42105 } } peers { permanent_uuid: "67a81d55b1bd4d2e9b587c6477d624a2" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40197 } attrs { replace: true } } peers { permanent_uuid: "c9ab11aee0734f048a27a267e9a34ee4" member_type: NON_VOTER last_known_addr { host: "127.15.255.130" port: 34399 } attrs { promote: true } }
I20240502 21:28:35.617365 811 raft_consensus.cc:729] T 39d0ac5ec0cd4a33b947de380b5469be P c9ab11aee0734f048a27a267e9a34ee4 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c9ab11aee0734f048a27a267e9a34ee4, State: Initialized, Role: LEARNER
I20240502 21:28:35.617813 811 consensus_queue.cc:260] T 39d0ac5ec0cd4a33b947de380b5469be P c9ab11aee0734f048a27a267e9a34ee4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e0a4e8376dae403e9b812a429b8960af" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 42105 } } peers { permanent_uuid: "67a81d55b1bd4d2e9b587c6477d624a2" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40197 } attrs { replace: true } } peers { permanent_uuid: "c9ab11aee0734f048a27a267e9a34ee4" member_type: NON_VOTER last_known_addr { host: "127.15.255.130" port: 34399 } attrs { promote: true } }
I20240502 21:28:35.619267 811 ts_tablet_manager.cc:1398] T 39d0ac5ec0cd4a33b947de380b5469be P c9ab11aee0734f048a27a267e9a34ee4: Time spent starting tablet: real 0.007s user 0.002s sys 0.004s
I20240502 21:28:35.620132 32543 tablet_copy_service.cc:342] P e0a4e8376dae403e9b812a429b8960af: Request end of tablet copy session c9ab11aee0734f048a27a267e9a34ee4-39d0ac5ec0cd4a33b947de380b5469be received from {username='slave'} at 127.15.255.130:60007
I20240502 21:28:35.620321 32543 tablet_copy_service.cc:433] P e0a4e8376dae403e9b812a429b8960af: ending tablet copy session c9ab11aee0734f048a27a267e9a34ee4-39d0ac5ec0cd4a33b947de380b5469be on tablet 39d0ac5ec0cd4a33b947de380b5469be with peer c9ab11aee0734f048a27a267e9a34ee4
I20240502 21:28:35.622041 826 log.cc:824] T d7c55e3c68744428a3bc7f2a1fe50c1f P c38cec78fdf74d05b3c75f21eeb8612c: Log is configured to *not* fsync() on all Append() calls
I20240502 21:28:35.622881 811 ts_tablet_manager.cc:902] T 4ea23c1a76784237a5619fefa55335fc P c9ab11aee0734f048a27a267e9a34ee4: Initiating tablet copy from peer e0a4e8376dae403e9b812a429b8960af (127.15.255.132:42105)
I20240502 21:28:35.629077 811 tablet_copy_client.cc:308] T 4ea23c1a76784237a5619fefa55335fc P c9ab11aee0734f048a27a267e9a34ee4: tablet copy: Beginning tablet copy session from remote peer at address 127.15.255.132:42105
I20240502 21:28:35.629673 32543 tablet_copy_service.cc:140] P e0a4e8376dae403e9b812a429b8960af: Received BeginTabletCopySession request for tablet 4ea23c1a76784237a5619fefa55335fc from peer c9ab11aee0734f048a27a267e9a34ee4 ({username='slave'} at 127.15.255.130:60007)
I20240502 21:28:35.629792 32543 tablet_copy_service.cc:161] P e0a4e8376dae403e9b812a429b8960af: Beginning new tablet copy session on tablet 4ea23c1a76784237a5619fefa55335fc from peer c9ab11aee0734f048a27a267e9a34ee4 at {username='slave'} at 127.15.255.130:60007: session id = c9ab11aee0734f048a27a267e9a34ee4-4ea23c1a76784237a5619fefa55335fc
I20240502 21:28:35.631626 32543 tablet_copy_source_session.cc:205] T 4ea23c1a76784237a5619fefa55335fc P e0a4e8376dae403e9b812a429b8960af: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:28:35.632589 811 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ea23c1a76784237a5619fefa55335fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:35.636644 811 tablet_copy_client.cc:783] T 4ea23c1a76784237a5619fefa55335fc P c9ab11aee0734f048a27a267e9a34ee4: tablet copy: Starting download of 0 data blocks...
I20240502 21:28:35.636883 811 tablet_copy_client.cc:647] T 4ea23c1a76784237a5619fefa55335fc P c9ab11aee0734f048a27a267e9a34ee4: tablet copy: Starting download of 1 WAL segments...
I20240502 21:28:35.638305 811 tablet_copy_client.cc:523] T 4ea23c1a76784237a5619fefa55335fc P c9ab11aee0734f048a27a267e9a34ee4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:28:35.640475 811 tablet_bootstrap.cc:492] T 4ea23c1a76784237a5619fefa55335fc P c9ab11aee0734f048a27a267e9a34ee4: Bootstrap starting.
I20240502 21:28:35.652510 826 tablet_bootstrap.cc:492] T d7c55e3c68744428a3bc7f2a1fe50c1f P c38cec78fdf74d05b3c75f21eeb8612c: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:28:35.652884 826 tablet_bootstrap.cc:492] T d7c55e3c68744428a3bc7f2a1fe50c1f P c38cec78fdf74d05b3c75f21eeb8612c: Bootstrap complete.
I20240502 21:28:35.653218 826 ts_tablet_manager.cc:1367] T d7c55e3c68744428a3bc7f2a1fe50c1f P c38cec78fdf74d05b3c75f21eeb8612c: Time spent bootstrapping tablet: real 0.041s user 0.039s sys 0.004s
I20240502 21:28:35.657403 826 raft_consensus.cc:348] T d7c55e3c68744428a3bc7f2a1fe50c1f P c38cec78fdf74d05b3c75f21eeb8612c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e0a4e8376dae403e9b812a429b8960af" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 42105 } attrs { replace: true } } peers { permanent_uuid: "67a81d55b1bd4d2e9b587c6477d624a2" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40197 } } peers { permanent_uuid: "c38cec78fdf74d05b3c75f21eeb8612c" member_type: NON_VOTER last_known_addr { host: "127.15.255.131" port: 45205 } attrs { promote: true } }
I20240502 21:28:35.657713 826 raft_consensus.cc:729] T d7c55e3c68744428a3bc7f2a1fe50c1f P c38cec78fdf74d05b3c75f21eeb8612c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c38cec78fdf74d05b3c75f21eeb8612c, State: Initialized, Role: LEARNER
I20240502 21:28:35.658147 826 consensus_queue.cc:260] T d7c55e3c68744428a3bc7f2a1fe50c1f P c38cec78fdf74d05b3c75f21eeb8612c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e0a4e8376dae403e9b812a429b8960af" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 42105 } attrs { replace: true } } peers { permanent_uuid: "67a81d55b1bd4d2e9b587c6477d624a2" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40197 } } peers { permanent_uuid: "c38cec78fdf74d05b3c75f21eeb8612c" member_type: NON_VOTER last_known_addr { host: "127.15.255.131" port: 45205 } attrs { promote: true } }
I20240502 21:28:35.659509 826 ts_tablet_manager.cc:1398] T d7c55e3c68744428a3bc7f2a1fe50c1f P c38cec78fdf74d05b3c75f21eeb8612c: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20240502 21:28:35.660379 32543 tablet_copy_service.cc:342] P e0a4e8376dae403e9b812a429b8960af: Request end of tablet copy session c38cec78fdf74d05b3c75f21eeb8612c-d7c55e3c68744428a3bc7f2a1fe50c1f received from {username='slave'} at 127.15.255.131:50157
I20240502 21:28:35.660552 32543 tablet_copy_service.cc:433] P e0a4e8376dae403e9b812a429b8960af: ending tablet copy session c38cec78fdf74d05b3c75f21eeb8612c-d7c55e3c68744428a3bc7f2a1fe50c1f on tablet d7c55e3c68744428a3bc7f2a1fe50c1f with peer c38cec78fdf74d05b3c75f21eeb8612c
I20240502 21:28:35.670773 811 tablet_bootstrap.cc:492] T 4ea23c1a76784237a5619fefa55335fc P c9ab11aee0734f048a27a267e9a34ee4: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:28:35.671057 811 tablet_bootstrap.cc:492] T 4ea23c1a76784237a5619fefa55335fc P c9ab11aee0734f048a27a267e9a34ee4: Bootstrap complete.
I20240502 21:28:35.671252 811 ts_tablet_manager.cc:1367] T 4ea23c1a76784237a5619fefa55335fc P c9ab11aee0734f048a27a267e9a34ee4: Time spent bootstrapping tablet: real 0.031s user 0.022s sys 0.008s
I20240502 21:28:35.671869 811 raft_consensus.cc:348] T 4ea23c1a76784237a5619fefa55335fc P c9ab11aee0734f048a27a267e9a34ee4 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e0a4e8376dae403e9b812a429b8960af" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 42105 } } peers { permanent_uuid: "67a81d55b1bd4d2e9b587c6477d624a2" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40197 } attrs { replace: true } } peers { permanent_uuid: "c9ab11aee0734f048a27a267e9a34ee4" member_type: NON_VOTER last_known_addr { host: "127.15.255.130" port: 34399 } attrs { promote: true } }
I20240502 21:28:35.672050 811 raft_consensus.cc:729] T 4ea23c1a76784237a5619fefa55335fc P c9ab11aee0734f048a27a267e9a34ee4 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c9ab11aee0734f048a27a267e9a34ee4, State: Initialized, Role: LEARNER
I20240502 21:28:35.672206 811 consensus_queue.cc:260] T 4ea23c1a76784237a5619fefa55335fc P c9ab11aee0734f048a27a267e9a34ee4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e0a4e8376dae403e9b812a429b8960af" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 42105 } } peers { permanent_uuid: "67a81d55b1bd4d2e9b587c6477d624a2" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40197 } attrs { replace: true } } peers { permanent_uuid: "c9ab11aee0734f048a27a267e9a34ee4" member_type: NON_VOTER last_known_addr { host: "127.15.255.130" port: 34399 } attrs { promote: true } }
I20240502 21:28:35.672672 811 ts_tablet_manager.cc:1398] T 4ea23c1a76784237a5619fefa55335fc P c9ab11aee0734f048a27a267e9a34ee4: Time spent starting tablet: real 0.001s user 0.004s sys 0.000s
I20240502 21:28:35.673339 32543 tablet_copy_service.cc:342] P e0a4e8376dae403e9b812a429b8960af: Request end of tablet copy session c9ab11aee0734f048a27a267e9a34ee4-4ea23c1a76784237a5619fefa55335fc received from {username='slave'} at 127.15.255.130:60007
I20240502 21:28:35.673460 32543 tablet_copy_service.cc:433] P e0a4e8376dae403e9b812a429b8960af: ending tablet copy session c9ab11aee0734f048a27a267e9a34ee4-4ea23c1a76784237a5619fefa55335fc on tablet 4ea23c1a76784237a5619fefa55335fc with peer c9ab11aee0734f048a27a267e9a34ee4
I20240502 21:28:35.750977 826 ts_tablet_manager.cc:902] T 5b4ff662667a442cabfee1a1818d8dca P c38cec78fdf74d05b3c75f21eeb8612c: Initiating tablet copy from peer 67a81d55b1bd4d2e9b587c6477d624a2 (127.15.255.129:40197)
I20240502 21:28:35.752218 826 tablet_copy_client.cc:308] T 5b4ff662667a442cabfee1a1818d8dca P c38cec78fdf74d05b3c75f21eeb8612c: tablet copy: Beginning tablet copy session from remote peer at address 127.15.255.129:40197
I20240502 21:28:35.757951 32146 tablet_copy_service.cc:140] P 67a81d55b1bd4d2e9b587c6477d624a2: Received BeginTabletCopySession request for tablet 5b4ff662667a442cabfee1a1818d8dca from peer c38cec78fdf74d05b3c75f21eeb8612c ({username='slave'} at 127.15.255.131:46775)
I20240502 21:28:35.758172 32146 tablet_copy_service.cc:161] P 67a81d55b1bd4d2e9b587c6477d624a2: Beginning new tablet copy session on tablet 5b4ff662667a442cabfee1a1818d8dca from peer c38cec78fdf74d05b3c75f21eeb8612c at {username='slave'} at 127.15.255.131:46775: session id = c38cec78fdf74d05b3c75f21eeb8612c-5b4ff662667a442cabfee1a1818d8dca
I20240502 21:28:35.761047 32146 tablet_copy_source_session.cc:205] T 5b4ff662667a442cabfee1a1818d8dca P 67a81d55b1bd4d2e9b587c6477d624a2: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:28:35.762833 826 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b4ff662667a442cabfee1a1818d8dca. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:35.767853 826 tablet_copy_client.cc:783] T 5b4ff662667a442cabfee1a1818d8dca P c38cec78fdf74d05b3c75f21eeb8612c: tablet copy: Starting download of 0 data blocks...
I20240502 21:28:35.768159 826 tablet_copy_client.cc:647] T 5b4ff662667a442cabfee1a1818d8dca P c38cec78fdf74d05b3c75f21eeb8612c: tablet copy: Starting download of 1 WAL segments...
I20240502 21:28:35.770144 826 tablet_copy_client.cc:523] T 5b4ff662667a442cabfee1a1818d8dca P c38cec78fdf74d05b3c75f21eeb8612c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:28:35.772675 826 tablet_bootstrap.cc:492] T 5b4ff662667a442cabfee1a1818d8dca P c38cec78fdf74d05b3c75f21eeb8612c: Bootstrap starting.
I20240502 21:28:35.791705 811 ts_tablet_manager.cc:902] T 2531188c066d4c33976b53333caed52c P c9ab11aee0734f048a27a267e9a34ee4: Initiating tablet copy from peer e0a4e8376dae403e9b812a429b8960af (127.15.255.132:42105)
I20240502 21:28:35.794477 875 ts_tablet_manager.cc:902] T 0bf5f9db2efe49029fa1da7399ab5373 P c38cec78fdf74d05b3c75f21eeb8612c: Initiating tablet copy from peer 67a81d55b1bd4d2e9b587c6477d624a2 (127.15.255.129:40197)
I20240502 21:28:35.809438 811 tablet_copy_client.cc:308] T 2531188c066d4c33976b53333caed52c P c9ab11aee0734f048a27a267e9a34ee4: tablet copy: Beginning tablet copy session from remote peer at address 127.15.255.132:42105
I20240502 21:28:35.810187 32543 tablet_copy_service.cc:140] P e0a4e8376dae403e9b812a429b8960af: Received BeginTabletCopySession request for tablet 2531188c066d4c33976b53333caed52c from peer c9ab11aee0734f048a27a267e9a34ee4 ({username='slave'} at 127.15.255.130:60007)
I20240502 21:28:35.810364 32543 tablet_copy_service.cc:161] P e0a4e8376dae403e9b812a429b8960af: Beginning new tablet copy session on tablet 2531188c066d4c33976b53333caed52c from peer c9ab11aee0734f048a27a267e9a34ee4 at {username='slave'} at 127.15.255.130:60007: session id = c9ab11aee0734f048a27a267e9a34ee4-2531188c066d4c33976b53333caed52c
I20240502 21:28:35.815001 32543 tablet_copy_source_session.cc:205] T 2531188c066d4c33976b53333caed52c P e0a4e8376dae403e9b812a429b8960af: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:28:35.816319 811 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2531188c066d4c33976b53333caed52c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:35.818670 826 tablet_bootstrap.cc:492] T 5b4ff662667a442cabfee1a1818d8dca P c38cec78fdf74d05b3c75f21eeb8612c: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:28:35.819268 826 tablet_bootstrap.cc:492] T 5b4ff662667a442cabfee1a1818d8dca P c38cec78fdf74d05b3c75f21eeb8612c: Bootstrap complete.
I20240502 21:28:35.819594 826 ts_tablet_manager.cc:1367] T 5b4ff662667a442cabfee1a1818d8dca P c38cec78fdf74d05b3c75f21eeb8612c: Time spent bootstrapping tablet: real 0.047s user 0.041s sys 0.000s
I20240502 21:28:35.820684 826 raft_consensus.cc:348] T 5b4ff662667a442cabfee1a1818d8dca P c38cec78fdf74d05b3c75f21eeb8612c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e0a4e8376dae403e9b812a429b8960af" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 42105 } attrs { replace: true } } peers { permanent_uuid: "67a81d55b1bd4d2e9b587c6477d624a2" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40197 } } peers { permanent_uuid: "c38cec78fdf74d05b3c75f21eeb8612c" member_type: NON_VOTER last_known_addr { host: "127.15.255.131" port: 45205 } attrs { promote: true } }
I20240502 21:28:35.820950 826 raft_consensus.cc:729] T 5b4ff662667a442cabfee1a1818d8dca P c38cec78fdf74d05b3c75f21eeb8612c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c38cec78fdf74d05b3c75f21eeb8612c, State: Initialized, Role: LEARNER
I20240502 21:28:35.821274 826 consensus_queue.cc:260] T 5b4ff662667a442cabfee1a1818d8dca P c38cec78fdf74d05b3c75f21eeb8612c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e0a4e8376dae403e9b812a429b8960af" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 42105 } attrs { replace: true } } peers { permanent_uuid: "67a81d55b1bd4d2e9b587c6477d624a2" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40197 } } peers { permanent_uuid: "c38cec78fdf74d05b3c75f21eeb8612c" member_type: NON_VOTER last_known_addr { host: "127.15.255.131" port: 45205 } attrs { promote: true } }
I20240502 21:28:35.822667 875 tablet_copy_client.cc:308] T 0bf5f9db2efe49029fa1da7399ab5373 P c38cec78fdf74d05b3c75f21eeb8612c: tablet copy: Beginning tablet copy session from remote peer at address 127.15.255.129:40197
I20240502 21:28:35.822674 826 ts_tablet_manager.cc:1398] T 5b4ff662667a442cabfee1a1818d8dca P c38cec78fdf74d05b3c75f21eeb8612c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:28:35.824695 32145 tablet_copy_service.cc:342] P 67a81d55b1bd4d2e9b587c6477d624a2: Request end of tablet copy session c38cec78fdf74d05b3c75f21eeb8612c-5b4ff662667a442cabfee1a1818d8dca received from {username='slave'} at 127.15.255.131:46775
I20240502 21:28:35.824857 32145 tablet_copy_service.cc:433] P 67a81d55b1bd4d2e9b587c6477d624a2: ending tablet copy session c38cec78fdf74d05b3c75f21eeb8612c-5b4ff662667a442cabfee1a1818d8dca on tablet 5b4ff662667a442cabfee1a1818d8dca with peer c38cec78fdf74d05b3c75f21eeb8612c
I20240502 21:28:35.825562 32146 tablet_copy_service.cc:140] P 67a81d55b1bd4d2e9b587c6477d624a2: Received BeginTabletCopySession request for tablet 0bf5f9db2efe49029fa1da7399ab5373 from peer c38cec78fdf74d05b3c75f21eeb8612c ({username='slave'} at 127.15.255.131:46775)
I20240502 21:28:35.825742 32146 tablet_copy_service.cc:161] P 67a81d55b1bd4d2e9b587c6477d624a2: Beginning new tablet copy session on tablet 0bf5f9db2efe49029fa1da7399ab5373 from peer c38cec78fdf74d05b3c75f21eeb8612c at {username='slave'} at 127.15.255.131:46775: session id = c38cec78fdf74d05b3c75f21eeb8612c-0bf5f9db2efe49029fa1da7399ab5373
I20240502 21:28:35.828114 32146 tablet_copy_source_session.cc:205] T 0bf5f9db2efe49029fa1da7399ab5373 P 67a81d55b1bd4d2e9b587c6477d624a2: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:28:35.828734 811 tablet_copy_client.cc:783] T 2531188c066d4c33976b53333caed52c P c9ab11aee0734f048a27a267e9a34ee4: tablet copy: Starting download of 0 data blocks...
I20240502 21:28:35.828977 811 tablet_copy_client.cc:647] T 2531188c066d4c33976b53333caed52c P c9ab11aee0734f048a27a267e9a34ee4: tablet copy: Starting download of 1 WAL segments...
I20240502 21:28:35.829427 875 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0bf5f9db2efe49029fa1da7399ab5373. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:35.830361 811 tablet_copy_client.cc:523] T 2531188c066d4c33976b53333caed52c P c9ab11aee0734f048a27a267e9a34ee4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:28:35.832620 811 tablet_bootstrap.cc:492] T 2531188c066d4c33976b53333caed52c P c9ab11aee0734f048a27a267e9a34ee4: Bootstrap starting.
I20240502 21:28:35.833492 875 tablet_copy_client.cc:783] T 0bf5f9db2efe49029fa1da7399ab5373 P c38cec78fdf74d05b3c75f21eeb8612c: tablet copy: Starting download of 0 data blocks...
I20240502 21:28:35.833721 875 tablet_copy_client.cc:647] T 0bf5f9db2efe49029fa1da7399ab5373 P c38cec78fdf74d05b3c75f21eeb8612c: tablet copy: Starting download of 1 WAL segments...
I20240502 21:28:35.835161 875 tablet_copy_client.cc:523] T 0bf5f9db2efe49029fa1da7399ab5373 P c38cec78fdf74d05b3c75f21eeb8612c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:28:35.837255 875 tablet_bootstrap.cc:492] T 0bf5f9db2efe49029fa1da7399ab5373 P c38cec78fdf74d05b3c75f21eeb8612c: Bootstrap starting.
I20240502 21:28:35.863948 811 tablet_bootstrap.cc:492] T 2531188c066d4c33976b53333caed52c P c9ab11aee0734f048a27a267e9a34ee4: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:28:35.864288 811 tablet_bootstrap.cc:492] T 2531188c066d4c33976b53333caed52c P c9ab11aee0734f048a27a267e9a34ee4: Bootstrap complete.
I20240502 21:28:35.864527 811 ts_tablet_manager.cc:1367] T 2531188c066d4c33976b53333caed52c P c9ab11aee0734f048a27a267e9a34ee4: Time spent bootstrapping tablet: real 0.032s user 0.029s sys 0.003s
I20240502 21:28:35.865128 811 raft_consensus.cc:348] T 2531188c066d4c33976b53333caed52c P c9ab11aee0734f048a27a267e9a34ee4 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67a81d55b1bd4d2e9b587c6477d624a2" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40197 } attrs { replace: true } } peers { permanent_uuid: "e0a4e8376dae403e9b812a429b8960af" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 42105 } } peers { permanent_uuid: "c9ab11aee0734f048a27a267e9a34ee4" member_type: NON_VOTER last_known_addr { host: "127.15.255.130" port: 34399 } attrs { promote: true } }
I20240502 21:28:35.865306 811 raft_consensus.cc:729] T 2531188c066d4c33976b53333caed52c P c9ab11aee0734f048a27a267e9a34ee4 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c9ab11aee0734f048a27a267e9a34ee4, State: Initialized, Role: LEARNER
I20240502 21:28:35.865489 811 consensus_queue.cc:260] T 2531188c066d4c33976b53333caed52c P c9ab11aee0734f048a27a267e9a34ee4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67a81d55b1bd4d2e9b587c6477d624a2" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40197 } attrs { replace: true } } peers { permanent_uuid: "e0a4e8376dae403e9b812a429b8960af" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 42105 } } peers { permanent_uuid: "c9ab11aee0734f048a27a267e9a34ee4" member_type: NON_VOTER last_known_addr { host: "127.15.255.130" port: 34399 } attrs { promote: true } }
I20240502 21:28:35.865983 811 ts_tablet_manager.cc:1398] T 2531188c066d4c33976b53333caed52c P c9ab11aee0734f048a27a267e9a34ee4: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 21:28:35.866077 875 tablet_bootstrap.cc:492] T 0bf5f9db2efe49029fa1da7399ab5373 P c38cec78fdf74d05b3c75f21eeb8612c: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:28:35.866506 875 tablet_bootstrap.cc:492] T 0bf5f9db2efe49029fa1da7399ab5373 P c38cec78fdf74d05b3c75f21eeb8612c: Bootstrap complete.
I20240502 21:28:35.866760 875 ts_tablet_manager.cc:1367] T 0bf5f9db2efe49029fa1da7399ab5373 P c38cec78fdf74d05b3c75f21eeb8612c: Time spent bootstrapping tablet: real 0.030s user 0.024s sys 0.004s
I20240502 21:28:35.866736 32543 tablet_copy_service.cc:342] P e0a4e8376dae403e9b812a429b8960af: Request end of tablet copy session c9ab11aee0734f048a27a267e9a34ee4-2531188c066d4c33976b53333caed52c received from {username='slave'} at 127.15.255.130:60007
I20240502 21:28:35.866860 32543 tablet_copy_service.cc:433] P e0a4e8376dae403e9b812a429b8960af: ending tablet copy session c9ab11aee0734f048a27a267e9a34ee4-2531188c066d4c33976b53333caed52c on tablet 2531188c066d4c33976b53333caed52c with peer c9ab11aee0734f048a27a267e9a34ee4
I20240502 21:28:35.867674 875 raft_consensus.cc:348] T 0bf5f9db2efe49029fa1da7399ab5373 P c38cec78fdf74d05b3c75f21eeb8612c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67a81d55b1bd4d2e9b587c6477d624a2" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40197 } } peers { permanent_uuid: "e0a4e8376dae403e9b812a429b8960af" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 42105 } attrs { replace: true } } peers { permanent_uuid: "c38cec78fdf74d05b3c75f21eeb8612c" member_type: NON_VOTER last_known_addr { host: "127.15.255.131" port: 45205 } attrs { promote: true } }
I20240502 21:28:35.867911 875 raft_consensus.cc:729] T 0bf5f9db2efe49029fa1da7399ab5373 P c38cec78fdf74d05b3c75f21eeb8612c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c38cec78fdf74d05b3c75f21eeb8612c, State: Initialized, Role: LEARNER
I20240502 21:28:35.868129 875 consensus_queue.cc:260] T 0bf5f9db2efe49029fa1da7399ab5373 P c38cec78fdf74d05b3c75f21eeb8612c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "67a81d55b1bd4d2e9b587c6477d624a2" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40197 } } peers { permanent_uuid: "e0a4e8376dae403e9b812a429b8960af" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 42105 } attrs { replace: true } } peers { permanent_uuid: "c38cec78fdf74d05b3c75f21eeb8612c" member_type: NON_VOTER last_known_addr { host: "127.15.255.131" port: 45205 } attrs { promote: true } }
I20240502 21:28:35.868714 875 ts_tablet_manager.cc:1398] T 0bf5f9db2efe49029fa1da7399ab5373 P c38cec78fdf74d05b3c75f21eeb8612c: Time spent starting tablet: real 0.002s user 0.003s sys 0.001s
I20240502 21:28:35.869460 32146 tablet_copy_service.cc:342] P 67a81d55b1bd4d2e9b587c6477d624a2: Request end of tablet copy session c38cec78fdf74d05b3c75f21eeb8612c-0bf5f9db2efe49029fa1da7399ab5373 received from {username='slave'} at 127.15.255.131:46775
I20240502 21:28:35.869586 32146 tablet_copy_service.cc:433] P 67a81d55b1bd4d2e9b587c6477d624a2: ending tablet copy session c38cec78fdf74d05b3c75f21eeb8612c-0bf5f9db2efe49029fa1da7399ab5373 on tablet 0bf5f9db2efe49029fa1da7399ab5373 with peer c38cec78fdf74d05b3c75f21eeb8612c
I20240502 21:28:35.982939 394 raft_consensus.cc:1203] T 39d0ac5ec0cd4a33b947de380b5469be P c9ab11aee0734f048a27a267e9a34ee4 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 21:28:36.114948 526 raft_consensus.cc:1203] T d7c55e3c68744428a3bc7f2a1fe50c1f P c38cec78fdf74d05b3c75f21eeb8612c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 21:28:36.167769 394 raft_consensus.cc:1203] T 2531188c066d4c33976b53333caed52c P c9ab11aee0734f048a27a267e9a34ee4 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 21:28:36.197888 526 raft_consensus.cc:1203] T 5b4ff662667a442cabfee1a1818d8dca P c38cec78fdf74d05b3c75f21eeb8612c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 21:28:36.207798 394 raft_consensus.cc:1203] T 4ea23c1a76784237a5619fefa55335fc P c9ab11aee0734f048a27a267e9a34ee4 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 21:28:36.254105 526 raft_consensus.cc:1203] T 0bf5f9db2efe49029fa1da7399ab5373 P c38cec78fdf74d05b3c75f21eeb8612c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 21:28:36.579725 887 raft_consensus.cc:1050] T 39d0ac5ec0cd4a33b947de380b5469be P e0a4e8376dae403e9b812a429b8960af: attempting to promote NON_VOTER c9ab11aee0734f048a27a267e9a34ee4 to VOTER
I20240502 21:28:36.580453 887 consensus_queue.cc:237] T 39d0ac5ec0cd4a33b947de380b5469be P e0a4e8376dae403e9b812a429b8960af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e0a4e8376dae403e9b812a429b8960af" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 42105 } } peers { permanent_uuid: "67a81d55b1bd4d2e9b587c6477d624a2" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40197 } attrs { replace: true } } peers { permanent_uuid: "c9ab11aee0734f048a27a267e9a34ee4" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 34399 } attrs { promote: false } }
I20240502 21:28:36.582247 32124 raft_consensus.cc:1261] T 39d0ac5ec0cd4a33b947de380b5469be P 67a81d55b1bd4d2e9b587c6477d624a2 [term 1 FOLLOWER]: Refusing update from remote peer e0a4e8376dae403e9b812a429b8960af: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:28:36.582353 394 raft_consensus.cc:1261] T 39d0ac5ec0cd4a33b947de380b5469be P c9ab11aee0734f048a27a267e9a34ee4 [term 1 LEARNER]: Refusing update from remote peer e0a4e8376dae403e9b812a429b8960af: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:28:36.582783 733 consensus_queue.cc:1035] T 39d0ac5ec0cd4a33b947de380b5469be P e0a4e8376dae403e9b812a429b8960af [LEADER]: Connected to new peer: Peer: permanent_uuid: "67a81d55b1bd4d2e9b587c6477d624a2" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40197 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:28:36.583112 888 consensus_queue.cc:1035] T 39d0ac5ec0cd4a33b947de380b5469be P e0a4e8376dae403e9b812a429b8960af [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9ab11aee0734f048a27a267e9a34ee4" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 34399 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:28:36.585664 733 raft_consensus.cc:2932] T 39d0ac5ec0cd4a33b947de380b5469be P e0a4e8376dae403e9b812a429b8960af [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c9ab11aee0734f048a27a267e9a34ee4 (127.15.255.130) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e0a4e8376dae403e9b812a429b8960af" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 42105 } } peers { permanent_uuid: "67a81d55b1bd4d2e9b587c6477d624a2" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40197 } attrs { replace: true } } peers { permanent_uuid: "c9ab11aee0734f048a27a267e9a34ee4" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 34399 } attrs { promote: false } } }
I20240502 21:28:36.586338 32124 raft_consensus.cc:2932] T 39d0ac5ec0cd4a33b947de380b5469be P 67a81d55b1bd4d2e9b587c6477d624a2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c9ab11aee0734f048a27a267e9a34ee4 (127.15.255.130) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e0a4e8376dae403e9b812a429b8960af" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 42105 } } peers { permanent_uuid: "67a81d55b1bd4d2e9b587c6477d624a2" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40197 } attrs { replace: true } } peers { permanent_uuid: "c9ab11aee0734f048a27a267e9a34ee4" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 34399 } attrs { promote: false } } }
I20240502 21:28:36.590054 394 raft_consensus.cc:2932] T 39d0ac5ec0cd4a33b947de380b5469be P c9ab11aee0734f048a27a267e9a34ee4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c9ab11aee0734f048a27a267e9a34ee4 (127.15.255.130) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e0a4e8376dae403e9b812a429b8960af" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 42105 } } peers { permanent_uuid: "67a81d55b1bd4d2e9b587c6477d624a2" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40197 } attrs { replace: true } } peers { permanent_uuid: "c9ab11aee0734f048a27a267e9a34ee4" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 34399 } attrs { promote: false } } }
I20240502 21:28:36.590374 31978 catalog_manager.cc:5484] T 39d0ac5ec0cd4a33b947de380b5469be P 67a81d55b1bd4d2e9b587c6477d624a2 reported cstate change: config changed from index 4 to 5, c9ab11aee0734f048a27a267e9a34ee4 (127.15.255.130) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e0a4e8376dae403e9b812a429b8960af" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e0a4e8376dae403e9b812a429b8960af" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 42105 } } peers { permanent_uuid: "67a81d55b1bd4d2e9b587c6477d624a2" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40197 } attrs { replace: true } } peers { permanent_uuid: "c9ab11aee0734f048a27a267e9a34ee4" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 34399 } attrs { promote: false } } }
I20240502 21:28:36.594185 887 raft_consensus.cc:1050] T d7c55e3c68744428a3bc7f2a1fe50c1f P e0a4e8376dae403e9b812a429b8960af: attempting to promote NON_VOTER c38cec78fdf74d05b3c75f21eeb8612c to VOTER
I20240502 21:28:36.594854 887 consensus_queue.cc:237] T d7c55e3c68744428a3bc7f2a1fe50c1f P e0a4e8376dae403e9b812a429b8960af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e0a4e8376dae403e9b812a429b8960af" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 42105 } attrs { replace: true } } peers { permanent_uuid: "67a81d55b1bd4d2e9b587c6477d624a2" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40197 } } peers { permanent_uuid: "c38cec78fdf74d05b3c75f21eeb8612c" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 45205 } attrs { promote: false } }
I20240502 21:28:36.596984 32124 raft_consensus.cc:1261] T d7c55e3c68744428a3bc7f2a1fe50c1f P 67a81d55b1bd4d2e9b587c6477d624a2 [term 1 FOLLOWER]: Refusing update from remote peer e0a4e8376dae403e9b812a429b8960af: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:28:36.597028 526 raft_consensus.cc:1261] T d7c55e3c68744428a3bc7f2a1fe50c1f P c38cec78fdf74d05b3c75f21eeb8612c [term 1 LEARNER]: Refusing update from remote peer e0a4e8376dae403e9b812a429b8960af: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:28:36.597491 887 consensus_queue.cc:1035] T d7c55e3c68744428a3bc7f2a1fe50c1f P e0a4e8376dae403e9b812a429b8960af [LEADER]: Connected to new peer: Peer: permanent_uuid: "67a81d55b1bd4d2e9b587c6477d624a2" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:28:36.598346 887 consensus_queue.cc:1035] T d7c55e3c68744428a3bc7f2a1fe50c1f P e0a4e8376dae403e9b812a429b8960af [LEADER]: Connected to new peer: Peer: permanent_uuid: "c38cec78fdf74d05b3c75f21eeb8612c" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 45205 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:28:36.598534 32523 consensus_queue.cc:237] T 39d0ac5ec0cd4a33b947de380b5469be P e0a4e8376dae403e9b812a429b8960af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e0a4e8376dae403e9b812a429b8960af" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 42105 } } peers { permanent_uuid: "c9ab11aee0734f048a27a267e9a34ee4" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 34399 } attrs { promote: false } }
I20240502 21:28:36.600031 394 raft_consensus.cc:1261] T 39d0ac5ec0cd4a33b947de380b5469be P c9ab11aee0734f048a27a267e9a34ee4 [term 1 FOLLOWER]: Refusing update from remote peer e0a4e8376dae403e9b812a429b8960af: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:28:36.600571 887 consensus_queue.cc:1035] T 39d0ac5ec0cd4a33b947de380b5469be P e0a4e8376dae403e9b812a429b8960af [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9ab11aee0734f048a27a267e9a34ee4" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 34399 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:28:36.600929 733 raft_consensus.cc:2932] T d7c55e3c68744428a3bc7f2a1fe50c1f P e0a4e8376dae403e9b812a429b8960af [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c38cec78fdf74d05b3c75f21eeb8612c (127.15.255.131) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e0a4e8376dae403e9b812a429b8960af" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 42105 } attrs { replace: true } } peers { permanent_uuid: "67a81d55b1bd4d2e9b587c6477d624a2" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40197 } } peers { permanent_uuid: "c38cec78fdf74d05b3c75f21eeb8612c" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 45205 } attrs { promote: false } } }
I20240502 21:28:36.601771 32124 raft_consensus.cc:2932] T d7c55e3c68744428a3bc7f2a1fe50c1f P 67a81d55b1bd4d2e9b587c6477d624a2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c38cec78fdf74d05b3c75f21eeb8612c (127.15.255.131) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e0a4e8376dae403e9b812a429b8960af" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 42105 } attrs { replace: true } } peers { permanent_uuid: "67a81d55b1bd4d2e9b587c6477d624a2" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40197 } } peers { permanent_uuid: "c38cec78fdf74d05b3c75f21eeb8612c" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 45205 } attrs { promote: false } } }
I20240502 21:28:36.604353 526 raft_consensus.cc:2932] T d7c55e3c68744428a3bc7f2a1fe50c1f P c38cec78fdf74d05b3c75f21eeb8612c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c38cec78fdf74d05b3c75f21eeb8612c (127.15.255.131) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e0a4e8376dae403e9b812a429b8960af" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 42105 } attrs { replace: true } } peers { permanent_uuid: "67a81d55b1bd4d2e9b587c6477d624a2" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40197 } } peers { permanent_uuid: "c38cec78fdf74d05b3c75f21eeb8612c" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 45205 } attrs { promote: false } } }
I20240502 21:28:36.605273 733 raft_consensus.cc:2932] T 39d0ac5ec0cd4a33b947de380b5469be P e0a4e8376dae403e9b812a429b8960af [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 67a81d55b1bd4d2e9b587c6477d624a2 (127.15.255.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e0a4e8376dae403e9b812a429b8960af" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 42105 } } peers { permanent_uuid: "c9ab11aee0734f048a27a267e9a34ee4" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 34399 } attrs { promote: false } } }
I20240502 21:28:36.605872 31980 catalog_manager.cc:5484] T d7c55e3c68744428a3bc7f2a1fe50c1f P e0a4e8376dae403e9b812a429b8960af reported cstate change: config changed from index 4 to 5, c38cec78fdf74d05b3c75f21eeb8612c (127.15.255.131) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e0a4e8376dae403e9b812a429b8960af" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e0a4e8376dae403e9b812a429b8960af" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 42105 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "67a81d55b1bd4d2e9b587c6477d624a2" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40197 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c38cec78fdf74d05b3c75f21eeb8612c" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 45205 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:28:36.608479 31967 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 39d0ac5ec0cd4a33b947de380b5469be with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:28:36.609009 394 raft_consensus.cc:2932] T 39d0ac5ec0cd4a33b947de380b5469be P c9ab11aee0734f048a27a267e9a34ee4 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 67a81d55b1bd4d2e9b587c6477d624a2 (127.15.255.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e0a4e8376dae403e9b812a429b8960af" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 42105 } } peers { permanent_uuid: "c9ab11aee0734f048a27a267e9a34ee4" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 34399 } attrs { promote: false } } }
I20240502 21:28:36.611855 31978 catalog_manager.cc:5484] T 39d0ac5ec0cd4a33b947de380b5469be P e0a4e8376dae403e9b812a429b8960af reported cstate change: config changed from index 5 to 6, VOTER 67a81d55b1bd4d2e9b587c6477d624a2 (127.15.255.129) evicted. New cstate: current_term: 1 leader_uuid: "e0a4e8376dae403e9b812a429b8960af" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e0a4e8376dae403e9b812a429b8960af" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 42105 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c9ab11aee0734f048a27a267e9a34ee4" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 34399 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:28:36.615254 32104 tablet_service.cc:1514] Processing DeleteTablet for tablet 39d0ac5ec0cd4a33b947de380b5469be with delete_type TABLET_DATA_TOMBSTONED (TS 67a81d55b1bd4d2e9b587c6477d624a2 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:52164
I20240502 21:28:36.615981 995 tablet_replica.cc:330] T 39d0ac5ec0cd4a33b947de380b5469be P 67a81d55b1bd4d2e9b587c6477d624a2: stopping tablet replica
I20240502 21:28:36.616365 995 raft_consensus.cc:2229] T 39d0ac5ec0cd4a33b947de380b5469be P 67a81d55b1bd4d2e9b587c6477d624a2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:28:36.616818 995 raft_consensus.cc:2258] T 39d0ac5ec0cd4a33b947de380b5469be P 67a81d55b1bd4d2e9b587c6477d624a2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:28:36.617794 994 raft_consensus.cc:1050] T 2531188c066d4c33976b53333caed52c P e0a4e8376dae403e9b812a429b8960af: attempting to promote NON_VOTER c9ab11aee0734f048a27a267e9a34ee4 to VOTER
I20240502 21:28:36.618471 995 ts_tablet_manager.cc:1875] T 39d0ac5ec0cd4a33b947de380b5469be P 67a81d55b1bd4d2e9b587c6477d624a2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:28:36.618505 994 consensus_queue.cc:237] T 2531188c066d4c33976b53333caed52c P e0a4e8376dae403e9b812a429b8960af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "67a81d55b1bd4d2e9b587c6477d624a2" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40197 } attrs { replace: true } } peers { permanent_uuid: "e0a4e8376dae403e9b812a429b8960af" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 42105 } } peers { permanent_uuid: "c9ab11aee0734f048a27a267e9a34ee4" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 34399 } attrs { promote: false } }
I20240502 21:28:36.620481 394 raft_consensus.cc:1261] T 2531188c066d4c33976b53333caed52c P c9ab11aee0734f048a27a267e9a34ee4 [term 1 LEARNER]: Refusing update from remote peer e0a4e8376dae403e9b812a429b8960af: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:28:36.620496 32124 raft_consensus.cc:1261] T 2531188c066d4c33976b53333caed52c P 67a81d55b1bd4d2e9b587c6477d624a2 [term 1 FOLLOWER]: Refusing update from remote peer e0a4e8376dae403e9b812a429b8960af: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:28:36.620983 887 consensus_queue.cc:1035] T 2531188c066d4c33976b53333caed52c P e0a4e8376dae403e9b812a429b8960af [LEADER]: Connected to new peer: Peer: permanent_uuid: "67a81d55b1bd4d2e9b587c6477d624a2" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40197 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:28:36.621245 733 consensus_queue.cc:1035] T 2531188c066d4c33976b53333caed52c P e0a4e8376dae403e9b812a429b8960af [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9ab11aee0734f048a27a267e9a34ee4" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 34399 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:28:36.623863 733 raft_consensus.cc:2932] T 2531188c066d4c33976b53333caed52c P e0a4e8376dae403e9b812a429b8960af [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c9ab11aee0734f048a27a267e9a34ee4 (127.15.255.130) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "67a81d55b1bd4d2e9b587c6477d624a2" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40197 } attrs { replace: true } } peers { permanent_uuid: "e0a4e8376dae403e9b812a429b8960af" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 42105 } } peers { permanent_uuid: "c9ab11aee0734f048a27a267e9a34ee4" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 34399 } attrs { promote: false } } }
I20240502 21:28:36.624581 32124 raft_consensus.cc:2932] T 2531188c066d4c33976b53333caed52c P 67a81d55b1bd4d2e9b587c6477d624a2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c9ab11aee0734f048a27a267e9a34ee4 (127.15.255.130) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "67a81d55b1bd4d2e9b587c6477d624a2" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40197 } attrs { replace: true } } peers { permanent_uuid: "e0a4e8376dae403e9b812a429b8960af" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 42105 } } peers { permanent_uuid: "c9ab11aee0734f048a27a267e9a34ee4" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 34399 } attrs { promote: false } } }
I20240502 21:28:36.625650 995 ts_tablet_manager.cc:1888] T 39d0ac5ec0cd4a33b947de380b5469be P 67a81d55b1bd4d2e9b587c6477d624a2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 21:28:36.625795 995 log.cc:1196] T 39d0ac5ec0cd4a33b947de380b5469be P 67a81d55b1bd4d2e9b587c6477d624a2: Deleting WAL directory at /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-0/wal/wals/39d0ac5ec0cd4a33b947de380b5469be
I20240502 21:28:36.626730 31965 catalog_manager.cc:4830] TS 67a81d55b1bd4d2e9b587c6477d624a2 (127.15.255.129:40197): tablet 39d0ac5ec0cd4a33b947de380b5469be (table rebalance_test_table_1 [id=ec8886b062014ed78b9bf3e9a156193a]) successfully deleted
I20240502 21:28:36.628257 31978 catalog_manager.cc:5484] T 2531188c066d4c33976b53333caed52c P e0a4e8376dae403e9b812a429b8960af reported cstate change: config changed from index 4 to 5, c9ab11aee0734f048a27a267e9a34ee4 (127.15.255.130) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e0a4e8376dae403e9b812a429b8960af" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "67a81d55b1bd4d2e9b587c6477d624a2" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40197 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e0a4e8376dae403e9b812a429b8960af" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 42105 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c9ab11aee0734f048a27a267e9a34ee4" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 34399 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 21:28:36.637290 394 raft_consensus.cc:2932] T 2531188c066d4c33976b53333caed52c P c9ab11aee0734f048a27a267e9a34ee4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c9ab11aee0734f048a27a267e9a34ee4 (127.15.255.130) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "67a81d55b1bd4d2e9b587c6477d624a2" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40197 } attrs { replace: true } } peers { permanent_uuid: "e0a4e8376dae403e9b812a429b8960af" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 42105 } } peers { permanent_uuid: "c9ab11aee0734f048a27a267e9a34ee4" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 34399 } attrs { promote: false } } }
I20240502 21:28:36.639562 32523 consensus_queue.cc:237] T 2531188c066d4c33976b53333caed52c P e0a4e8376dae403e9b812a429b8960af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e0a4e8376dae403e9b812a429b8960af" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 42105 } } peers { permanent_uuid: "c9ab11aee0734f048a27a267e9a34ee4" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 34399 } attrs { promote: false } }
I20240502 21:28:36.641009 394 raft_consensus.cc:1261] T 2531188c066d4c33976b53333caed52c P c9ab11aee0734f048a27a267e9a34ee4 [term 1 FOLLOWER]: Refusing update from remote peer e0a4e8376dae403e9b812a429b8960af: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:28:36.641477 733 consensus_queue.cc:1035] T 2531188c066d4c33976b53333caed52c P e0a4e8376dae403e9b812a429b8960af [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9ab11aee0734f048a27a267e9a34ee4" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 34399 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:28:36.643291 887 raft_consensus.cc:2932] T 2531188c066d4c33976b53333caed52c P e0a4e8376dae403e9b812a429b8960af [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 67a81d55b1bd4d2e9b587c6477d624a2 (127.15.255.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e0a4e8376dae403e9b812a429b8960af" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 42105 } } peers { permanent_uuid: "c9ab11aee0734f048a27a267e9a34ee4" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 34399 } attrs { promote: false } } }
I20240502 21:28:36.643893 394 raft_consensus.cc:2932] T 2531188c066d4c33976b53333caed52c P c9ab11aee0734f048a27a267e9a34ee4 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 67a81d55b1bd4d2e9b587c6477d624a2 (127.15.255.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e0a4e8376dae403e9b812a429b8960af" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 42105 } } peers { permanent_uuid: "c9ab11aee0734f048a27a267e9a34ee4" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 34399 } attrs { promote: false } } }
I20240502 21:28:36.645756 31967 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 2531188c066d4c33976b53333caed52c with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:28:36.647073 31981 catalog_manager.cc:5484] T 2531188c066d4c33976b53333caed52c P e0a4e8376dae403e9b812a429b8960af reported cstate change: config changed from index 5 to 6, VOTER 67a81d55b1bd4d2e9b587c6477d624a2 (127.15.255.129) evicted. New cstate: current_term: 1 leader_uuid: "e0a4e8376dae403e9b812a429b8960af" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e0a4e8376dae403e9b812a429b8960af" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 42105 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c9ab11aee0734f048a27a267e9a34ee4" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 34399 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:28:36.649745 32104 tablet_service.cc:1514] Processing DeleteTablet for tablet 2531188c066d4c33976b53333caed52c with delete_type TABLET_DATA_TOMBSTONED (TS 67a81d55b1bd4d2e9b587c6477d624a2 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:52164
I20240502 21:28:36.650049 995 tablet_replica.cc:330] T 2531188c066d4c33976b53333caed52c P 67a81d55b1bd4d2e9b587c6477d624a2: stopping tablet replica
I20240502 21:28:36.650251 995 raft_consensus.cc:2229] T 2531188c066d4c33976b53333caed52c P 67a81d55b1bd4d2e9b587c6477d624a2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:28:36.650561 995 raft_consensus.cc:2258] T 2531188c066d4c33976b53333caed52c P 67a81d55b1bd4d2e9b587c6477d624a2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:28:36.651321 995 ts_tablet_manager.cc:1875] T 2531188c066d4c33976b53333caed52c P 67a81d55b1bd4d2e9b587c6477d624a2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:28:36.655493 995 ts_tablet_manager.cc:1888] T 2531188c066d4c33976b53333caed52c P 67a81d55b1bd4d2e9b587c6477d624a2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 21:28:36.655586 995 log.cc:1196] T 2531188c066d4c33976b53333caed52c P 67a81d55b1bd4d2e9b587c6477d624a2: Deleting WAL directory at /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-0/wal/wals/2531188c066d4c33976b53333caed52c
I20240502 21:28:36.656177 31965 catalog_manager.cc:4830] TS 67a81d55b1bd4d2e9b587c6477d624a2 (127.15.255.129:40197): tablet 2531188c066d4c33976b53333caed52c (table rebalance_test_table_0 [id=7b825a608b1348f09ea4d42250b107e2]) successfully deleted
I20240502 21:28:36.658636 921 raft_consensus.cc:1050] T 0bf5f9db2efe49029fa1da7399ab5373 P 67a81d55b1bd4d2e9b587c6477d624a2: attempting to promote NON_VOTER c38cec78fdf74d05b3c75f21eeb8612c to VOTER
I20240502 21:28:36.659327 921 consensus_queue.cc:237] T 0bf5f9db2efe49029fa1da7399ab5373 P 67a81d55b1bd4d2e9b587c6477d624a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "67a81d55b1bd4d2e9b587c6477d624a2" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40197 } } peers { permanent_uuid: "e0a4e8376dae403e9b812a429b8960af" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 42105 } attrs { replace: true } } peers { permanent_uuid: "c38cec78fdf74d05b3c75f21eeb8612c" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 45205 } attrs { promote: false } }
I20240502 21:28:36.661131 32523 raft_consensus.cc:1261] T 0bf5f9db2efe49029fa1da7399ab5373 P e0a4e8376dae403e9b812a429b8960af [term 1 FOLLOWER]: Refusing update from remote peer 67a81d55b1bd4d2e9b587c6477d624a2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:28:36.661170 526 raft_consensus.cc:1261] T 0bf5f9db2efe49029fa1da7399ab5373 P c38cec78fdf74d05b3c75f21eeb8612c [term 1 LEARNER]: Refusing update from remote peer 67a81d55b1bd4d2e9b587c6477d624a2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:28:36.661697 922 consensus_queue.cc:1035] T 0bf5f9db2efe49029fa1da7399ab5373 P 67a81d55b1bd4d2e9b587c6477d624a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0a4e8376dae403e9b812a429b8960af" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 42105 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:28:36.662034 708 consensus_queue.cc:1035] T 0bf5f9db2efe49029fa1da7399ab5373 P 67a81d55b1bd4d2e9b587c6477d624a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c38cec78fdf74d05b3c75f21eeb8612c" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 45205 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:28:36.664327 708 raft_consensus.cc:2932] T 0bf5f9db2efe49029fa1da7399ab5373 P 67a81d55b1bd4d2e9b587c6477d624a2 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c38cec78fdf74d05b3c75f21eeb8612c (127.15.255.131) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "67a81d55b1bd4d2e9b587c6477d624a2" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40197 } } peers { permanent_uuid: "e0a4e8376dae403e9b812a429b8960af" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 42105 } attrs { replace: true } } peers { permanent_uuid: "c38cec78fdf74d05b3c75f21eeb8612c" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 45205 } attrs { promote: false } } }
I20240502 21:28:36.665144 32523 raft_consensus.cc:2932] T 0bf5f9db2efe49029fa1da7399ab5373 P e0a4e8376dae403e9b812a429b8960af [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c38cec78fdf74d05b3c75f21eeb8612c (127.15.255.131) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "67a81d55b1bd4d2e9b587c6477d624a2" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40197 } } peers { permanent_uuid: "e0a4e8376dae403e9b812a429b8960af" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 42105 } attrs { replace: true } } peers { permanent_uuid: "c38cec78fdf74d05b3c75f21eeb8612c" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 45205 } attrs { promote: false } } }
I20240502 21:28:36.665186 526 raft_consensus.cc:2932] T 0bf5f9db2efe49029fa1da7399ab5373 P c38cec78fdf74d05b3c75f21eeb8612c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c38cec78fdf74d05b3c75f21eeb8612c (127.15.255.131) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "67a81d55b1bd4d2e9b587c6477d624a2" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40197 } } peers { permanent_uuid: "e0a4e8376dae403e9b812a429b8960af" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 42105 } attrs { replace: true } } peers { permanent_uuid: "c38cec78fdf74d05b3c75f21eeb8612c" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 45205 } attrs { promote: false } } }
I20240502 21:28:36.668043 31978 catalog_manager.cc:5484] T 0bf5f9db2efe49029fa1da7399ab5373 P 67a81d55b1bd4d2e9b587c6477d624a2 reported cstate change: config changed from index 4 to 5, c38cec78fdf74d05b3c75f21eeb8612c (127.15.255.131) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "67a81d55b1bd4d2e9b587c6477d624a2" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "67a81d55b1bd4d2e9b587c6477d624a2" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40197 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e0a4e8376dae403e9b812a429b8960af" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 42105 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c38cec78fdf74d05b3c75f21eeb8612c" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 45205 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:28:36.672448 32124 consensus_queue.cc:237] T 0bf5f9db2efe49029fa1da7399ab5373 P 67a81d55b1bd4d2e9b587c6477d624a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "67a81d55b1bd4d2e9b587c6477d624a2" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40197 } } peers { permanent_uuid: "c38cec78fdf74d05b3c75f21eeb8612c" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 45205 } attrs { promote: false } }
I20240502 21:28:36.673772 526 raft_consensus.cc:1261] T 0bf5f9db2efe49029fa1da7399ab5373 P c38cec78fdf74d05b3c75f21eeb8612c [term 1 FOLLOWER]: Refusing update from remote peer 67a81d55b1bd4d2e9b587c6477d624a2: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:28:36.674199 920 consensus_queue.cc:1035] T 0bf5f9db2efe49029fa1da7399ab5373 P 67a81d55b1bd4d2e9b587c6477d624a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c38cec78fdf74d05b3c75f21eeb8612c" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 45205 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:28:36.675876 921 raft_consensus.cc:2932] T 0bf5f9db2efe49029fa1da7399ab5373 P 67a81d55b1bd4d2e9b587c6477d624a2 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e0a4e8376dae403e9b812a429b8960af (127.15.255.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "67a81d55b1bd4d2e9b587c6477d624a2" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40197 } } peers { permanent_uuid: "c38cec78fdf74d05b3c75f21eeb8612c" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 45205 } attrs { promote: false } } }
I20240502 21:28:36.676463 526 raft_consensus.cc:2932] T 0bf5f9db2efe49029fa1da7399ab5373 P c38cec78fdf74d05b3c75f21eeb8612c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e0a4e8376dae403e9b812a429b8960af (127.15.255.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "67a81d55b1bd4d2e9b587c6477d624a2" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40197 } } peers { permanent_uuid: "c38cec78fdf74d05b3c75f21eeb8612c" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 45205 } attrs { promote: false } } }
I20240502 21:28:36.679275 31965 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 0bf5f9db2efe49029fa1da7399ab5373 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:28:36.680370 31980 catalog_manager.cc:5484] T 0bf5f9db2efe49029fa1da7399ab5373 P 67a81d55b1bd4d2e9b587c6477d624a2 reported cstate change: config changed from index 5 to 6, VOTER e0a4e8376dae403e9b812a429b8960af (127.15.255.132) evicted. New cstate: current_term: 1 leader_uuid: "67a81d55b1bd4d2e9b587c6477d624a2" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "67a81d55b1bd4d2e9b587c6477d624a2" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40197 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c38cec78fdf74d05b3c75f21eeb8612c" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 45205 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:28:36.683261 32503 tablet_service.cc:1514] Processing DeleteTablet for tablet 0bf5f9db2efe49029fa1da7399ab5373 with delete_type TABLET_DATA_TOMBSTONED (TS e0a4e8376dae403e9b812a429b8960af not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:56374
I20240502 21:28:36.684034 1002 tablet_replica.cc:330] T 0bf5f9db2efe49029fa1da7399ab5373 P e0a4e8376dae403e9b812a429b8960af: stopping tablet replica
I20240502 21:28:36.684396 1002 raft_consensus.cc:2229] T 0bf5f9db2efe49029fa1da7399ab5373 P e0a4e8376dae403e9b812a429b8960af [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:28:36.684734 1002 raft_consensus.cc:2258] T 0bf5f9db2efe49029fa1da7399ab5373 P e0a4e8376dae403e9b812a429b8960af [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:28:36.686307 1002 ts_tablet_manager.cc:1875] T 0bf5f9db2efe49029fa1da7399ab5373 P e0a4e8376dae403e9b812a429b8960af: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:28:36.690729 1002 ts_tablet_manager.cc:1888] T 0bf5f9db2efe49029fa1da7399ab5373 P e0a4e8376dae403e9b812a429b8960af: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 21:28:36.690855 1002 log.cc:1196] T 0bf5f9db2efe49029fa1da7399ab5373 P e0a4e8376dae403e9b812a429b8960af: Deleting WAL directory at /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-3/wal/wals/0bf5f9db2efe49029fa1da7399ab5373
I20240502 21:28:36.691520 31967 catalog_manager.cc:4830] TS e0a4e8376dae403e9b812a429b8960af (127.15.255.132:42105): tablet 0bf5f9db2efe49029fa1da7399ab5373 (table rebalance_test_table_1 [id=ec8886b062014ed78b9bf3e9a156193a]) successfully deleted
I20240502 21:28:36.809196 733 raft_consensus.cc:1050] T 4ea23c1a76784237a5619fefa55335fc P e0a4e8376dae403e9b812a429b8960af: attempting to promote NON_VOTER c9ab11aee0734f048a27a267e9a34ee4 to VOTER
I20240502 21:28:36.810012 733 consensus_queue.cc:237] T 4ea23c1a76784237a5619fefa55335fc P e0a4e8376dae403e9b812a429b8960af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e0a4e8376dae403e9b812a429b8960af" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 42105 } } peers { permanent_uuid: "67a81d55b1bd4d2e9b587c6477d624a2" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40197 } attrs { replace: true } } peers { permanent_uuid: "c9ab11aee0734f048a27a267e9a34ee4" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 34399 } attrs { promote: false } }
I20240502 21:28:36.812399 32124 raft_consensus.cc:1261] T 4ea23c1a76784237a5619fefa55335fc P 67a81d55b1bd4d2e9b587c6477d624a2 [term 1 FOLLOWER]: Refusing update from remote peer e0a4e8376dae403e9b812a429b8960af: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:28:36.812474 394 raft_consensus.cc:1261] T 4ea23c1a76784237a5619fefa55335fc P c9ab11aee0734f048a27a267e9a34ee4 [term 1 LEARNER]: Refusing update from remote peer e0a4e8376dae403e9b812a429b8960af: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:28:36.812988 887 consensus_queue.cc:1035] T 4ea23c1a76784237a5619fefa55335fc P e0a4e8376dae403e9b812a429b8960af [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9ab11aee0734f048a27a267e9a34ee4" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 34399 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:28:36.813565 921 raft_consensus.cc:1050] T 5b4ff662667a442cabfee1a1818d8dca P 67a81d55b1bd4d2e9b587c6477d624a2: attempting to promote NON_VOTER c38cec78fdf74d05b3c75f21eeb8612c to VOTER
I20240502 21:28:36.813614 994 consensus_queue.cc:1035] T 4ea23c1a76784237a5619fefa55335fc P e0a4e8376dae403e9b812a429b8960af [LEADER]: Connected to new peer: Peer: permanent_uuid: "67a81d55b1bd4d2e9b587c6477d624a2" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40197 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:28:36.814230 921 consensus_queue.cc:237] T 5b4ff662667a442cabfee1a1818d8dca P 67a81d55b1bd4d2e9b587c6477d624a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e0a4e8376dae403e9b812a429b8960af" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 42105 } attrs { replace: true } } peers { permanent_uuid: "67a81d55b1bd4d2e9b587c6477d624a2" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40197 } } peers { permanent_uuid: "c38cec78fdf74d05b3c75f21eeb8612c" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 45205 } attrs { promote: false } }
I20240502 21:28:36.816597 32523 raft_consensus.cc:1261] T 5b4ff662667a442cabfee1a1818d8dca P e0a4e8376dae403e9b812a429b8960af [term 1 FOLLOWER]: Refusing update from remote peer 67a81d55b1bd4d2e9b587c6477d624a2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:28:36.816586 526 raft_consensus.cc:1261] T 5b4ff662667a442cabfee1a1818d8dca P c38cec78fdf74d05b3c75f21eeb8612c [term 1 LEARNER]: Refusing update from remote peer 67a81d55b1bd4d2e9b587c6477d624a2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:28:36.816391 994 raft_consensus.cc:2932] T 4ea23c1a76784237a5619fefa55335fc P e0a4e8376dae403e9b812a429b8960af [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c9ab11aee0734f048a27a267e9a34ee4 (127.15.255.130) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e0a4e8376dae403e9b812a429b8960af" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 42105 } } peers { permanent_uuid: "67a81d55b1bd4d2e9b587c6477d624a2" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40197 } attrs { replace: true } } peers { permanent_uuid: "c9ab11aee0734f048a27a267e9a34ee4" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 34399 } attrs { promote: false } } }
I20240502 21:28:36.817371 708 consensus_queue.cc:1035] T 5b4ff662667a442cabfee1a1818d8dca P 67a81d55b1bd4d2e9b587c6477d624a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0a4e8376dae403e9b812a429b8960af" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 42105 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:28:36.817934 708 consensus_queue.cc:1035] T 5b4ff662667a442cabfee1a1818d8dca P 67a81d55b1bd4d2e9b587c6477d624a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c38cec78fdf74d05b3c75f21eeb8612c" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 45205 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:28:36.817679 394 raft_consensus.cc:2932] T 4ea23c1a76784237a5619fefa55335fc P c9ab11aee0734f048a27a267e9a34ee4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c9ab11aee0734f048a27a267e9a34ee4 (127.15.255.130) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e0a4e8376dae403e9b812a429b8960af" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 42105 } } peers { permanent_uuid: "67a81d55b1bd4d2e9b587c6477d624a2" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40197 } attrs { replace: true } } peers { permanent_uuid: "c9ab11aee0734f048a27a267e9a34ee4" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 34399 } attrs { promote: false } } }
I20240502 21:28:36.821887 31981 catalog_manager.cc:5484] T 4ea23c1a76784237a5619fefa55335fc P e0a4e8376dae403e9b812a429b8960af reported cstate change: config changed from index 4 to 5, c9ab11aee0734f048a27a267e9a34ee4 (127.15.255.130) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e0a4e8376dae403e9b812a429b8960af" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e0a4e8376dae403e9b812a429b8960af" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 42105 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "67a81d55b1bd4d2e9b587c6477d624a2" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40197 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9ab11aee0734f048a27a267e9a34ee4" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 34399 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:28:36.822173 920 raft_consensus.cc:2932] T 5b4ff662667a442cabfee1a1818d8dca P 67a81d55b1bd4d2e9b587c6477d624a2 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c38cec78fdf74d05b3c75f21eeb8612c (127.15.255.131) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e0a4e8376dae403e9b812a429b8960af" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 42105 } attrs { replace: true } } peers { permanent_uuid: "67a81d55b1bd4d2e9b587c6477d624a2" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40197 } } peers { permanent_uuid: "c38cec78fdf74d05b3c75f21eeb8612c" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 45205 } attrs { promote: false } } }
I20240502 21:28:36.828598 32522 consensus_queue.cc:237] T 4ea23c1a76784237a5619fefa55335fc P e0a4e8376dae403e9b812a429b8960af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e0a4e8376dae403e9b812a429b8960af" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 42105 } } peers { permanent_uuid: "c9ab11aee0734f048a27a267e9a34ee4" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 34399 } attrs { promote: false } }
I20240502 21:28:36.829295 526 raft_consensus.cc:2932] T 5b4ff662667a442cabfee1a1818d8dca P c38cec78fdf74d05b3c75f21eeb8612c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c38cec78fdf74d05b3c75f21eeb8612c (127.15.255.131) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e0a4e8376dae403e9b812a429b8960af" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 42105 } attrs { replace: true } } peers { permanent_uuid: "67a81d55b1bd4d2e9b587c6477d624a2" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40197 } } peers { permanent_uuid: "c38cec78fdf74d05b3c75f21eeb8612c" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 45205 } attrs { promote: false } } }
I20240502 21:28:36.830598 394 raft_consensus.cc:1261] T 4ea23c1a76784237a5619fefa55335fc P c9ab11aee0734f048a27a267e9a34ee4 [term 1 FOLLOWER]: Refusing update from remote peer e0a4e8376dae403e9b812a429b8960af: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:28:36.831126 994 consensus_queue.cc:1035] T 4ea23c1a76784237a5619fefa55335fc P e0a4e8376dae403e9b812a429b8960af [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9ab11aee0734f048a27a267e9a34ee4" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 34399 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:28:36.833388 994 raft_consensus.cc:2932] T 4ea23c1a76784237a5619fefa55335fc P e0a4e8376dae403e9b812a429b8960af [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 67a81d55b1bd4d2e9b587c6477d624a2 (127.15.255.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e0a4e8376dae403e9b812a429b8960af" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 42105 } } peers { permanent_uuid: "c9ab11aee0734f048a27a267e9a34ee4" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 34399 } attrs { promote: false } } }
I20240502 21:28:36.837106 32523 raft_consensus.cc:2932] T 5b4ff662667a442cabfee1a1818d8dca P e0a4e8376dae403e9b812a429b8960af [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c38cec78fdf74d05b3c75f21eeb8612c (127.15.255.131) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e0a4e8376dae403e9b812a429b8960af" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 42105 } attrs { replace: true } } peers { permanent_uuid: "67a81d55b1bd4d2e9b587c6477d624a2" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40197 } } peers { permanent_uuid: "c38cec78fdf74d05b3c75f21eeb8612c" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 45205 } attrs { promote: false } } }
I20240502 21:28:36.838232 394 raft_consensus.cc:2932] T 4ea23c1a76784237a5619fefa55335fc P c9ab11aee0734f048a27a267e9a34ee4 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 67a81d55b1bd4d2e9b587c6477d624a2 (127.15.255.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e0a4e8376dae403e9b812a429b8960af" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 42105 } } peers { permanent_uuid: "c9ab11aee0734f048a27a267e9a34ee4" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 34399 } attrs { promote: false } } }
I20240502 21:28:36.841041 31967 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 4ea23c1a76784237a5619fefa55335fc with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:28:36.842727 31980 catalog_manager.cc:5484] T 5b4ff662667a442cabfee1a1818d8dca P 67a81d55b1bd4d2e9b587c6477d624a2 reported cstate change: config changed from index 4 to 5, c38cec78fdf74d05b3c75f21eeb8612c (127.15.255.131) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "67a81d55b1bd4d2e9b587c6477d624a2" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e0a4e8376dae403e9b812a429b8960af" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 42105 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "67a81d55b1bd4d2e9b587c6477d624a2" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40197 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c38cec78fdf74d05b3c75f21eeb8612c" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 45205 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:28:36.842888 31978 catalog_manager.cc:5484] T 4ea23c1a76784237a5619fefa55335fc P e0a4e8376dae403e9b812a429b8960af reported cstate change: config changed from index 5 to 6, VOTER 67a81d55b1bd4d2e9b587c6477d624a2 (127.15.255.129) evicted. New cstate: current_term: 1 leader_uuid: "e0a4e8376dae403e9b812a429b8960af" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e0a4e8376dae403e9b812a429b8960af" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 42105 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c9ab11aee0734f048a27a267e9a34ee4" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 34399 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:28:36.846841 32124 consensus_queue.cc:237] T 5b4ff662667a442cabfee1a1818d8dca P 67a81d55b1bd4d2e9b587c6477d624a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "67a81d55b1bd4d2e9b587c6477d624a2" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40197 } } peers { permanent_uuid: "c38cec78fdf74d05b3c75f21eeb8612c" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 45205 } attrs { promote: false } }
I20240502 21:28:36.847648 32104 tablet_service.cc:1514] Processing DeleteTablet for tablet 4ea23c1a76784237a5619fefa55335fc with delete_type TABLET_DATA_TOMBSTONED (TS 67a81d55b1bd4d2e9b587c6477d624a2 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:52164
I20240502 21:28:36.848057 995 tablet_replica.cc:330] T 4ea23c1a76784237a5619fefa55335fc P 67a81d55b1bd4d2e9b587c6477d624a2: stopping tablet replica
I20240502 21:28:36.848278 995 raft_consensus.cc:2229] T 4ea23c1a76784237a5619fefa55335fc P 67a81d55b1bd4d2e9b587c6477d624a2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:28:36.848593 995 pending_rounds.cc:62] T 4ea23c1a76784237a5619fefa55335fc P 67a81d55b1bd4d2e9b587c6477d624a2: Trying to abort 1 pending ops.
I20240502 21:28:36.848704 995 pending_rounds.cc:68] T 4ea23c1a76784237a5619fefa55335fc P 67a81d55b1bd4d2e9b587c6477d624a2: Aborting op as it isn't in flight: id { term: 1 index: 5 } timestamp: 7023351057651875840 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "4ea23c1a76784237a5619fefa55335fc" old_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e0a4e8376dae403e9b812a429b8960af" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 42105 } } peers { permanent_uuid: "67a81d55b1bd4d2e9b587c6477d624a2" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40197 } attrs { replace: true } } peers { permanent_uuid: "c9ab11aee0734f048a27a267e9a34ee4" member_type: NON_VOTER last_known_addr { host: "127.15.255.130" port: 34399 } attrs { promote: true } } } new_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e0a4e8376dae403e9b812a429b8960af" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 42105 } } peers { permanent_uuid: "67a81d55b1bd4d2e9b587c6477d624a2" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40197 } attrs { replace: true } } peers { permanent_uuid: "c9ab11aee0734f048a27a267e9a34ee4" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 34399 } attrs { promote: false } } } }
I20240502 21:28:36.849094 995 raft_consensus.cc:2894] T 4ea23c1a76784237a5619fefa55335fc P 67a81d55b1bd4d2e9b587c6477d624a2 [term 1 FOLLOWER]: Aborting config change with OpId 1.5: Aborted: Op aborted
I20240502 21:28:36.849196 995 raft_consensus.cc:2866] T 4ea23c1a76784237a5619fefa55335fc P 67a81d55b1bd4d2e9b587c6477d624a2 [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240502 21:28:36.849300 995 raft_consensus.cc:2258] T 4ea23c1a76784237a5619fefa55335fc P 67a81d55b1bd4d2e9b587c6477d624a2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:28:36.849896 525 raft_consensus.cc:1261] T 5b4ff662667a442cabfee1a1818d8dca P c38cec78fdf74d05b3c75f21eeb8612c [term 1 FOLLOWER]: Refusing update from remote peer 67a81d55b1bd4d2e9b587c6477d624a2: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:28:36.850658 920 consensus_queue.cc:1035] T 5b4ff662667a442cabfee1a1818d8dca P 67a81d55b1bd4d2e9b587c6477d624a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c38cec78fdf74d05b3c75f21eeb8612c" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 45205 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:28:36.851601 995 ts_tablet_manager.cc:1875] T 4ea23c1a76784237a5619fefa55335fc P 67a81d55b1bd4d2e9b587c6477d624a2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:28:36.853217 708 raft_consensus.cc:2932] T 5b4ff662667a442cabfee1a1818d8dca P 67a81d55b1bd4d2e9b587c6477d624a2 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e0a4e8376dae403e9b812a429b8960af (127.15.255.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "67a81d55b1bd4d2e9b587c6477d624a2" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40197 } } peers { permanent_uuid: "c38cec78fdf74d05b3c75f21eeb8612c" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 45205 } attrs { promote: false } } }
I20240502 21:28:36.856043 31965 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 5b4ff662667a442cabfee1a1818d8dca with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:28:36.856943 525 raft_consensus.cc:2932] T 5b4ff662667a442cabfee1a1818d8dca P c38cec78fdf74d05b3c75f21eeb8612c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e0a4e8376dae403e9b812a429b8960af (127.15.255.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "67a81d55b1bd4d2e9b587c6477d624a2" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40197 } } peers { permanent_uuid: "c38cec78fdf74d05b3c75f21eeb8612c" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 45205 } attrs { promote: false } } }
I20240502 21:28:36.858814 995 ts_tablet_manager.cc:1888] T 4ea23c1a76784237a5619fefa55335fc P 67a81d55b1bd4d2e9b587c6477d624a2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 21:28:36.858981 995 log.cc:1196] T 4ea23c1a76784237a5619fefa55335fc P 67a81d55b1bd4d2e9b587c6477d624a2: Deleting WAL directory at /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-0/wal/wals/4ea23c1a76784237a5619fefa55335fc
I20240502 21:28:36.858796 31980 catalog_manager.cc:5484] T 5b4ff662667a442cabfee1a1818d8dca P 67a81d55b1bd4d2e9b587c6477d624a2 reported cstate change: config changed from index 5 to 6, VOTER e0a4e8376dae403e9b812a429b8960af (127.15.255.132) evicted. New cstate: current_term: 1 leader_uuid: "67a81d55b1bd4d2e9b587c6477d624a2" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "67a81d55b1bd4d2e9b587c6477d624a2" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40197 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c38cec78fdf74d05b3c75f21eeb8612c" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 45205 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:28:36.859772 31965 catalog_manager.cc:4830] TS 67a81d55b1bd4d2e9b587c6477d624a2 (127.15.255.129:40197): tablet 4ea23c1a76784237a5619fefa55335fc (table rebalance_test_table_2 [id=b49ef2928b8449da9a95493239d775d2]) successfully deleted
I20240502 21:28:36.862186 32503 tablet_service.cc:1514] Processing DeleteTablet for tablet 5b4ff662667a442cabfee1a1818d8dca with delete_type TABLET_DATA_TOMBSTONED (TS e0a4e8376dae403e9b812a429b8960af not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:56374
I20240502 21:28:36.862509 1002 tablet_replica.cc:330] T 5b4ff662667a442cabfee1a1818d8dca P e0a4e8376dae403e9b812a429b8960af: stopping tablet replica
I20240502 21:28:36.862699 1002 raft_consensus.cc:2229] T 5b4ff662667a442cabfee1a1818d8dca P e0a4e8376dae403e9b812a429b8960af [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:28:36.862926 1002 raft_consensus.cc:2258] T 5b4ff662667a442cabfee1a1818d8dca P e0a4e8376dae403e9b812a429b8960af [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:28:36.863827 1002 ts_tablet_manager.cc:1875] T 5b4ff662667a442cabfee1a1818d8dca P e0a4e8376dae403e9b812a429b8960af: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:28:36.866922 32523 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d7c55e3c68744428a3bc7f2a1fe50c1f"
dest_uuid: "e0a4e8376dae403e9b812a429b8960af"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:56688
I20240502 21:28:36.867162 32523 raft_consensus.cc:595] T d7c55e3c68744428a3bc7f2a1fe50c1f P e0a4e8376dae403e9b812a429b8960af [term 1 LEADER]: Received request to transfer leadership
I20240502 21:28:36.870575 1002 ts_tablet_manager.cc:1888] T 5b4ff662667a442cabfee1a1818d8dca P e0a4e8376dae403e9b812a429b8960af: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 21:28:36.870677 1002 log.cc:1196] T 5b4ff662667a442cabfee1a1818d8dca P e0a4e8376dae403e9b812a429b8960af: Deleting WAL directory at /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-3/wal/wals/5b4ff662667a442cabfee1a1818d8dca
I20240502 21:28:36.871305 31967 catalog_manager.cc:4830] TS e0a4e8376dae403e9b812a429b8960af (127.15.255.132:42105): tablet 5b4ff662667a442cabfee1a1818d8dca (table rebalance_test_table_2 [id=b49ef2928b8449da9a95493239d775d2]) successfully deleted
I20240502 21:28:36.905822 32523 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d7c55e3c68744428a3bc7f2a1fe50c1f"
dest_uuid: "e0a4e8376dae403e9b812a429b8960af"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:56730
I20240502 21:28:36.905980 32523 raft_consensus.cc:595] T d7c55e3c68744428a3bc7f2a1fe50c1f P e0a4e8376dae403e9b812a429b8960af [term 1 LEADER]: Received request to transfer leadership
I20240502 21:28:36.920233 32523 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d7c55e3c68744428a3bc7f2a1fe50c1f"
dest_uuid: "e0a4e8376dae403e9b812a429b8960af"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:56756
I20240502 21:28:36.920382 32523 raft_consensus.cc:595] T d7c55e3c68744428a3bc7f2a1fe50c1f P e0a4e8376dae403e9b812a429b8960af [term 1 LEADER]: Received request to transfer leadership
I20240502 21:28:37.024191 887 raft_consensus.cc:979] T d7c55e3c68744428a3bc7f2a1fe50c1f P e0a4e8376dae403e9b812a429b8960af: : Instructing follower 67a81d55b1bd4d2e9b587c6477d624a2 to start an election
I20240502 21:28:37.024355 887 raft_consensus.cc:1067] T d7c55e3c68744428a3bc7f2a1fe50c1f P e0a4e8376dae403e9b812a429b8960af [term 1 LEADER]: Signalling peer 67a81d55b1bd4d2e9b587c6477d624a2 to start an election
I20240502 21:28:37.025177 32124 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d7c55e3c68744428a3bc7f2a1fe50c1f"
dest_uuid: "67a81d55b1bd4d2e9b587c6477d624a2"
from {username='slave'} at 127.15.255.132:59363
I20240502 21:28:37.025393 32124 raft_consensus.cc:482] T d7c55e3c68744428a3bc7f2a1fe50c1f P 67a81d55b1bd4d2e9b587c6477d624a2 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 21:28:37.025501 32124 raft_consensus.cc:3037] T d7c55e3c68744428a3bc7f2a1fe50c1f P 67a81d55b1bd4d2e9b587c6477d624a2 [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:28:37.027338 32124 raft_consensus.cc:504] T d7c55e3c68744428a3bc7f2a1fe50c1f P 67a81d55b1bd4d2e9b587c6477d624a2 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e0a4e8376dae403e9b812a429b8960af" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 42105 } attrs { replace: true } } peers { permanent_uuid: "67a81d55b1bd4d2e9b587c6477d624a2" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40197 } } peers { permanent_uuid: "c38cec78fdf74d05b3c75f21eeb8612c" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 45205 } attrs { promote: false } }
I20240502 21:28:37.027964 32124 leader_election.cc:288] T d7c55e3c68744428a3bc7f2a1fe50c1f P 67a81d55b1bd4d2e9b587c6477d624a2 [CANDIDATE]: Term 2 election: Requested vote from peers e0a4e8376dae403e9b812a429b8960af (127.15.255.132:42105), c38cec78fdf74d05b3c75f21eeb8612c (127.15.255.131:45205)
I20240502 21:28:37.028332 32523 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7c55e3c68744428a3bc7f2a1fe50c1f" candidate_uuid: "67a81d55b1bd4d2e9b587c6477d624a2" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "e0a4e8376dae403e9b812a429b8960af"
I20240502 21:28:37.028564 32523 raft_consensus.cc:3032] T d7c55e3c68744428a3bc7f2a1fe50c1f P e0a4e8376dae403e9b812a429b8960af [term 1 LEADER]: Stepping down as leader of term 1
I20240502 21:28:37.028671 32523 raft_consensus.cc:729] T d7c55e3c68744428a3bc7f2a1fe50c1f P e0a4e8376dae403e9b812a429b8960af [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e0a4e8376dae403e9b812a429b8960af, State: Running, Role: LEADER
I20240502 21:28:37.028640 525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7c55e3c68744428a3bc7f2a1fe50c1f" candidate_uuid: "67a81d55b1bd4d2e9b587c6477d624a2" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "c38cec78fdf74d05b3c75f21eeb8612c"
I20240502 21:28:37.028951 525 raft_consensus.cc:3037] T d7c55e3c68744428a3bc7f2a1fe50c1f P c38cec78fdf74d05b3c75f21eeb8612c [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:28:37.028961 32523 consensus_queue.cc:260] T d7c55e3c68744428a3bc7f2a1fe50c1f P e0a4e8376dae403e9b812a429b8960af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e0a4e8376dae403e9b812a429b8960af" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 42105 } attrs { replace: true } } peers { permanent_uuid: "67a81d55b1bd4d2e9b587c6477d624a2" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40197 } } peers { permanent_uuid: "c38cec78fdf74d05b3c75f21eeb8612c" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 45205 } attrs { promote: false } }
I20240502 21:28:37.029374 32523 raft_consensus.cc:3037] T d7c55e3c68744428a3bc7f2a1fe50c1f P e0a4e8376dae403e9b812a429b8960af [term 1 FOLLOWER]: Advancing to term 2
I20240502 21:28:37.030967 525 raft_consensus.cc:2454] T d7c55e3c68744428a3bc7f2a1fe50c1f P c38cec78fdf74d05b3c75f21eeb8612c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67a81d55b1bd4d2e9b587c6477d624a2 in term 2.
I20240502 21:28:37.031133 32523 raft_consensus.cc:2454] T d7c55e3c68744428a3bc7f2a1fe50c1f P e0a4e8376dae403e9b812a429b8960af [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67a81d55b1bd4d2e9b587c6477d624a2 in term 2.
I20240502 21:28:37.031361 32058 leader_election.cc:302] T d7c55e3c68744428a3bc7f2a1fe50c1f P 67a81d55b1bd4d2e9b587c6477d624a2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67a81d55b1bd4d2e9b587c6477d624a2, c38cec78fdf74d05b3c75f21eeb8612c; no voters:
I20240502 21:28:37.031636 920 raft_consensus.cc:2781] T d7c55e3c68744428a3bc7f2a1fe50c1f P 67a81d55b1bd4d2e9b587c6477d624a2 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 21:28:37.031810 920 raft_consensus.cc:686] T d7c55e3c68744428a3bc7f2a1fe50c1f P 67a81d55b1bd4d2e9b587c6477d624a2 [term 2 LEADER]: Becoming Leader. State: Replica: 67a81d55b1bd4d2e9b587c6477d624a2, State: Running, Role: LEADER
I20240502 21:28:37.032058 920 consensus_queue.cc:237] T d7c55e3c68744428a3bc7f2a1fe50c1f P 67a81d55b1bd4d2e9b587c6477d624a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e0a4e8376dae403e9b812a429b8960af" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 42105 } attrs { replace: true } } peers { permanent_uuid: "67a81d55b1bd4d2e9b587c6477d624a2" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40197 } } peers { permanent_uuid: "c38cec78fdf74d05b3c75f21eeb8612c" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 45205 } attrs { promote: false } }
I20240502 21:28:37.034230 31981 catalog_manager.cc:5484] T d7c55e3c68744428a3bc7f2a1fe50c1f P 67a81d55b1bd4d2e9b587c6477d624a2 reported cstate change: term changed from 1 to 2, leader changed from e0a4e8376dae403e9b812a429b8960af (127.15.255.132) to 67a81d55b1bd4d2e9b587c6477d624a2 (127.15.255.129). New cstate: current_term: 2 leader_uuid: "67a81d55b1bd4d2e9b587c6477d624a2" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e0a4e8376dae403e9b812a429b8960af" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 42105 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "67a81d55b1bd4d2e9b587c6477d624a2" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40197 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c38cec78fdf74d05b3c75f21eeb8612c" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 45205 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 21:28:37.484963 32523 raft_consensus.cc:1261] T d7c55e3c68744428a3bc7f2a1fe50c1f P e0a4e8376dae403e9b812a429b8960af [term 2 FOLLOWER]: Refusing update from remote peer 67a81d55b1bd4d2e9b587c6477d624a2: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 21:28:37.485606 920 consensus_queue.cc:1035] T d7c55e3c68744428a3bc7f2a1fe50c1f P 67a81d55b1bd4d2e9b587c6477d624a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0a4e8376dae403e9b812a429b8960af" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 42105 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:28:37.497836 525 raft_consensus.cc:1261] T d7c55e3c68744428a3bc7f2a1fe50c1f P c38cec78fdf74d05b3c75f21eeb8612c [term 2 FOLLOWER]: Refusing update from remote peer 67a81d55b1bd4d2e9b587c6477d624a2: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 21:28:37.498875 921 consensus_queue.cc:1035] T d7c55e3c68744428a3bc7f2a1fe50c1f P 67a81d55b1bd4d2e9b587c6477d624a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c38cec78fdf74d05b3c75f21eeb8612c" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 45205 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:28:38.008312 32124 consensus_queue.cc:237] T d7c55e3c68744428a3bc7f2a1fe50c1f P 67a81d55b1bd4d2e9b587c6477d624a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "67a81d55b1bd4d2e9b587c6477d624a2" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40197 } } peers { permanent_uuid: "c38cec78fdf74d05b3c75f21eeb8612c" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 45205 } attrs { promote: false } }
I20240502 21:28:38.009945 525 raft_consensus.cc:1261] T d7c55e3c68744428a3bc7f2a1fe50c1f P c38cec78fdf74d05b3c75f21eeb8612c [term 2 FOLLOWER]: Refusing update from remote peer 67a81d55b1bd4d2e9b587c6477d624a2: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 21:28:38.010444 1107 consensus_queue.cc:1035] T d7c55e3c68744428a3bc7f2a1fe50c1f P 67a81d55b1bd4d2e9b587c6477d624a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c38cec78fdf74d05b3c75f21eeb8612c" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 45205 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:28:38.012683 1106 raft_consensus.cc:2932] T d7c55e3c68744428a3bc7f2a1fe50c1f P 67a81d55b1bd4d2e9b587c6477d624a2 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER e0a4e8376dae403e9b812a429b8960af (127.15.255.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "67a81d55b1bd4d2e9b587c6477d624a2" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40197 } } peers { permanent_uuid: "c38cec78fdf74d05b3c75f21eeb8612c" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 45205 } attrs { promote: false } } }
I20240502 21:28:38.013267 525 raft_consensus.cc:2932] T d7c55e3c68744428a3bc7f2a1fe50c1f P c38cec78fdf74d05b3c75f21eeb8612c [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER e0a4e8376dae403e9b812a429b8960af (127.15.255.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "67a81d55b1bd4d2e9b587c6477d624a2" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40197 } } peers { permanent_uuid: "c38cec78fdf74d05b3c75f21eeb8612c" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 45205 } attrs { promote: false } } }
I20240502 21:28:38.015477 31965 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet d7c55e3c68744428a3bc7f2a1fe50c1f with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:28:38.016775 31980 catalog_manager.cc:5484] T d7c55e3c68744428a3bc7f2a1fe50c1f P 67a81d55b1bd4d2e9b587c6477d624a2 reported cstate change: config changed from index 5 to 7, VOTER e0a4e8376dae403e9b812a429b8960af (127.15.255.132) evicted. New cstate: current_term: 2 leader_uuid: "67a81d55b1bd4d2e9b587c6477d624a2" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "67a81d55b1bd4d2e9b587c6477d624a2" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 40197 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c38cec78fdf74d05b3c75f21eeb8612c" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 45205 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:28:38.020953 32503 tablet_service.cc:1514] Processing DeleteTablet for tablet d7c55e3c68744428a3bc7f2a1fe50c1f with delete_type TABLET_DATA_TOMBSTONED (TS e0a4e8376dae403e9b812a429b8960af not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:56374
I20240502 21:28:38.022596 1112 tablet_replica.cc:330] T d7c55e3c68744428a3bc7f2a1fe50c1f P e0a4e8376dae403e9b812a429b8960af: stopping tablet replica
I20240502 21:28:38.022816 1112 raft_consensus.cc:2229] T d7c55e3c68744428a3bc7f2a1fe50c1f P e0a4e8376dae403e9b812a429b8960af [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 21:28:38.023041 1112 raft_consensus.cc:2258] T d7c55e3c68744428a3bc7f2a1fe50c1f P e0a4e8376dae403e9b812a429b8960af [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 21:28:38.023929 1112 ts_tablet_manager.cc:1875] T d7c55e3c68744428a3bc7f2a1fe50c1f P e0a4e8376dae403e9b812a429b8960af: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:28:38.028467 1112 ts_tablet_manager.cc:1888] T d7c55e3c68744428a3bc7f2a1fe50c1f P e0a4e8376dae403e9b812a429b8960af: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240502 21:28:38.028573 1112 log.cc:1196] T d7c55e3c68744428a3bc7f2a1fe50c1f P e0a4e8376dae403e9b812a429b8960af: Deleting WAL directory at /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.Rebalance_idx7_rf2_t3_l_1_2_1.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-3/wal/wals/d7c55e3c68744428a3bc7f2a1fe50c1f
I20240502 21:28:38.029232 31967 catalog_manager.cc:4830] TS e0a4e8376dae403e9b812a429b8960af (127.15.255.132:42105): tablet d7c55e3c68744428a3bc7f2a1fe50c1f (table rebalance_test_table_0 [id=7b825a608b1348f09ea4d42250b107e2]) successfully deleted
I20240502 21:28:38.255214 32104 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:28:38.285676 32503 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:28:38.286453 506 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:28:38.287663 374 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:28:38.444844 32104 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:28:38.449988 506 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:28:38.454324 374 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:28:38.464115 32503 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:28:38.589527 32104 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:28:38.599267 506 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:28:38.604995 32503 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 21:28:38.607241 374 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:28:38.786070 16382 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvNciQZ/build/asan/bin/kudu pid 32042
I20240502 21:28:38.807803 32165 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 21:28:38.932121 16382 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvNciQZ/build/asan/bin/kudu with pid 32042
W20240502 21:28:38.952584 32455 connection.cc:535] server connection from 127.15.255.129:53837 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 21:28:38.952909 16382 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvNciQZ/build/asan/bin/kudu pid 311
I20240502 21:28:38.958536 435 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 21:28:39.060649 16382 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvNciQZ/build/asan/bin/kudu with pid 311
I20240502 21:28:39.077921 16382 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvNciQZ/build/asan/bin/kudu pid 444
I20240502 21:28:39.083348 566 generic_service.cc:196] Checking for leaks (request via RPC)
W20240502 21:28:39.140998 32455 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.15.255.129:40197: connect: Connection refused (error 111)
W20240502 21:28:39.142416 32455 consensus_peers.cc:485] T 1707bbe6fcd44b09ab452215e3997199 P e0a4e8376dae403e9b812a429b8960af -> Peer 67a81d55b1bd4d2e9b587c6477d624a2 (127.15.255.129:40197): Couldn't send request to peer 67a81d55b1bd4d2e9b587c6477d624a2. Status: Network error: Client connection negotiation failed: client connection to 127.15.255.129:40197: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 21:28:39.187112 16382 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvNciQZ/build/asan/bin/kudu with pid 444
I20240502 21:28:39.204358 16382 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvNciQZ/build/asan/bin/kudu pid 32440
I20240502 21:28:39.210328 32563 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 21:28:39.337553 16382 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvNciQZ/build/asan/bin/kudu with pid 32440
I20240502 21:28:39.357929 16382 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvNciQZ/build/asan/bin/kudu pid 31950
I20240502 21:28:39.358990 32013 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 21:28:39.488914 16382 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvNciQZ/build/asan/bin/kudu with pid 31950
2024-05-02T21:28:39Z chronyd exiting
[ OK ] LocationAwareRebalancingParamTest.Rebalance/idx7_rf2_t3_l_1_2_1 (15531 ms)
[ RUN ] LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer/idx6_rf7_t7_l_3_4_3
I20240502 21:28:39.536576 16382 test_util.cc:268] Using random seed: 1230987156
I20240502 21:28:39.538365 16382 ts_itest-base.cc:112] Starting cluster with:
I20240502 21:28:39.538437 16382 ts_itest-base.cc:113] --------------
I20240502 21:28:39.538476 16382 ts_itest-base.cc:114] 10 tablet servers
I20240502 21:28:39.538519 16382 ts_itest-base.cc:115] 7 replicas per TS
I20240502 21:28:39.538555 16382 ts_itest-base.cc:116] --------------
2024-05-02T21:28:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-02T21:28:39Z Disabled control of system clock
I20240502 21:28:39.590386 16382 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvNciQZ/build/asan/bin/kudu
/tmp/dist-test-taskvNciQZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.255.190:43173
--webserver_interface=127.15.255.190
--webserver_port=0
--builtin_ntp_servers=127.15.255.148:41767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.15.255.190:43173
--location_mapping_cmd=/tmp/dist-test-taskvNciQZ/build/asan/bin/testdata/assign-location.py --state_store=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/location-assignment.state --map /A:3 --map /B:4 --map /C:3
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=true
--enable_range_replica_placement=false
--master_client_location_assignment_enabled=false
--allow_unsafe_replication_factor with env {}
W20240502 21:28:39.904891 1237 flags.cc:425] Enabled unsafe flag: --allow_unsafe_replication_factor=true
W20240502 21:28:39.905181 1237 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:28:39.905254 1237 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:28:39.905328 1237 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:28:39.912381 1237 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 21:28:39.912515 1237 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240502 21:28:39.912590 1237 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:28:39.912637 1237 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240502 21:28:39.912693 1237 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240502 21:28:39.921341 1237 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.255.148:41767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/master-0/wal
--allow_unsafe_replication_factor=true
--enable_range_replica_placement=false
--location_mapping_cmd=/tmp/dist-test-taskvNciQZ/build/asan/bin/testdata/assign-location.py --state_store=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/location-assignment.state --map /A:3 --map /B:4 --map /C:3
--ipki_ca_key_size=768
--master_addresses=127.15.255.190:43173
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.15.255.190:43173
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.15.255.190
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:28:39.922528 1237 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:28:39.924085 1237 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:28:39.934363 1243 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:28:39.934938 1244 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:28:39.935369 1246 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:28:39.935928 1237 server_base.cc:1034] running on GCE node
I20240502 21:28:39.936671 1237 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:28:39.938016 1237 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:28:39.939208 1237 hybrid_clock.cc:648] HybridClock initialized: now 1714685319939143 us; error 45 us; skew 500 ppm
I20240502 21:28:39.939502 1237 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:28:39.941617 1237 webserver.cc:460] Webserver started at http://127.15.255.190:46247/ using document root <none> and password file <none>
I20240502 21:28:39.942292 1237 fs_manager.cc:359] Metadata directory not provided
I20240502 21:28:39.942416 1237 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:28:39.942695 1237 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:28:39.945166 1237 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "12dea922ad1d47bbbcef6c798f8f907d"
format_stamp: "Formatted at 2024-05-02 21:28:39 on dist-test-slave-jdfx"
I20240502 21:28:39.945880 1237 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "12dea922ad1d47bbbcef6c798f8f907d"
format_stamp: "Formatted at 2024-05-02 21:28:39 on dist-test-slave-jdfx"
I20240502 21:28:39.950510 1237 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20240502 21:28:39.953512 1253 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:28:39.954810 1237 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240502 21:28:39.954960 1237 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/master-0/wal
uuid: "12dea922ad1d47bbbcef6c798f8f907d"
format_stamp: "Formatted at 2024-05-02 21:28:39 on dist-test-slave-jdfx"
I20240502 21:28:39.955113 1237 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:28:39.975984 1237 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:28:39.976830 1237 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:28:39.977133 1237 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:28:39.993533 1237 rpc_server.cc:307] RPC server started. Bound to: 127.15.255.190:43173
I20240502 21:28:39.993568 1305 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.255.190:43173 every 8 connection(s)
I20240502 21:28:39.994736 1237 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240502 21:28:39.998018 1306 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:40.002909 16382 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvNciQZ/build/asan/bin/kudu as pid 1237
I20240502 21:28:40.003152 16382 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240502 21:28:40.007496 1306 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12dea922ad1d47bbbcef6c798f8f907d: Bootstrap starting.
I20240502 21:28:40.010603 1306 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 12dea922ad1d47bbbcef6c798f8f907d: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:40.011704 1306 log.cc:824] T 00000000000000000000000000000000 P 12dea922ad1d47bbbcef6c798f8f907d: Log is configured to *not* fsync() on all Append() calls
I20240502 21:28:40.014266 1306 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12dea922ad1d47bbbcef6c798f8f907d: No bootstrap required, opened a new log
I20240502 21:28:40.020287 1306 raft_consensus.cc:348] T 00000000000000000000000000000000 P 12dea922ad1d47bbbcef6c798f8f907d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12dea922ad1d47bbbcef6c798f8f907d" member_type: VOTER last_known_addr { host: "127.15.255.190" port: 43173 } }
I20240502 21:28:40.020618 1306 raft_consensus.cc:374] T 00000000000000000000000000000000 P 12dea922ad1d47bbbcef6c798f8f907d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:40.020713 1306 raft_consensus.cc:729] T 00000000000000000000000000000000 P 12dea922ad1d47bbbcef6c798f8f907d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12dea922ad1d47bbbcef6c798f8f907d, State: Initialized, Role: FOLLOWER
I20240502 21:28:40.021274 1306 consensus_queue.cc:260] T 00000000000000000000000000000000 P 12dea922ad1d47bbbcef6c798f8f907d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12dea922ad1d47bbbcef6c798f8f907d" member_type: VOTER last_known_addr { host: "127.15.255.190" port: 43173 } }
I20240502 21:28:40.021466 1306 raft_consensus.cc:388] T 00000000000000000000000000000000 P 12dea922ad1d47bbbcef6c798f8f907d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 21:28:40.021580 1306 raft_consensus.cc:482] T 00000000000000000000000000000000 P 12dea922ad1d47bbbcef6c798f8f907d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 21:28:40.021713 1306 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 12dea922ad1d47bbbcef6c798f8f907d [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:40.023768 1306 raft_consensus.cc:504] T 00000000000000000000000000000000 P 12dea922ad1d47bbbcef6c798f8f907d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12dea922ad1d47bbbcef6c798f8f907d" member_type: VOTER last_known_addr { host: "127.15.255.190" port: 43173 } }
I20240502 21:28:40.024199 1306 leader_election.cc:302] T 00000000000000000000000000000000 P 12dea922ad1d47bbbcef6c798f8f907d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 12dea922ad1d47bbbcef6c798f8f907d; no voters:
I20240502 21:28:40.024765 1306 leader_election.cc:288] T 00000000000000000000000000000000 P 12dea922ad1d47bbbcef6c798f8f907d [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 21:28:40.024893 1311 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 12dea922ad1d47bbbcef6c798f8f907d [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:28:40.025440 1311 raft_consensus.cc:686] T 00000000000000000000000000000000 P 12dea922ad1d47bbbcef6c798f8f907d [term 1 LEADER]: Becoming Leader. State: Replica: 12dea922ad1d47bbbcef6c798f8f907d, State: Running, Role: LEADER
I20240502 21:28:40.025872 1311 consensus_queue.cc:237] T 00000000000000000000000000000000 P 12dea922ad1d47bbbcef6c798f8f907d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12dea922ad1d47bbbcef6c798f8f907d" member_type: VOTER last_known_addr { host: "127.15.255.190" port: 43173 } }
I20240502 21:28:40.026402 1306 sys_catalog.cc:564] T 00000000000000000000000000000000 P 12dea922ad1d47bbbcef6c798f8f907d [sys.catalog]: configured and running, proceeding with master startup.
I20240502 21:28:40.028491 1312 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12dea922ad1d47bbbcef6c798f8f907d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "12dea922ad1d47bbbcef6c798f8f907d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12dea922ad1d47bbbcef6c798f8f907d" member_type: VOTER last_known_addr { host: "127.15.255.190" port: 43173 } } }
I20240502 21:28:40.028827 1312 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12dea922ad1d47bbbcef6c798f8f907d [sys.catalog]: This master's current role is: LEADER
I20240502 21:28:40.028460 1314 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12dea922ad1d47bbbcef6c798f8f907d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 12dea922ad1d47bbbcef6c798f8f907d. Latest consensus state: current_term: 1 leader_uuid: "12dea922ad1d47bbbcef6c798f8f907d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12dea922ad1d47bbbcef6c798f8f907d" member_type: VOTER last_known_addr { host: "127.15.255.190" port: 43173 } } }
I20240502 21:28:40.029678 1314 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12dea922ad1d47bbbcef6c798f8f907d [sys.catalog]: This master's current role is: LEADER
I20240502 21:28:40.033386 1317 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 21:28:40.037689 1317 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 21:28:40.044725 1317 catalog_manager.cc:1347] Generated new cluster ID: ce5a8991f18f42f48d23d4c677519f48
I20240502 21:28:40.044822 1317 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 21:28:40.075582 1317 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 21:28:40.077180 1317 catalog_manager.cc:1504] Loading token signing keys...
I20240502 21:28:40.098392 1317 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 12dea922ad1d47bbbcef6c798f8f907d: Generated new TSK 0
I20240502 21:28:40.099363 1317 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 21:28:40.114607 16382 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvNciQZ/build/asan/bin/kudu
/tmp/dist-test-taskvNciQZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.255.129:0
--local_ip_for_outbound_sockets=127.15.255.129
--webserver_interface=127.15.255.129
--webserver_port=0
--tserver_master_addrs=127.15.255.190:43173
--builtin_ntp_servers=127.15.255.148:41767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240502 21:28:40.461426 1331 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:28:40.461706 1331 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:28:40.461848 1331 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:28:40.469798 1331 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 21:28:40.469918 1331 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:28:40.470069 1331 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.255.129
I20240502 21:28:40.479256 1331 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.255.148:41767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.255.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.15.255.129
--webserver_port=0
--tserver_master_addrs=127.15.255.190:43173
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.255.129
--log_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:28:40.480682 1331 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:28:40.482407 1331 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:28:40.493896 1336 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:28:40.494534 1337 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:28:40.494942 1339 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:28:40.495592 1331 server_base.cc:1034] running on GCE node
I20240502 21:28:40.496122 1331 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:28:40.496994 1331 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:28:40.498193 1331 hybrid_clock.cc:648] HybridClock initialized: now 1714685320498151 us; error 37 us; skew 500 ppm
I20240502 21:28:40.498473 1331 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:28:40.500481 1331 webserver.cc:460] Webserver started at http://127.15.255.129:42505/ using document root <none> and password file <none>
I20240502 21:28:40.501142 1331 fs_manager.cc:359] Metadata directory not provided
I20240502 21:28:40.501248 1331 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:28:40.501535 1331 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:28:40.504022 1331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "7d1e5c1b047443cbb28d7b0a9744babf"
format_stamp: "Formatted at 2024-05-02 21:28:40 on dist-test-slave-jdfx"
I20240502 21:28:40.504767 1331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "7d1e5c1b047443cbb28d7b0a9744babf"
format_stamp: "Formatted at 2024-05-02 21:28:40 on dist-test-slave-jdfx"
I20240502 21:28:40.509384 1331 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20240502 21:28:40.512436 1346 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:28:40.513669 1331 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240502 21:28:40.513804 1331 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "7d1e5c1b047443cbb28d7b0a9744babf"
format_stamp: "Formatted at 2024-05-02 21:28:40 on dist-test-slave-jdfx"
I20240502 21:28:40.513979 1331 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:28:40.538656 1331 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:28:40.539438 1331 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:28:40.539680 1331 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:28:40.540606 1331 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:28:40.541993 1331 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:28:40.542084 1331 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:28:40.542188 1331 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:28:40.542270 1331 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 21:28:40.570410 1331 rpc_server.cc:307] RPC server started. Bound to: 127.15.255.129:37897
I20240502 21:28:40.570469 1459 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.255.129:37897 every 8 connection(s)
I20240502 21:28:40.571671 1331 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240502 21:28:40.575258 16382 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvNciQZ/build/asan/bin/kudu as pid 1331
I20240502 21:28:40.575471 16382 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240502 21:28:40.578874 16382 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvNciQZ/build/asan/bin/kudu
/tmp/dist-test-taskvNciQZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.255.130:0
--local_ip_for_outbound_sockets=127.15.255.130
--webserver_interface=127.15.255.130
--webserver_port=0
--tserver_master_addrs=127.15.255.190:43173
--builtin_ntp_servers=127.15.255.148:41767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 21:28:40.584007 1460 heartbeater.cc:347] Connected to a master server at 127.15.255.190:43173
I20240502 21:28:40.584271 1460 heartbeater.cc:464] Registering TS with master...
I20240502 21:28:40.585053 1460 heartbeater.cc:511] Master 127.15.255.190:43173 requested a full tablet report, sending...
I20240502 21:28:40.628137 1270 ts_manager.cc:194] Registered new tserver with Master: 7d1e5c1b047443cbb28d7b0a9744babf (127.15.255.129:37897)
I20240502 21:28:40.630517 1270 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.255.129:48851
W20240502 21:28:40.897789 1464 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:28:40.898028 1464 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:28:40.898149 1464 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:28:40.905236 1464 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 21:28:40.905344 1464 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:28:40.905454 1464 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.255.130
I20240502 21:28:40.913899 1464 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.255.148:41767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.255.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.15.255.130
--webserver_port=0
--tserver_master_addrs=127.15.255.190:43173
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.255.130
--log_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:28:40.915002 1464 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:28:40.916602 1464 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:28:40.927871 1470 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:28:40.928452 1471 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:28:40.928781 1473 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:28:40.929536 1464 server_base.cc:1034] running on GCE node
I20240502 21:28:40.930042 1464 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:28:40.930878 1464 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:28:40.932107 1464 hybrid_clock.cc:648] HybridClock initialized: now 1714685320932058 us; error 38 us; skew 500 ppm
I20240502 21:28:40.932371 1464 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:28:40.934317 1464 webserver.cc:460] Webserver started at http://127.15.255.130:36363/ using document root <none> and password file <none>
I20240502 21:28:40.934938 1464 fs_manager.cc:359] Metadata directory not provided
I20240502 21:28:40.935034 1464 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:28:40.935288 1464 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:28:40.937743 1464 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "a62a935ff43d494ca35d0c417943133d"
format_stamp: "Formatted at 2024-05-02 21:28:40 on dist-test-slave-jdfx"
I20240502 21:28:40.938442 1464 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "a62a935ff43d494ca35d0c417943133d"
format_stamp: "Formatted at 2024-05-02 21:28:40 on dist-test-slave-jdfx"
I20240502 21:28:40.942972 1464 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20240502 21:28:40.945974 1479 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:28:40.947119 1464 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240502 21:28:40.947254 1464 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "a62a935ff43d494ca35d0c417943133d"
format_stamp: "Formatted at 2024-05-02 21:28:40 on dist-test-slave-jdfx"
I20240502 21:28:40.947435 1464 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:28:40.965926 1464 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:28:40.966703 1464 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:28:40.966951 1464 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:28:40.967828 1464 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:28:40.969259 1464 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:28:40.969355 1464 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:28:40.969458 1464 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:28:40.969525 1464 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 21:28:40.997900 1464 rpc_server.cc:307] RPC server started. Bound to: 127.15.255.130:37991
I20240502 21:28:40.997946 1596 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.255.130:37991 every 8 connection(s)
I20240502 21:28:40.999118 1464 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240502 21:28:40.999667 16382 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvNciQZ/build/asan/bin/kudu as pid 1464
I20240502 21:28:40.999841 16382 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240502 21:28:41.003582 16382 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvNciQZ/build/asan/bin/kudu
/tmp/dist-test-taskvNciQZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.255.131:0
--local_ip_for_outbound_sockets=127.15.255.131
--webserver_interface=127.15.255.131
--webserver_port=0
--tserver_master_addrs=127.15.255.190:43173
--builtin_ntp_servers=127.15.255.148:41767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 21:28:41.011854 1597 heartbeater.cc:347] Connected to a master server at 127.15.255.190:43173
I20240502 21:28:41.012171 1597 heartbeater.cc:464] Registering TS with master...
I20240502 21:28:41.012972 1597 heartbeater.cc:511] Master 127.15.255.190:43173 requested a full tablet report, sending...
I20240502 21:28:41.058499 1270 ts_manager.cc:194] Registered new tserver with Master: a62a935ff43d494ca35d0c417943133d (127.15.255.130:37991)
I20240502 21:28:41.059478 1270 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.255.130:51247
W20240502 21:28:41.336282 1600 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:28:41.336582 1600 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:28:41.336719 1600 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:28:41.343957 1600 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 21:28:41.344067 1600 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:28:41.344223 1600 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.255.131
I20240502 21:28:41.352963 1600 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.255.148:41767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.255.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.15.255.131
--webserver_port=0
--tserver_master_addrs=127.15.255.190:43173
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.255.131
--log_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:28:41.354130 1600 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:28:41.355736 1600 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:28:41.366906 1606 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:28:41.367556 1607 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:28:41.367928 1609 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:28:41.368724 1600 server_base.cc:1034] running on GCE node
I20240502 21:28:41.369297 1600 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:28:41.370110 1600 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:28:41.371312 1600 hybrid_clock.cc:648] HybridClock initialized: now 1714685321371267 us; error 41 us; skew 500 ppm
I20240502 21:28:41.371603 1600 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:28:41.373646 1600 webserver.cc:460] Webserver started at http://127.15.255.131:45189/ using document root <none> and password file <none>
I20240502 21:28:41.374300 1600 fs_manager.cc:359] Metadata directory not provided
I20240502 21:28:41.374406 1600 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:28:41.374688 1600 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:28:41.377113 1600 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "16b3a337bc5e462f83ba895ad8660deb"
format_stamp: "Formatted at 2024-05-02 21:28:41 on dist-test-slave-jdfx"
I20240502 21:28:41.377791 1600 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "16b3a337bc5e462f83ba895ad8660deb"
format_stamp: "Formatted at 2024-05-02 21:28:41 on dist-test-slave-jdfx"
I20240502 21:28:41.382325 1600 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20240502 21:28:41.385356 1615 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:28:41.386463 1600 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240502 21:28:41.386600 1600 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "16b3a337bc5e462f83ba895ad8660deb"
format_stamp: "Formatted at 2024-05-02 21:28:41 on dist-test-slave-jdfx"
I20240502 21:28:41.386771 1600 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:28:41.417016 1600 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:28:41.417824 1600 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:28:41.418128 1600 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:28:41.419070 1600 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:28:41.420552 1600 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:28:41.420646 1600 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:28:41.420747 1600 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:28:41.420814 1600 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 21:28:41.448582 1600 rpc_server.cc:307] RPC server started. Bound to: 127.15.255.131:36033
I20240502 21:28:41.448642 1731 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.255.131:36033 every 8 connection(s)
I20240502 21:28:41.449759 1600 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240502 21:28:41.450498 16382 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvNciQZ/build/asan/bin/kudu as pid 1600
I20240502 21:28:41.450685 16382 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240502 21:28:41.454717 16382 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvNciQZ/build/asan/bin/kudu
/tmp/dist-test-taskvNciQZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.255.132:0
--local_ip_for_outbound_sockets=127.15.255.132
--webserver_interface=127.15.255.132
--webserver_port=0
--tserver_master_addrs=127.15.255.190:43173
--builtin_ntp_servers=127.15.255.148:41767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 21:28:41.463136 1732 heartbeater.cc:347] Connected to a master server at 127.15.255.190:43173
I20240502 21:28:41.463428 1732 heartbeater.cc:464] Registering TS with master...
I20240502 21:28:41.464321 1732 heartbeater.cc:511] Master 127.15.255.190:43173 requested a full tablet report, sending...
I20240502 21:28:41.520673 1270 ts_manager.cc:194] Registered new tserver with Master: 16b3a337bc5e462f83ba895ad8660deb (127.15.255.131:36033)
I20240502 21:28:41.521623 1270 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.255.131:44911
I20240502 21:28:41.634245 1460 heartbeater.cc:503] Master 127.15.255.190:43173 was elected leader, sending a full tablet report...
W20240502 21:28:41.787022 1735 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:28:41.787282 1735 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:28:41.787395 1735 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:28:41.794488 1735 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 21:28:41.794608 1735 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:28:41.794754 1735 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.255.132
I20240502 21:28:41.803076 1735 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.255.148:41767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.255.132:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.15.255.132
--webserver_port=0
--tserver_master_addrs=127.15.255.190:43173
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.255.132
--log_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:28:41.804208 1735 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:28:41.805783 1735 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:28:41.816843 1744 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:28:41.817260 1741 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:28:41.818003 1742 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:28:41.818702 1735 server_base.cc:1034] running on GCE node
I20240502 21:28:41.819218 1735 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:28:41.820035 1735 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:28:41.821218 1735 hybrid_clock.cc:648] HybridClock initialized: now 1714685321821180 us; error 31 us; skew 500 ppm
I20240502 21:28:41.821503 1735 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:28:41.823484 1735 webserver.cc:460] Webserver started at http://127.15.255.132:39997/ using document root <none> and password file <none>
I20240502 21:28:41.824182 1735 fs_manager.cc:359] Metadata directory not provided
I20240502 21:28:41.824283 1735 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:28:41.824605 1735 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:28:41.826992 1735 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "153026dd7f8943cc93fedf6f48450fb4"
format_stamp: "Formatted at 2024-05-02 21:28:41 on dist-test-slave-jdfx"
I20240502 21:28:41.827684 1735 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "153026dd7f8943cc93fedf6f48450fb4"
format_stamp: "Formatted at 2024-05-02 21:28:41 on dist-test-slave-jdfx"
I20240502 21:28:41.832332 1735 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20240502 21:28:41.835371 1750 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:28:41.836578 1735 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240502 21:28:41.836704 1735 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "153026dd7f8943cc93fedf6f48450fb4"
format_stamp: "Formatted at 2024-05-02 21:28:41 on dist-test-slave-jdfx"
I20240502 21:28:41.836861 1735 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:28:41.851097 1735 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:28:41.851828 1735 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:28:41.852044 1735 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:28:41.852928 1735 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:28:41.854239 1735 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:28:41.854327 1735 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:28:41.854403 1735 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:28:41.854455 1735 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 21:28:41.883898 1735 rpc_server.cc:307] RPC server started. Bound to: 127.15.255.132:44947
I20240502 21:28:41.883946 1864 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.255.132:44947 every 8 connection(s)
I20240502 21:28:41.885108 1735 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240502 21:28:41.886220 16382 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvNciQZ/build/asan/bin/kudu as pid 1735
I20240502 21:28:41.886408 16382 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240502 21:28:41.889561 16382 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvNciQZ/build/asan/bin/kudu
/tmp/dist-test-taskvNciQZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.255.133:0
--local_ip_for_outbound_sockets=127.15.255.133
--webserver_interface=127.15.255.133
--webserver_port=0
--tserver_master_addrs=127.15.255.190:43173
--builtin_ntp_servers=127.15.255.148:41767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 21:28:41.897456 1865 heartbeater.cc:347] Connected to a master server at 127.15.255.190:43173
I20240502 21:28:41.897720 1865 heartbeater.cc:464] Registering TS with master...
I20240502 21:28:41.898469 1865 heartbeater.cc:511] Master 127.15.255.190:43173 requested a full tablet report, sending...
I20240502 21:28:41.939654 1270 ts_manager.cc:194] Registered new tserver with Master: 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132:44947)
I20240502 21:28:41.940932 1270 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.255.132:47999
I20240502 21:28:42.062510 1597 heartbeater.cc:503] Master 127.15.255.190:43173 was elected leader, sending a full tablet report...
W20240502 21:28:42.220319 1868 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:28:42.220613 1868 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:28:42.220712 1868 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:28:42.228675 1868 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 21:28:42.228789 1868 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:28:42.228909 1868 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.255.133
I20240502 21:28:42.237604 1868 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.255.148:41767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.255.133:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.15.255.133
--webserver_port=0
--tserver_master_addrs=127.15.255.190:43173
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.255.133
--log_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:28:42.238786 1868 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:28:42.240410 1868 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:28:42.251523 1874 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:28:42.251899 1875 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:28:42.252504 1877 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:28:42.253086 1868 server_base.cc:1034] running on GCE node
I20240502 21:28:42.253588 1868 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:28:42.254385 1868 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:28:42.255584 1868 hybrid_clock.cc:648] HybridClock initialized: now 1714685322255538 us; error 40 us; skew 500 ppm
I20240502 21:28:42.255873 1868 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:28:42.257815 1868 webserver.cc:460] Webserver started at http://127.15.255.133:42981/ using document root <none> and password file <none>
I20240502 21:28:42.258486 1868 fs_manager.cc:359] Metadata directory not provided
I20240502 21:28:42.258595 1868 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:28:42.258877 1868 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:28:42.261341 1868 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "e632c553ff4341328acf1cbaca678392"
format_stamp: "Formatted at 2024-05-02 21:28:42 on dist-test-slave-jdfx"
I20240502 21:28:42.262004 1868 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "e632c553ff4341328acf1cbaca678392"
format_stamp: "Formatted at 2024-05-02 21:28:42 on dist-test-slave-jdfx"
I20240502 21:28:42.266466 1868 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20240502 21:28:42.269374 1883 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:28:42.270491 1868 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240502 21:28:42.270643 1868 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "e632c553ff4341328acf1cbaca678392"
format_stamp: "Formatted at 2024-05-02 21:28:42 on dist-test-slave-jdfx"
I20240502 21:28:42.270814 1868 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:28:42.291832 1868 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:28:42.292570 1868 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:28:42.292802 1868 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:28:42.293607 1868 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:28:42.294911 1868 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:28:42.295007 1868 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:28:42.295109 1868 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:28:42.295176 1868 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 21:28:42.322563 1868 rpc_server.cc:307] RPC server started. Bound to: 127.15.255.133:37439
I20240502 21:28:42.322621 2000 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.255.133:37439 every 8 connection(s)
I20240502 21:28:42.323701 1868 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240502 21:28:42.330554 16382 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvNciQZ/build/asan/bin/kudu as pid 1868
I20240502 21:28:42.330720 16382 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240502 21:28:42.333992 16382 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvNciQZ/build/asan/bin/kudu
/tmp/dist-test-taskvNciQZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.255.134:0
--local_ip_for_outbound_sockets=127.15.255.134
--webserver_interface=127.15.255.134
--webserver_port=0
--tserver_master_addrs=127.15.255.190:43173
--builtin_ntp_servers=127.15.255.148:41767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 21:28:42.335947 2001 heartbeater.cc:347] Connected to a master server at 127.15.255.190:43173
I20240502 21:28:42.336256 2001 heartbeater.cc:464] Registering TS with master...
I20240502 21:28:42.337173 2001 heartbeater.cc:511] Master 127.15.255.190:43173 requested a full tablet report, sending...
I20240502 21:28:42.381878 1270 ts_manager.cc:194] Registered new tserver with Master: e632c553ff4341328acf1cbaca678392 (127.15.255.133:37439)
I20240502 21:28:42.383189 1270 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.255.133:59187
I20240502 21:28:42.524262 1732 heartbeater.cc:503] Master 127.15.255.190:43173 was elected leader, sending a full tablet report...
W20240502 21:28:42.654040 2004 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:28:42.654295 2004 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:28:42.654428 2004 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:28:42.661548 2004 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 21:28:42.661657 2004 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:28:42.661808 2004 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.255.134
I20240502 21:28:42.670217 2004 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.255.148:41767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.255.134:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.15.255.134
--webserver_port=0
--tserver_master_addrs=127.15.255.190:43173
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.255.134
--log_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:28:42.671418 2004 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:28:42.673039 2004 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:28:42.684594 2011 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:28:42.684867 2013 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:28:42.685433 2010 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:28:42.685972 2004 server_base.cc:1034] running on GCE node
I20240502 21:28:42.686472 2004 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:28:42.687268 2004 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:28:42.688467 2004 hybrid_clock.cc:648] HybridClock initialized: now 1714685322688413 us; error 41 us; skew 500 ppm
I20240502 21:28:42.688746 2004 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:28:42.690759 2004 webserver.cc:460] Webserver started at http://127.15.255.134:42355/ using document root <none> and password file <none>
I20240502 21:28:42.691466 2004 fs_manager.cc:359] Metadata directory not provided
I20240502 21:28:42.691569 2004 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:28:42.691891 2004 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:28:42.694423 2004 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "521bb93035324cecbebed2f6f2a630f8"
format_stamp: "Formatted at 2024-05-02 21:28:42 on dist-test-slave-jdfx"
I20240502 21:28:42.695107 2004 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "521bb93035324cecbebed2f6f2a630f8"
format_stamp: "Formatted at 2024-05-02 21:28:42 on dist-test-slave-jdfx"
I20240502 21:28:42.699680 2004 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20240502 21:28:42.702734 2021 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:28:42.703884 2004 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 21:28:42.704025 2004 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "521bb93035324cecbebed2f6f2a630f8"
format_stamp: "Formatted at 2024-05-02 21:28:42 on dist-test-slave-jdfx"
I20240502 21:28:42.704193 2004 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:28:42.731029 2004 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:28:42.731797 2004 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:28:42.732043 2004 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:28:42.732945 2004 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:28:42.734378 2004 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:28:42.734472 2004 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:28:42.734571 2004 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:28:42.734640 2004 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 21:28:42.763605 2004 rpc_server.cc:307] RPC server started. Bound to: 127.15.255.134:40821
I20240502 21:28:42.763662 2137 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.255.134:40821 every 8 connection(s)
I20240502 21:28:42.764832 2004 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240502 21:28:42.774559 16382 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvNciQZ/build/asan/bin/kudu as pid 2004
I20240502 21:28:42.774785 16382 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240502 21:28:42.776342 2138 heartbeater.cc:347] Connected to a master server at 127.15.255.190:43173
I20240502 21:28:42.776688 2138 heartbeater.cc:464] Registering TS with master...
I20240502 21:28:42.777561 2138 heartbeater.cc:511] Master 127.15.255.190:43173 requested a full tablet report, sending...
I20240502 21:28:42.778194 16382 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvNciQZ/build/asan/bin/kudu
/tmp/dist-test-taskvNciQZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.255.135:0
--local_ip_for_outbound_sockets=127.15.255.135
--webserver_interface=127.15.255.135
--webserver_port=0
--tserver_master_addrs=127.15.255.190:43173
--builtin_ntp_servers=127.15.255.148:41767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 21:28:42.830152 1270 ts_manager.cc:194] Registered new tserver with Master: 521bb93035324cecbebed2f6f2a630f8 (127.15.255.134:40821)
I20240502 21:28:42.831174 1270 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.255.134:57391
I20240502 21:28:42.943506 1865 heartbeater.cc:503] Master 127.15.255.190:43173 was elected leader, sending a full tablet report...
W20240502 21:28:43.108173 2143 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:28:43.108525 2143 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:28:43.108649 2143 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:28:43.115705 2143 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 21:28:43.115821 2143 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:28:43.115959 2143 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.255.135
I20240502 21:28:43.124490 2143 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.255.148:41767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.255.135:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.15.255.135
--webserver_port=0
--tserver_master_addrs=127.15.255.190:43173
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.255.135
--log_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:28:43.125633 2143 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:28:43.127323 2143 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:28:43.138378 2148 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:28:43.138932 2149 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:28:43.139369 2151 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:28:43.140013 2143 server_base.cc:1034] running on GCE node
I20240502 21:28:43.140566 2143 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:28:43.141415 2143 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:28:43.142606 2143 hybrid_clock.cc:648] HybridClock initialized: now 1714685323142571 us; error 46 us; skew 500 ppm
I20240502 21:28:43.142894 2143 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:28:43.144876 2143 webserver.cc:460] Webserver started at http://127.15.255.135:38145/ using document root <none> and password file <none>
I20240502 21:28:43.145515 2143 fs_manager.cc:359] Metadata directory not provided
I20240502 21:28:43.145620 2143 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:28:43.145895 2143 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:28:43.148223 2143 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-6/data/instance:
uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0"
format_stamp: "Formatted at 2024-05-02 21:28:43 on dist-test-slave-jdfx"
I20240502 21:28:43.148939 2143 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-6/wal/instance:
uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0"
format_stamp: "Formatted at 2024-05-02 21:28:43 on dist-test-slave-jdfx"
I20240502 21:28:43.153605 2143 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20240502 21:28:43.156616 2157 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:28:43.157778 2143 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240502 21:28:43.157920 2143 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0"
format_stamp: "Formatted at 2024-05-02 21:28:43 on dist-test-slave-jdfx"
I20240502 21:28:43.158102 2143 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:28:43.180744 2143 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:28:43.181468 2143 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:28:43.181711 2143 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:28:43.182567 2143 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:28:43.183916 2143 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:28:43.184007 2143 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:28:43.184119 2143 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:28:43.184186 2143 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 21:28:43.212241 2143 rpc_server.cc:307] RPC server started. Bound to: 127.15.255.135:34701
I20240502 21:28:43.212297 2273 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.255.135:34701 every 8 connection(s)
I20240502 21:28:43.213419 2143 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240502 21:28:43.221629 16382 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvNciQZ/build/asan/bin/kudu as pid 2143
I20240502 21:28:43.221798 16382 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-6/wal/instance
I20240502 21:28:43.225133 16382 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvNciQZ/build/asan/bin/kudu
/tmp/dist-test-taskvNciQZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.255.136:0
--local_ip_for_outbound_sockets=127.15.255.136
--webserver_interface=127.15.255.136
--webserver_port=0
--tserver_master_addrs=127.15.255.190:43173
--builtin_ntp_servers=127.15.255.148:41767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 21:28:43.225422 2274 heartbeater.cc:347] Connected to a master server at 127.15.255.190:43173
I20240502 21:28:43.225706 2274 heartbeater.cc:464] Registering TS with master...
I20240502 21:28:43.226627 2274 heartbeater.cc:511] Master 127.15.255.190:43173 requested a full tablet report, sending...
I20240502 21:28:43.299870 1270 ts_manager.cc:194] Registered new tserver with Master: 6ab8df3c796f4b6cb3e29d5045ad3ea0 (127.15.255.135:34701)
I20240502 21:28:43.300877 1270 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.255.135:45901
I20240502 21:28:43.385836 2001 heartbeater.cc:503] Master 127.15.255.190:43173 was elected leader, sending a full tablet report...
W20240502 21:28:43.545231 2277 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:28:43.545467 2277 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:28:43.545559 2277 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:28:43.552675 2277 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 21:28:43.552843 2277 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:28:43.552996 2277 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.255.136
I20240502 21:28:43.563320 2277 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.255.148:41767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.255.136:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.15.255.136
--webserver_port=0
--tserver_master_addrs=127.15.255.190:43173
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.255.136
--log_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:28:43.564499 2277 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:28:43.566067 2277 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:28:43.577801 2283 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:28:43.578173 2286 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:28:43.578867 2284 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:28:43.579561 2277 server_base.cc:1034] running on GCE node
I20240502 21:28:43.580056 2277 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:28:43.580899 2277 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:28:43.582100 2277 hybrid_clock.cc:648] HybridClock initialized: now 1714685323582056 us; error 33 us; skew 500 ppm
I20240502 21:28:43.582396 2277 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:28:43.584509 2277 webserver.cc:460] Webserver started at http://127.15.255.136:38591/ using document root <none> and password file <none>
I20240502 21:28:43.585171 2277 fs_manager.cc:359] Metadata directory not provided
I20240502 21:28:43.585278 2277 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:28:43.585559 2277 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:28:43.587953 2277 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-7/data/instance:
uuid: "d8d7297186694f77bb98de9174e4279c"
format_stamp: "Formatted at 2024-05-02 21:28:43 on dist-test-slave-jdfx"
I20240502 21:28:43.588699 2277 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-7/wal/instance:
uuid: "d8d7297186694f77bb98de9174e4279c"
format_stamp: "Formatted at 2024-05-02 21:28:43 on dist-test-slave-jdfx"
I20240502 21:28:43.593437 2277 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20240502 21:28:43.596386 2292 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:28:43.597538 2277 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240502 21:28:43.597677 2277 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "d8d7297186694f77bb98de9174e4279c"
format_stamp: "Formatted at 2024-05-02 21:28:43 on dist-test-slave-jdfx"
I20240502 21:28:43.597853 2277 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:28:43.617341 2277 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:28:43.618018 2277 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:28:43.618281 2277 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:28:43.619117 2277 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:28:43.620473 2277 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:28:43.620568 2277 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:28:43.620667 2277 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:28:43.620731 2277 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 21:28:43.648689 2277 rpc_server.cc:307] RPC server started. Bound to: 127.15.255.136:41459
I20240502 21:28:43.648746 2407 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.255.136:41459 every 8 connection(s)
I20240502 21:28:43.649870 2277 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240502 21:28:43.656574 16382 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvNciQZ/build/asan/bin/kudu as pid 2277
I20240502 21:28:43.656772 16382 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-7/wal/instance
I20240502 21:28:43.661217 16382 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvNciQZ/build/asan/bin/kudu
/tmp/dist-test-taskvNciQZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-8/wal
--fs_data_dirs=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-8/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-8/logs
--server_dump_info_path=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-8/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.255.137:0
--local_ip_for_outbound_sockets=127.15.255.137
--webserver_interface=127.15.255.137
--webserver_port=0
--tserver_master_addrs=127.15.255.190:43173
--builtin_ntp_servers=127.15.255.148:41767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 21:28:43.679781 2408 heartbeater.cc:347] Connected to a master server at 127.15.255.190:43173
I20240502 21:28:43.680124 2408 heartbeater.cc:464] Registering TS with master...
I20240502 21:28:43.681178 2408 heartbeater.cc:511] Master 127.15.255.190:43173 requested a full tablet report, sending...
I20240502 21:28:43.751231 1270 ts_manager.cc:194] Registered new tserver with Master: d8d7297186694f77bb98de9174e4279c (127.15.255.136:41459)
I20240502 21:28:43.752632 1270 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.255.136:37617
I20240502 21:28:43.834419 2138 heartbeater.cc:503] Master 127.15.255.190:43173 was elected leader, sending a full tablet report...
W20240502 21:28:44.008513 2411 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:28:44.008771 2411 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:28:44.008898 2411 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:28:44.016124 2411 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 21:28:44.016233 2411 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:28:44.016367 2411 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.255.137
I20240502 21:28:44.024946 2411 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.255.148:41767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-8/data
--fs_wal_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-8/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.255.137:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-8/data/info.pb
--webserver_interface=127.15.255.137
--webserver_port=0
--tserver_master_addrs=127.15.255.190:43173
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.255.137
--log_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-8/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:28:44.026064 2411 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:28:44.027696 2411 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:28:44.038834 2417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:28:44.039286 2418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:28:44.039705 2420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:28:44.040309 2411 server_base.cc:1034] running on GCE node
I20240502 21:28:44.040874 2411 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:28:44.041673 2411 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:28:44.042858 2411 hybrid_clock.cc:648] HybridClock initialized: now 1714685324042812 us; error 40 us; skew 500 ppm
I20240502 21:28:44.043134 2411 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:28:44.049449 2411 webserver.cc:460] Webserver started at http://127.15.255.137:46781/ using document root <none> and password file <none>
I20240502 21:28:44.050087 2411 fs_manager.cc:359] Metadata directory not provided
I20240502 21:28:44.050194 2411 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:28:44.050480 2411 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:28:44.052865 2411 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-8/data/instance:
uuid: "16a97c1394f346f0b987f97cce7371b6"
format_stamp: "Formatted at 2024-05-02 21:28:44 on dist-test-slave-jdfx"
I20240502 21:28:44.053627 2411 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-8/wal/instance:
uuid: "16a97c1394f346f0b987f97cce7371b6"
format_stamp: "Formatted at 2024-05-02 21:28:44 on dist-test-slave-jdfx"
I20240502 21:28:44.058193 2411 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20240502 21:28:44.061194 2426 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:28:44.062343 2411 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240502 21:28:44.062490 2411 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-8/data,/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-8/wal
uuid: "16a97c1394f346f0b987f97cce7371b6"
format_stamp: "Formatted at 2024-05-02 21:28:44 on dist-test-slave-jdfx"
I20240502 21:28:44.062664 2411 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-8/wal
metadata directory: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-8/wal
1 data directories: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-8/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:28:44.078191 2411 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:28:44.078855 2411 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:28:44.079087 2411 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:28:44.079887 2411 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:28:44.081219 2411 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:28:44.081319 2411 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:28:44.081419 2411 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:28:44.081485 2411 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 21:28:44.110122 2411 rpc_server.cc:307] RPC server started. Bound to: 127.15.255.137:43717
I20240502 21:28:44.110178 2543 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.255.137:43717 every 8 connection(s)
I20240502 21:28:44.111271 2411 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-8/data/info.pb
I20240502 21:28:44.115128 16382 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvNciQZ/build/asan/bin/kudu as pid 2411
I20240502 21:28:44.115307 16382 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-8/wal/instance
I20240502 21:28:44.118816 16382 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvNciQZ/build/asan/bin/kudu
/tmp/dist-test-taskvNciQZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-9/wal
--fs_data_dirs=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-9/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-9/logs
--server_dump_info_path=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-9/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.255.138:0
--local_ip_for_outbound_sockets=127.15.255.138
--webserver_interface=127.15.255.138
--webserver_port=0
--tserver_master_addrs=127.15.255.190:43173
--builtin_ntp_servers=127.15.255.148:41767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 21:28:44.124109 2544 heartbeater.cc:347] Connected to a master server at 127.15.255.190:43173
I20240502 21:28:44.124358 2544 heartbeater.cc:464] Registering TS with master...
I20240502 21:28:44.125090 2544 heartbeater.cc:511] Master 127.15.255.190:43173 requested a full tablet report, sending...
I20240502 21:28:44.182868 1270 ts_manager.cc:194] Registered new tserver with Master: 16a97c1394f346f0b987f97cce7371b6 (127.15.255.137:43717)
I20240502 21:28:44.183861 1270 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.255.137:36425
I20240502 21:28:44.303889 2274 heartbeater.cc:503] Master 127.15.255.190:43173 was elected leader, sending a full tablet report...
W20240502 21:28:44.456933 2547 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:28:44.457170 2547 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:28:44.457290 2547 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:28:44.464830 2547 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 21:28:44.464933 2547 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:28:44.465078 2547 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.255.138
I20240502 21:28:44.473526 2547 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.255.148:41767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-9/data
--fs_wal_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-9/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.255.138:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-9/data/info.pb
--webserver_interface=127.15.255.138
--webserver_port=0
--tserver_master_addrs=127.15.255.190:43173
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.255.138
--log_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-9/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:28:44.474617 2547 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:28:44.476204 2547 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:28:44.487182 2553 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:28:44.487658 2554 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:28:44.488260 2556 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:28:44.488843 2547 server_base.cc:1034] running on GCE node
I20240502 21:28:44.489346 2547 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:28:44.490150 2547 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:28:44.491355 2547 hybrid_clock.cc:648] HybridClock initialized: now 1714685324491302 us; error 49 us; skew 500 ppm
I20240502 21:28:44.491652 2547 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:28:44.493594 2547 webserver.cc:460] Webserver started at http://127.15.255.138:40023/ using document root <none> and password file <none>
I20240502 21:28:44.494203 2547 fs_manager.cc:359] Metadata directory not provided
I20240502 21:28:44.494290 2547 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:28:44.494544 2547 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 21:28:44.496958 2547 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-9/data/instance:
uuid: "1fab5794cf1740288abd82f2cebb9b1a"
format_stamp: "Formatted at 2024-05-02 21:28:44 on dist-test-slave-jdfx"
I20240502 21:28:44.497610 2547 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-9/wal/instance:
uuid: "1fab5794cf1740288abd82f2cebb9b1a"
format_stamp: "Formatted at 2024-05-02 21:28:44 on dist-test-slave-jdfx"
I20240502 21:28:44.502103 2547 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20240502 21:28:44.505249 2562 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:28:44.506371 2547 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.004s sys 0.000s
I20240502 21:28:44.506510 2547 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-9/data,/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-9/wal
uuid: "1fab5794cf1740288abd82f2cebb9b1a"
format_stamp: "Formatted at 2024-05-02 21:28:44 on dist-test-slave-jdfx"
I20240502 21:28:44.506688 2547 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-9/wal
metadata directory: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-9/wal
1 data directories: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-9/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:28:44.522516 2547 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:28:44.523223 2547 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:28:44.523461 2547 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:28:44.524310 2547 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:28:44.525662 2547 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:28:44.525753 2547 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:28:44.525853 2547 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:28:44.525923 2547 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 21:28:44.555510 2547 rpc_server.cc:307] RPC server started. Bound to: 127.15.255.138:33375
I20240502 21:28:44.555553 2675 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.255.138:33375 every 8 connection(s)
I20240502 21:28:44.556762 2547 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-9/data/info.pb
I20240502 21:28:44.557247 16382 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvNciQZ/build/asan/bin/kudu as pid 2547
I20240502 21:28:44.557457 16382 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-9/wal/instance
I20240502 21:28:44.568930 2676 heartbeater.cc:347] Connected to a master server at 127.15.255.190:43173
I20240502 21:28:44.569175 2676 heartbeater.cc:464] Registering TS with master...
I20240502 21:28:44.569885 2676 heartbeater.cc:511] Master 127.15.255.190:43173 requested a full tablet report, sending...
I20240502 21:28:44.612485 1270 ts_manager.cc:194] Registered new tserver with Master: 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138:33375)
I20240502 21:28:44.613361 1270 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.255.138:46721
I20240502 21:28:44.616698 16382 external_mini_cluster.cc:930] 10 TS(s) registered with all masters
I20240502 21:28:44.632395 16382 external_mini_cluster.cc:930] 10 TS(s) registered with all masters
I20240502 21:28:44.632547 16382 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvNciQZ/build/asan/bin/kudu pid 2277
I20240502 21:28:44.639678 2403 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 21:28:44.717872 16382 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvNciQZ/build/asan/bin/kudu with pid 2277
I20240502 21:28:44.732038 16382 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvNciQZ/build/asan/bin/kudu pid 2411
I20240502 21:28:44.739668 2539 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 21:28:44.817308 16382 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvNciQZ/build/asan/bin/kudu with pid 2411
I20240502 21:28:44.831511 16382 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskvNciQZ/build/asan/bin/kudu pid 2547
I20240502 21:28:44.839525 2671 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 21:28:44.916750 16382 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskvNciQZ/build/asan/bin/kudu with pid 2547
I20240502 21:28:48.682499 1270 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44934:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 7
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
W20240502 21:28:48.689121 1270 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table rebalance_test_table_0 in case of a server failure: 8 tablet servers would be needed, 7 are available. Consider bringing up more tablet servers.
I20240502 21:28:48.744175 1799 tablet_service.cc:1467] Processing CreateTablet for tablet 3c88daaac3734d918955a774829532c6 (DEFAULT_TABLE table=rebalance_test_table_0 [id=82737b0de4b945dc8ca0c7d6b92af37c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:48.746932 1798 tablet_service.cc:1467] Processing CreateTablet for tablet 8f86d1517df74ead9e397ca9bed44c3f (DEFAULT_TABLE table=rebalance_test_table_0 [id=82737b0de4b945dc8ca0c7d6b92af37c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:48.746603 2069 tablet_service.cc:1467] Processing CreateTablet for tablet 13af1b96121c492c9373f9acbb55725a (DEFAULT_TABLE table=rebalance_test_table_0 [id=82737b0de4b945dc8ca0c7d6b92af37c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:48.747033 1664 tablet_service.cc:1467] Processing CreateTablet for tablet 13af1b96121c492c9373f9acbb55725a (DEFAULT_TABLE table=rebalance_test_table_0 [id=82737b0de4b945dc8ca0c7d6b92af37c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:48.747906 1798 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f86d1517df74ead9e397ca9bed44c3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:48.747901 2069 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13af1b96121c492c9373f9acbb55725a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:48.749071 1664 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13af1b96121c492c9373f9acbb55725a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:48.751179 1800 tablet_service.cc:1467] Processing CreateTablet for tablet 13af1b96121c492c9373f9acbb55725a (DEFAULT_TABLE table=rebalance_test_table_0 [id=82737b0de4b945dc8ca0c7d6b92af37c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:48.751626 1800 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13af1b96121c492c9373f9acbb55725a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:48.746646 2067 tablet_service.cc:1467] Processing CreateTablet for tablet 8f86d1517df74ead9e397ca9bed44c3f (DEFAULT_TABLE table=rebalance_test_table_0 [id=82737b0de4b945dc8ca0c7d6b92af37c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:48.752063 2067 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f86d1517df74ead9e397ca9bed44c3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:48.753101 1799 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c88daaac3734d918955a774829532c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:48.754195 1933 tablet_service.cc:1467] Processing CreateTablet for tablet 13af1b96121c492c9373f9acbb55725a (DEFAULT_TABLE table=rebalance_test_table_0 [id=82737b0de4b945dc8ca0c7d6b92af37c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:48.755614 1933 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13af1b96121c492c9373f9acbb55725a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:48.756659 1931 tablet_service.cc:1467] Processing CreateTablet for tablet 8f86d1517df74ead9e397ca9bed44c3f (DEFAULT_TABLE table=rebalance_test_table_0 [id=82737b0de4b945dc8ca0c7d6b92af37c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:48.757160 1931 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f86d1517df74ead9e397ca9bed44c3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:48.760617 1932 tablet_service.cc:1467] Processing CreateTablet for tablet 3c88daaac3734d918955a774829532c6 (DEFAULT_TABLE table=rebalance_test_table_0 [id=82737b0de4b945dc8ca0c7d6b92af37c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:48.761065 1932 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c88daaac3734d918955a774829532c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:48.761569 2068 tablet_service.cc:1467] Processing CreateTablet for tablet 3c88daaac3734d918955a774829532c6 (DEFAULT_TABLE table=rebalance_test_table_0 [id=82737b0de4b945dc8ca0c7d6b92af37c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:48.762039 2068 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c88daaac3734d918955a774829532c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:48.770148 2711 tablet_bootstrap.cc:492] T 13af1b96121c492c9373f9acbb55725a P 153026dd7f8943cc93fedf6f48450fb4: Bootstrap starting.
I20240502 21:28:48.773564 2711 tablet_bootstrap.cc:654] T 13af1b96121c492c9373f9acbb55725a P 153026dd7f8943cc93fedf6f48450fb4: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:48.774904 2711 log.cc:824] T 13af1b96121c492c9373f9acbb55725a P 153026dd7f8943cc93fedf6f48450fb4: Log is configured to *not* fsync() on all Append() calls
I20240502 21:28:48.784852 2710 tablet_bootstrap.cc:492] T 8f86d1517df74ead9e397ca9bed44c3f P 521bb93035324cecbebed2f6f2a630f8: Bootstrap starting.
I20240502 21:28:48.784909 2711 tablet_bootstrap.cc:492] T 13af1b96121c492c9373f9acbb55725a P 153026dd7f8943cc93fedf6f48450fb4: No bootstrap required, opened a new log
I20240502 21:28:48.785553 2711 ts_tablet_manager.cc:1367] T 13af1b96121c492c9373f9acbb55725a P 153026dd7f8943cc93fedf6f48450fb4: Time spent bootstrapping tablet: real 0.016s user 0.013s sys 0.000s
I20240502 21:28:48.788290 2710 tablet_bootstrap.cc:654] T 8f86d1517df74ead9e397ca9bed44c3f P 521bb93035324cecbebed2f6f2a630f8: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:48.789497 1663 tablet_service.cc:1467] Processing CreateTablet for tablet 3c88daaac3734d918955a774829532c6 (DEFAULT_TABLE table=rebalance_test_table_0 [id=82737b0de4b945dc8ca0c7d6b92af37c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:48.789777 1662 tablet_service.cc:1467] Processing CreateTablet for tablet 8f86d1517df74ead9e397ca9bed44c3f (DEFAULT_TABLE table=rebalance_test_table_0 [id=82737b0de4b945dc8ca0c7d6b92af37c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:48.790047 1663 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c88daaac3734d918955a774829532c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:48.790375 1662 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f86d1517df74ead9e397ca9bed44c3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:48.791368 2710 log.cc:824] T 8f86d1517df74ead9e397ca9bed44c3f P 521bb93035324cecbebed2f6f2a630f8: Log is configured to *not* fsync() on all Append() calls
I20240502 21:28:48.802626 2711 raft_consensus.cc:348] T 13af1b96121c492c9373f9acbb55725a P 153026dd7f8943cc93fedf6f48450fb4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } }
I20240502 21:28:48.804489 2711 raft_consensus.cc:374] T 13af1b96121c492c9373f9acbb55725a P 153026dd7f8943cc93fedf6f48450fb4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:48.805433 2711 raft_consensus.cc:729] T 13af1b96121c492c9373f9acbb55725a P 153026dd7f8943cc93fedf6f48450fb4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 153026dd7f8943cc93fedf6f48450fb4, State: Initialized, Role: FOLLOWER
I20240502 21:28:48.806313 2711 consensus_queue.cc:260] T 13af1b96121c492c9373f9acbb55725a P 153026dd7f8943cc93fedf6f48450fb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } }
I20240502 21:28:48.807080 2710 tablet_bootstrap.cc:492] T 8f86d1517df74ead9e397ca9bed44c3f P 521bb93035324cecbebed2f6f2a630f8: No bootstrap required, opened a new log
I20240502 21:28:48.808912 2710 ts_tablet_manager.cc:1367] T 8f86d1517df74ead9e397ca9bed44c3f P 521bb93035324cecbebed2f6f2a630f8: Time spent bootstrapping tablet: real 0.024s user 0.013s sys 0.000s
I20240502 21:28:48.811482 2715 tablet_bootstrap.cc:492] T 8f86d1517df74ead9e397ca9bed44c3f P 16b3a337bc5e462f83ba895ad8660deb: Bootstrap starting.
I20240502 21:28:48.813603 2713 tablet_bootstrap.cc:492] T 13af1b96121c492c9373f9acbb55725a P e632c553ff4341328acf1cbaca678392: Bootstrap starting.
I20240502 21:28:48.814428 2711 ts_tablet_manager.cc:1398] T 13af1b96121c492c9373f9acbb55725a P 153026dd7f8943cc93fedf6f48450fb4: Time spent starting tablet: real 0.029s user 0.012s sys 0.004s
I20240502 21:28:48.819072 2711 tablet_bootstrap.cc:492] T 3c88daaac3734d918955a774829532c6 P 153026dd7f8943cc93fedf6f48450fb4: Bootstrap starting.
I20240502 21:28:48.820785 2711 tablet_bootstrap.cc:654] T 3c88daaac3734d918955a774829532c6 P 153026dd7f8943cc93fedf6f48450fb4: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:48.825673 2715 tablet_bootstrap.cc:654] T 8f86d1517df74ead9e397ca9bed44c3f P 16b3a337bc5e462f83ba895ad8660deb: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:48.825608 2710 raft_consensus.cc:348] T 8f86d1517df74ead9e397ca9bed44c3f P 521bb93035324cecbebed2f6f2a630f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } }
I20240502 21:28:48.827526 2715 log.cc:824] T 8f86d1517df74ead9e397ca9bed44c3f P 16b3a337bc5e462f83ba895ad8660deb: Log is configured to *not* fsync() on all Append() calls
I20240502 21:28:48.828545 2713 tablet_bootstrap.cc:654] T 13af1b96121c492c9373f9acbb55725a P e632c553ff4341328acf1cbaca678392: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:48.829751 2713 log.cc:824] T 13af1b96121c492c9373f9acbb55725a P e632c553ff4341328acf1cbaca678392: Log is configured to *not* fsync() on all Append() calls
I20240502 21:28:48.839309 2715 tablet_bootstrap.cc:492] T 8f86d1517df74ead9e397ca9bed44c3f P 16b3a337bc5e462f83ba895ad8660deb: No bootstrap required, opened a new log
I20240502 21:28:48.839630 2715 ts_tablet_manager.cc:1367] T 8f86d1517df74ead9e397ca9bed44c3f P 16b3a337bc5e462f83ba895ad8660deb: Time spent bootstrapping tablet: real 0.028s user 0.007s sys 0.000s
I20240502 21:28:48.839807 1528 tablet_service.cc:1467] Processing CreateTablet for tablet 13af1b96121c492c9373f9acbb55725a (DEFAULT_TABLE table=rebalance_test_table_0 [id=82737b0de4b945dc8ca0c7d6b92af37c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:48.840649 1392 tablet_service.cc:1467] Processing CreateTablet for tablet 8f86d1517df74ead9e397ca9bed44c3f (DEFAULT_TABLE table=rebalance_test_table_0 [id=82737b0de4b945dc8ca0c7d6b92af37c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:48.841547 1528 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13af1b96121c492c9373f9acbb55725a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:48.841574 1392 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f86d1517df74ead9e397ca9bed44c3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:48.844795 1394 tablet_service.cc:1467] Processing CreateTablet for tablet 13af1b96121c492c9373f9acbb55725a (DEFAULT_TABLE table=rebalance_test_table_0 [id=82737b0de4b945dc8ca0c7d6b92af37c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:48.845278 1394 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13af1b96121c492c9373f9acbb55725a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:48.849315 1527 tablet_service.cc:1467] Processing CreateTablet for tablet 3c88daaac3734d918955a774829532c6 (DEFAULT_TABLE table=rebalance_test_table_0 [id=82737b0de4b945dc8ca0c7d6b92af37c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:48.849546 1526 tablet_service.cc:1467] Processing CreateTablet for tablet 8f86d1517df74ead9e397ca9bed44c3f (DEFAULT_TABLE table=rebalance_test_table_0 [id=82737b0de4b945dc8ca0c7d6b92af37c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:48.849799 1527 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c88daaac3734d918955a774829532c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:48.850029 1526 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f86d1517df74ead9e397ca9bed44c3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:48.852054 2710 raft_consensus.cc:374] T 8f86d1517df74ead9e397ca9bed44c3f P 521bb93035324cecbebed2f6f2a630f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:48.852223 2710 raft_consensus.cc:729] T 8f86d1517df74ead9e397ca9bed44c3f P 521bb93035324cecbebed2f6f2a630f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 521bb93035324cecbebed2f6f2a630f8, State: Initialized, Role: FOLLOWER
I20240502 21:28:48.852972 2710 consensus_queue.cc:260] T 8f86d1517df74ead9e397ca9bed44c3f P 521bb93035324cecbebed2f6f2a630f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } }
I20240502 21:28:48.853255 2207 tablet_service.cc:1467] Processing CreateTablet for tablet 13af1b96121c492c9373f9acbb55725a (DEFAULT_TABLE table=rebalance_test_table_0 [id=82737b0de4b945dc8ca0c7d6b92af37c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:48.854640 2207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13af1b96121c492c9373f9acbb55725a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:48.855690 2715 raft_consensus.cc:348] T 8f86d1517df74ead9e397ca9bed44c3f P 16b3a337bc5e462f83ba895ad8660deb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } }
I20240502 21:28:48.856160 2715 raft_consensus.cc:374] T 8f86d1517df74ead9e397ca9bed44c3f P 16b3a337bc5e462f83ba895ad8660deb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:48.856257 2715 raft_consensus.cc:729] T 8f86d1517df74ead9e397ca9bed44c3f P 16b3a337bc5e462f83ba895ad8660deb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16b3a337bc5e462f83ba895ad8660deb, State: Initialized, Role: FOLLOWER
I20240502 21:28:48.856870 2206 tablet_service.cc:1467] Processing CreateTablet for tablet 3c88daaac3734d918955a774829532c6 (DEFAULT_TABLE table=rebalance_test_table_0 [id=82737b0de4b945dc8ca0c7d6b92af37c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:48.857060 2715 consensus_queue.cc:260] T 8f86d1517df74ead9e397ca9bed44c3f P 16b3a337bc5e462f83ba895ad8660deb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } }
I20240502 21:28:48.857311 2206 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c88daaac3734d918955a774829532c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:48.860898 2205 tablet_service.cc:1467] Processing CreateTablet for tablet 8f86d1517df74ead9e397ca9bed44c3f (DEFAULT_TABLE table=rebalance_test_table_0 [id=82737b0de4b945dc8ca0c7d6b92af37c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:48.861299 2205 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f86d1517df74ead9e397ca9bed44c3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:48.863967 2711 tablet_bootstrap.cc:492] T 3c88daaac3734d918955a774829532c6 P 153026dd7f8943cc93fedf6f48450fb4: No bootstrap required, opened a new log
I20240502 21:28:48.864158 2722 tablet_bootstrap.cc:492] T 8f86d1517df74ead9e397ca9bed44c3f P a62a935ff43d494ca35d0c417943133d: Bootstrap starting.
I20240502 21:28:48.864550 2711 ts_tablet_manager.cc:1367] T 3c88daaac3734d918955a774829532c6 P 153026dd7f8943cc93fedf6f48450fb4: Time spent bootstrapping tablet: real 0.046s user 0.019s sys 0.012s
I20240502 21:28:48.865509 2711 raft_consensus.cc:348] T 3c88daaac3734d918955a774829532c6 P 153026dd7f8943cc93fedf6f48450fb4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:28:48.865787 2711 raft_consensus.cc:374] T 3c88daaac3734d918955a774829532c6 P 153026dd7f8943cc93fedf6f48450fb4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:48.865893 2711 raft_consensus.cc:729] T 3c88daaac3734d918955a774829532c6 P 153026dd7f8943cc93fedf6f48450fb4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 153026dd7f8943cc93fedf6f48450fb4, State: Initialized, Role: FOLLOWER
I20240502 21:28:48.866149 2711 consensus_queue.cc:260] T 3c88daaac3734d918955a774829532c6 P 153026dd7f8943cc93fedf6f48450fb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:28:48.866706 2711 ts_tablet_manager.cc:1398] T 3c88daaac3734d918955a774829532c6 P 153026dd7f8943cc93fedf6f48450fb4: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 21:28:48.867040 2711 tablet_bootstrap.cc:492] T 8f86d1517df74ead9e397ca9bed44c3f P 153026dd7f8943cc93fedf6f48450fb4: Bootstrap starting.
I20240502 21:28:48.839802 1393 tablet_service.cc:1467] Processing CreateTablet for tablet 3c88daaac3734d918955a774829532c6 (DEFAULT_TABLE table=rebalance_test_table_0 [id=82737b0de4b945dc8ca0c7d6b92af37c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:48.868646 2711 tablet_bootstrap.cc:654] T 8f86d1517df74ead9e397ca9bed44c3f P 153026dd7f8943cc93fedf6f48450fb4: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:48.868834 2710 ts_tablet_manager.cc:1398] T 8f86d1517df74ead9e397ca9bed44c3f P 521bb93035324cecbebed2f6f2a630f8: Time spent starting tablet: real 0.059s user 0.015s sys 0.005s
I20240502 21:28:48.868911 1393 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c88daaac3734d918955a774829532c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:48.870107 2710 tablet_bootstrap.cc:492] T 3c88daaac3734d918955a774829532c6 P 521bb93035324cecbebed2f6f2a630f8: Bootstrap starting.
I20240502 21:28:48.871821 2710 tablet_bootstrap.cc:654] T 3c88daaac3734d918955a774829532c6 P 521bb93035324cecbebed2f6f2a630f8: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:48.875523 2715 ts_tablet_manager.cc:1398] T 8f86d1517df74ead9e397ca9bed44c3f P 16b3a337bc5e462f83ba895ad8660deb: Time spent starting tablet: real 0.036s user 0.013s sys 0.004s
I20240502 21:28:48.875937 2715 tablet_bootstrap.cc:492] T 3c88daaac3734d918955a774829532c6 P 16b3a337bc5e462f83ba895ad8660deb: Bootstrap starting.
I20240502 21:28:48.877547 2715 tablet_bootstrap.cc:654] T 3c88daaac3734d918955a774829532c6 P 16b3a337bc5e462f83ba895ad8660deb: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:48.878978 2722 tablet_bootstrap.cc:654] T 8f86d1517df74ead9e397ca9bed44c3f P a62a935ff43d494ca35d0c417943133d: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:48.880707 2710 tablet_bootstrap.cc:492] T 3c88daaac3734d918955a774829532c6 P 521bb93035324cecbebed2f6f2a630f8: No bootstrap required, opened a new log
I20240502 21:28:48.880869 2710 ts_tablet_manager.cc:1367] T 3c88daaac3734d918955a774829532c6 P 521bb93035324cecbebed2f6f2a630f8: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.001s
I20240502 21:28:48.881589 2722 log.cc:824] T 8f86d1517df74ead9e397ca9bed44c3f P a62a935ff43d494ca35d0c417943133d: Log is configured to *not* fsync() on all Append() calls
I20240502 21:28:48.881862 2710 raft_consensus.cc:348] T 3c88daaac3734d918955a774829532c6 P 521bb93035324cecbebed2f6f2a630f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:28:48.882179 2710 raft_consensus.cc:374] T 3c88daaac3734d918955a774829532c6 P 521bb93035324cecbebed2f6f2a630f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:48.882275 2710 raft_consensus.cc:729] T 3c88daaac3734d918955a774829532c6 P 521bb93035324cecbebed2f6f2a630f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 521bb93035324cecbebed2f6f2a630f8, State: Initialized, Role: FOLLOWER
I20240502 21:28:48.882601 2710 consensus_queue.cc:260] T 3c88daaac3734d918955a774829532c6 P 521bb93035324cecbebed2f6f2a630f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:28:48.883286 2710 ts_tablet_manager.cc:1398] T 3c88daaac3734d918955a774829532c6 P 521bb93035324cecbebed2f6f2a630f8: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 21:28:48.883571 2710 tablet_bootstrap.cc:492] T 13af1b96121c492c9373f9acbb55725a P 521bb93035324cecbebed2f6f2a630f8: Bootstrap starting.
I20240502 21:28:48.885155 2710 tablet_bootstrap.cc:654] T 13af1b96121c492c9373f9acbb55725a P 521bb93035324cecbebed2f6f2a630f8: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:48.886997 2711 tablet_bootstrap.cc:492] T 8f86d1517df74ead9e397ca9bed44c3f P 153026dd7f8943cc93fedf6f48450fb4: No bootstrap required, opened a new log
I20240502 21:28:48.887195 2711 ts_tablet_manager.cc:1367] T 8f86d1517df74ead9e397ca9bed44c3f P 153026dd7f8943cc93fedf6f48450fb4: Time spent bootstrapping tablet: real 0.020s user 0.008s sys 0.004s
I20240502 21:28:48.887925 2711 raft_consensus.cc:348] T 8f86d1517df74ead9e397ca9bed44c3f P 153026dd7f8943cc93fedf6f48450fb4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } }
I20240502 21:28:48.888283 2711 raft_consensus.cc:374] T 8f86d1517df74ead9e397ca9bed44c3f P 153026dd7f8943cc93fedf6f48450fb4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:48.888366 2711 raft_consensus.cc:729] T 8f86d1517df74ead9e397ca9bed44c3f P 153026dd7f8943cc93fedf6f48450fb4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 153026dd7f8943cc93fedf6f48450fb4, State: Initialized, Role: FOLLOWER
I20240502 21:28:48.888679 2711 consensus_queue.cc:260] T 8f86d1517df74ead9e397ca9bed44c3f P 153026dd7f8943cc93fedf6f48450fb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } }
I20240502 21:28:48.889421 2711 ts_tablet_manager.cc:1398] T 8f86d1517df74ead9e397ca9bed44c3f P 153026dd7f8943cc93fedf6f48450fb4: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
W20240502 21:28:48.891299 1866 tablet.cc:2369] T 13af1b96121c492c9373f9acbb55725a P 153026dd7f8943cc93fedf6f48450fb4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 21:28:48.901543 2722 tablet_bootstrap.cc:492] T 8f86d1517df74ead9e397ca9bed44c3f P a62a935ff43d494ca35d0c417943133d: No bootstrap required, opened a new log
I20240502 21:28:48.902014 2722 ts_tablet_manager.cc:1367] T 8f86d1517df74ead9e397ca9bed44c3f P a62a935ff43d494ca35d0c417943133d: Time spent bootstrapping tablet: real 0.038s user 0.010s sys 0.004s
I20240502 21:28:48.912377 2722 raft_consensus.cc:348] T 8f86d1517df74ead9e397ca9bed44c3f P a62a935ff43d494ca35d0c417943133d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } }
I20240502 21:28:48.912967 2722 raft_consensus.cc:374] T 8f86d1517df74ead9e397ca9bed44c3f P a62a935ff43d494ca35d0c417943133d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:48.913107 2722 raft_consensus.cc:729] T 8f86d1517df74ead9e397ca9bed44c3f P a62a935ff43d494ca35d0c417943133d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a62a935ff43d494ca35d0c417943133d, State: Initialized, Role: FOLLOWER
I20240502 21:28:48.913900 2722 consensus_queue.cc:260] T 8f86d1517df74ead9e397ca9bed44c3f P a62a935ff43d494ca35d0c417943133d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } }
I20240502 21:28:48.916177 2720 raft_consensus.cc:482] T 8f86d1517df74ead9e397ca9bed44c3f P 521bb93035324cecbebed2f6f2a630f8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:28:48.916446 2720 raft_consensus.cc:504] T 8f86d1517df74ead9e397ca9bed44c3f P 521bb93035324cecbebed2f6f2a630f8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } }
I20240502 21:28:48.920832 2720 leader_election.cc:288] T 8f86d1517df74ead9e397ca9bed44c3f P 521bb93035324cecbebed2f6f2a630f8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7d1e5c1b047443cbb28d7b0a9744babf (127.15.255.129:37897), a62a935ff43d494ca35d0c417943133d (127.15.255.130:37991), 16b3a337bc5e462f83ba895ad8660deb (127.15.255.131:36033), 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132:44947), e632c553ff4341328acf1cbaca678392 (127.15.255.133:37439), 6ab8df3c796f4b6cb3e29d5045ad3ea0 (127.15.255.135:34701)
I20240502 21:28:48.935941 2728 tablet_bootstrap.cc:492] T 13af1b96121c492c9373f9acbb55725a P 7d1e5c1b047443cbb28d7b0a9744babf: Bootstrap starting.
I20240502 21:28:48.940215 2730 tablet_bootstrap.cc:492] T 3c88daaac3734d918955a774829532c6 P 6ab8df3c796f4b6cb3e29d5045ad3ea0: Bootstrap starting.
I20240502 21:28:48.943519 2730 tablet_bootstrap.cc:654] T 3c88daaac3734d918955a774829532c6 P 6ab8df3c796f4b6cb3e29d5045ad3ea0: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:48.947919 2713 tablet_bootstrap.cc:492] T 13af1b96121c492c9373f9acbb55725a P e632c553ff4341328acf1cbaca678392: No bootstrap required, opened a new log
I20240502 21:28:48.948305 2713 ts_tablet_manager.cc:1367] T 13af1b96121c492c9373f9acbb55725a P e632c553ff4341328acf1cbaca678392: Time spent bootstrapping tablet: real 0.135s user 0.009s sys 0.006s
I20240502 21:28:48.948444 2715 tablet_bootstrap.cc:492] T 3c88daaac3734d918955a774829532c6 P 16b3a337bc5e462f83ba895ad8660deb: No bootstrap required, opened a new log
I20240502 21:28:48.948689 2715 ts_tablet_manager.cc:1367] T 3c88daaac3734d918955a774829532c6 P 16b3a337bc5e462f83ba895ad8660deb: Time spent bootstrapping tablet: real 0.073s user 0.015s sys 0.031s
I20240502 21:28:48.949725 2715 raft_consensus.cc:348] T 3c88daaac3734d918955a774829532c6 P 16b3a337bc5e462f83ba895ad8660deb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:28:48.950101 2715 raft_consensus.cc:374] T 3c88daaac3734d918955a774829532c6 P 16b3a337bc5e462f83ba895ad8660deb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:48.950233 2715 raft_consensus.cc:729] T 3c88daaac3734d918955a774829532c6 P 16b3a337bc5e462f83ba895ad8660deb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16b3a337bc5e462f83ba895ad8660deb, State: Initialized, Role: FOLLOWER
I20240502 21:28:48.950687 2715 consensus_queue.cc:260] T 3c88daaac3734d918955a774829532c6 P 16b3a337bc5e462f83ba895ad8660deb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:28:48.953788 2730 log.cc:824] T 3c88daaac3734d918955a774829532c6 P 6ab8df3c796f4b6cb3e29d5045ad3ea0: Log is configured to *not* fsync() on all Append() calls
I20240502 21:28:48.953948 2728 tablet_bootstrap.cc:654] T 13af1b96121c492c9373f9acbb55725a P 7d1e5c1b047443cbb28d7b0a9744babf: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:48.954104 2715 ts_tablet_manager.cc:1398] T 3c88daaac3734d918955a774829532c6 P 16b3a337bc5e462f83ba895ad8660deb: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240502 21:28:48.955241 2728 log.cc:824] T 13af1b96121c492c9373f9acbb55725a P 7d1e5c1b047443cbb28d7b0a9744babf: Log is configured to *not* fsync() on all Append() calls
W20240502 21:28:48.955602 1733 tablet.cc:2369] T 3c88daaac3734d918955a774829532c6 P 16b3a337bc5e462f83ba895ad8660deb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 21:28:48.957418 2710 tablet_bootstrap.cc:492] T 13af1b96121c492c9373f9acbb55725a P 521bb93035324cecbebed2f6f2a630f8: No bootstrap required, opened a new log
I20240502 21:28:48.957597 2710 ts_tablet_manager.cc:1367] T 13af1b96121c492c9373f9acbb55725a P 521bb93035324cecbebed2f6f2a630f8: Time spent bootstrapping tablet: real 0.074s user 0.027s sys 0.026s
I20240502 21:28:48.958325 2710 raft_consensus.cc:348] T 13af1b96121c492c9373f9acbb55725a P 521bb93035324cecbebed2f6f2a630f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } }
I20240502 21:28:48.958626 2710 raft_consensus.cc:374] T 13af1b96121c492c9373f9acbb55725a P 521bb93035324cecbebed2f6f2a630f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:48.958688 2710 raft_consensus.cc:729] T 13af1b96121c492c9373f9acbb55725a P 521bb93035324cecbebed2f6f2a630f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 521bb93035324cecbebed2f6f2a630f8, State: Initialized, Role: FOLLOWER
I20240502 21:28:48.958931 2710 consensus_queue.cc:260] T 13af1b96121c492c9373f9acbb55725a P 521bb93035324cecbebed2f6f2a630f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } }
I20240502 21:28:48.959537 2710 ts_tablet_manager.cc:1398] T 13af1b96121c492c9373f9acbb55725a P 521bb93035324cecbebed2f6f2a630f8: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 21:28:48.959740 1686 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f86d1517df74ead9e397ca9bed44c3f" candidate_uuid: "521bb93035324cecbebed2f6f2a630f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16b3a337bc5e462f83ba895ad8660deb" is_pre_election: true
I20240502 21:28:48.960248 1686 raft_consensus.cc:2454] T 8f86d1517df74ead9e397ca9bed44c3f P 16b3a337bc5e462f83ba895ad8660deb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 521bb93035324cecbebed2f6f2a630f8 in term 0.
I20240502 21:28:48.958865 1955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f86d1517df74ead9e397ca9bed44c3f" candidate_uuid: "521bb93035324cecbebed2f6f2a630f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e632c553ff4341328acf1cbaca678392" is_pre_election: true
I20240502 21:28:48.957471 1415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f86d1517df74ead9e397ca9bed44c3f" candidate_uuid: "521bb93035324cecbebed2f6f2a630f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" is_pre_election: true
W20240502 21:28:48.962709 2022 leader_election.cc:341] T 8f86d1517df74ead9e397ca9bed44c3f P 521bb93035324cecbebed2f6f2a630f8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e632c553ff4341328acf1cbaca678392 (127.15.255.133:37439): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:28:48.963367 2715 tablet_bootstrap.cc:492] T 13af1b96121c492c9373f9acbb55725a P 16b3a337bc5e462f83ba895ad8660deb: Bootstrap starting.
I20240502 21:28:48.964087 2722 ts_tablet_manager.cc:1398] T 8f86d1517df74ead9e397ca9bed44c3f P a62a935ff43d494ca35d0c417943133d: Time spent starting tablet: real 0.062s user 0.010s sys 0.012s
I20240502 21:28:48.965085 2715 tablet_bootstrap.cc:654] T 13af1b96121c492c9373f9acbb55725a P 16b3a337bc5e462f83ba895ad8660deb: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:48.964867 1820 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f86d1517df74ead9e397ca9bed44c3f" candidate_uuid: "521bb93035324cecbebed2f6f2a630f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "153026dd7f8943cc93fedf6f48450fb4" is_pre_election: true
I20240502 21:28:48.965387 1820 raft_consensus.cc:2454] T 8f86d1517df74ead9e397ca9bed44c3f P 153026dd7f8943cc93fedf6f48450fb4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 521bb93035324cecbebed2f6f2a630f8 in term 0.
W20240502 21:28:48.965921 2023 leader_election.cc:341] T 8f86d1517df74ead9e397ca9bed44c3f P 521bb93035324cecbebed2f6f2a630f8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7d1e5c1b047443cbb28d7b0a9744babf (127.15.255.129:37897): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:28:48.967999 2722 tablet_bootstrap.cc:492] T 3c88daaac3734d918955a774829532c6 P a62a935ff43d494ca35d0c417943133d: Bootstrap starting.
I20240502 21:28:48.970497 2722 tablet_bootstrap.cc:654] T 3c88daaac3734d918955a774829532c6 P a62a935ff43d494ca35d0c417943133d: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:48.971844 2728 tablet_bootstrap.cc:492] T 13af1b96121c492c9373f9acbb55725a P 7d1e5c1b047443cbb28d7b0a9744babf: No bootstrap required, opened a new log
I20240502 21:28:48.974561 2728 ts_tablet_manager.cc:1367] T 13af1b96121c492c9373f9acbb55725a P 7d1e5c1b047443cbb28d7b0a9744babf: Time spent bootstrapping tablet: real 0.039s user 0.007s sys 0.004s
I20240502 21:28:48.975344 2713 raft_consensus.cc:348] T 13af1b96121c492c9373f9acbb55725a P e632c553ff4341328acf1cbaca678392 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } }
I20240502 21:28:48.977420 1550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f86d1517df74ead9e397ca9bed44c3f" candidate_uuid: "521bb93035324cecbebed2f6f2a630f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a62a935ff43d494ca35d0c417943133d" is_pre_election: true
I20240502 21:28:48.977985 2730 tablet_bootstrap.cc:492] T 3c88daaac3734d918955a774829532c6 P 6ab8df3c796f4b6cb3e29d5045ad3ea0: No bootstrap required, opened a new log
I20240502 21:28:48.977965 1550 raft_consensus.cc:2454] T 8f86d1517df74ead9e397ca9bed44c3f P a62a935ff43d494ca35d0c417943133d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 521bb93035324cecbebed2f6f2a630f8 in term 0.
I20240502 21:28:48.978279 2730 ts_tablet_manager.cc:1367] T 3c88daaac3734d918955a774829532c6 P 6ab8df3c796f4b6cb3e29d5045ad3ea0: Time spent bootstrapping tablet: real 0.038s user 0.008s sys 0.000s
I20240502 21:28:48.978677 2023 leader_election.cc:302] T 8f86d1517df74ead9e397ca9bed44c3f P 521bb93035324cecbebed2f6f2a630f8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 6 responses out of 7 voters: 4 yes votes; 2 no votes. yes voters: 153026dd7f8943cc93fedf6f48450fb4, 16b3a337bc5e462f83ba895ad8660deb, 521bb93035324cecbebed2f6f2a630f8, a62a935ff43d494ca35d0c417943133d; no voters: 7d1e5c1b047443cbb28d7b0a9744babf, e632c553ff4341328acf1cbaca678392
I20240502 21:28:48.979202 2720 raft_consensus.cc:2781] T 8f86d1517df74ead9e397ca9bed44c3f P 521bb93035324cecbebed2f6f2a630f8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:28:48.979348 2720 raft_consensus.cc:482] T 8f86d1517df74ead9e397ca9bed44c3f P 521bb93035324cecbebed2f6f2a630f8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:28:48.979429 2720 raft_consensus.cc:3037] T 8f86d1517df74ead9e397ca9bed44c3f P 521bb93035324cecbebed2f6f2a630f8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:48.982529 2720 raft_consensus.cc:504] T 8f86d1517df74ead9e397ca9bed44c3f P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } }
I20240502 21:28:48.984000 1415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f86d1517df74ead9e397ca9bed44c3f" candidate_uuid: "521bb93035324cecbebed2f6f2a630f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d1e5c1b047443cbb28d7b0a9744babf"
I20240502 21:28:48.984560 2713 raft_consensus.cc:374] T 13af1b96121c492c9373f9acbb55725a P e632c553ff4341328acf1cbaca678392 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20240502 21:28:48.984624 2023 leader_election.cc:341] T 8f86d1517df74ead9e397ca9bed44c3f P 521bb93035324cecbebed2f6f2a630f8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7d1e5c1b047443cbb28d7b0a9744babf (127.15.255.129:37897): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:28:48.984733 2713 raft_consensus.cc:729] T 13af1b96121c492c9373f9acbb55725a P e632c553ff4341328acf1cbaca678392 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e632c553ff4341328acf1cbaca678392, State: Initialized, Role: FOLLOWER
I20240502 21:28:48.985596 2713 consensus_queue.cc:260] T 13af1b96121c492c9373f9acbb55725a P e632c553ff4341328acf1cbaca678392 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } }
I20240502 21:28:48.985772 1550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f86d1517df74ead9e397ca9bed44c3f" candidate_uuid: "521bb93035324cecbebed2f6f2a630f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a62a935ff43d494ca35d0c417943133d"
I20240502 21:28:48.985966 1550 raft_consensus.cc:3037] T 8f86d1517df74ead9e397ca9bed44c3f P a62a935ff43d494ca35d0c417943133d [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:48.989152 1550 raft_consensus.cc:2454] T 8f86d1517df74ead9e397ca9bed44c3f P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 521bb93035324cecbebed2f6f2a630f8 in term 1.
I20240502 21:28:48.990361 1686 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f86d1517df74ead9e397ca9bed44c3f" candidate_uuid: "521bb93035324cecbebed2f6f2a630f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16b3a337bc5e462f83ba895ad8660deb"
I20240502 21:28:48.990604 1686 raft_consensus.cc:3037] T 8f86d1517df74ead9e397ca9bed44c3f P 16b3a337bc5e462f83ba895ad8660deb [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:48.993913 1686 raft_consensus.cc:2454] T 8f86d1517df74ead9e397ca9bed44c3f P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 521bb93035324cecbebed2f6f2a630f8 in term 1.
I20240502 21:28:48.994585 2728 raft_consensus.cc:348] T 13af1b96121c492c9373f9acbb55725a P 7d1e5c1b047443cbb28d7b0a9744babf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } }
I20240502 21:28:48.995023 1820 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f86d1517df74ead9e397ca9bed44c3f" candidate_uuid: "521bb93035324cecbebed2f6f2a630f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "153026dd7f8943cc93fedf6f48450fb4"
I20240502 21:28:48.995226 1820 raft_consensus.cc:3037] T 8f86d1517df74ead9e397ca9bed44c3f P 153026dd7f8943cc93fedf6f48450fb4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:48.996676 2715 tablet_bootstrap.cc:492] T 13af1b96121c492c9373f9acbb55725a P 16b3a337bc5e462f83ba895ad8660deb: No bootstrap required, opened a new log
I20240502 21:28:48.996837 2715 ts_tablet_manager.cc:1367] T 13af1b96121c492c9373f9acbb55725a P 16b3a337bc5e462f83ba895ad8660deb: Time spent bootstrapping tablet: real 0.034s user 0.012s sys 0.020s
I20240502 21:28:48.996892 2713 ts_tablet_manager.cc:1398] T 13af1b96121c492c9373f9acbb55725a P e632c553ff4341328acf1cbaca678392: Time spent starting tablet: real 0.048s user 0.012s sys 0.000s
I20240502 21:28:48.997300 2713 tablet_bootstrap.cc:492] T 3c88daaac3734d918955a774829532c6 P e632c553ff4341328acf1cbaca678392: Bootstrap starting.
I20240502 21:28:48.997543 2715 raft_consensus.cc:348] T 13af1b96121c492c9373f9acbb55725a P 16b3a337bc5e462f83ba895ad8660deb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } }
I20240502 21:28:48.997792 2715 raft_consensus.cc:374] T 13af1b96121c492c9373f9acbb55725a P 16b3a337bc5e462f83ba895ad8660deb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:48.997857 2715 raft_consensus.cc:729] T 13af1b96121c492c9373f9acbb55725a P 16b3a337bc5e462f83ba895ad8660deb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16b3a337bc5e462f83ba895ad8660deb, State: Initialized, Role: FOLLOWER
I20240502 21:28:48.998133 2728 raft_consensus.cc:374] T 13af1b96121c492c9373f9acbb55725a P 7d1e5c1b047443cbb28d7b0a9744babf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:48.998240 2728 raft_consensus.cc:729] T 13af1b96121c492c9373f9acbb55725a P 7d1e5c1b047443cbb28d7b0a9744babf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d1e5c1b047443cbb28d7b0a9744babf, State: Initialized, Role: FOLLOWER
I20240502 21:28:48.998112 2715 consensus_queue.cc:260] T 13af1b96121c492c9373f9acbb55725a P 16b3a337bc5e462f83ba895ad8660deb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } }
I20240502 21:28:48.998796 2715 ts_tablet_manager.cc:1398] T 13af1b96121c492c9373f9acbb55725a P 16b3a337bc5e462f83ba895ad8660deb: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 21:28:48.999788 2713 tablet_bootstrap.cc:654] T 3c88daaac3734d918955a774829532c6 P e632c553ff4341328acf1cbaca678392: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:49.000921 2728 consensus_queue.cc:260] T 13af1b96121c492c9373f9acbb55725a P 7d1e5c1b047443cbb28d7b0a9744babf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } }
I20240502 21:28:49.001811 2730 raft_consensus.cc:348] T 3c88daaac3734d918955a774829532c6 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:28:49.002135 2730 raft_consensus.cc:374] T 3c88daaac3734d918955a774829532c6 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:49.002194 2730 raft_consensus.cc:729] T 3c88daaac3734d918955a774829532c6 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ab8df3c796f4b6cb3e29d5045ad3ea0, State: Initialized, Role: FOLLOWER
I20240502 21:28:49.002697 2730 consensus_queue.cc:260] T 3c88daaac3734d918955a774829532c6 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:28:49.003804 2728 ts_tablet_manager.cc:1398] T 13af1b96121c492c9373f9acbb55725a P 7d1e5c1b047443cbb28d7b0a9744babf: Time spent starting tablet: real 0.029s user 0.008s sys 0.000s
I20240502 21:28:49.004123 2728 tablet_bootstrap.cc:492] T 3c88daaac3734d918955a774829532c6 P 7d1e5c1b047443cbb28d7b0a9744babf: Bootstrap starting.
I20240502 21:28:49.005368 2713 tablet_bootstrap.cc:492] T 3c88daaac3734d918955a774829532c6 P e632c553ff4341328acf1cbaca678392: No bootstrap required, opened a new log
I20240502 21:28:49.005621 2728 tablet_bootstrap.cc:654] T 3c88daaac3734d918955a774829532c6 P 7d1e5c1b047443cbb28d7b0a9744babf: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:49.005872 2713 ts_tablet_manager.cc:1367] T 3c88daaac3734d918955a774829532c6 P e632c553ff4341328acf1cbaca678392: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.000s
W20240502 21:28:49.006417 1598 tablet.cc:2369] T 8f86d1517df74ead9e397ca9bed44c3f P a62a935ff43d494ca35d0c417943133d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 21:28:49.006811 2713 raft_consensus.cc:348] T 3c88daaac3734d918955a774829532c6 P e632c553ff4341328acf1cbaca678392 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:28:49.007102 2713 raft_consensus.cc:374] T 3c88daaac3734d918955a774829532c6 P e632c553ff4341328acf1cbaca678392 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:49.007192 2713 raft_consensus.cc:729] T 3c88daaac3734d918955a774829532c6 P e632c553ff4341328acf1cbaca678392 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e632c553ff4341328acf1cbaca678392, State: Initialized, Role: FOLLOWER
I20240502 21:28:49.007452 2713 consensus_queue.cc:260] T 3c88daaac3734d918955a774829532c6 P e632c553ff4341328acf1cbaca678392 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:28:49.008122 2722 tablet_bootstrap.cc:492] T 3c88daaac3734d918955a774829532c6 P a62a935ff43d494ca35d0c417943133d: No bootstrap required, opened a new log
I20240502 21:28:49.008282 2722 ts_tablet_manager.cc:1367] T 3c88daaac3734d918955a774829532c6 P a62a935ff43d494ca35d0c417943133d: Time spent bootstrapping tablet: real 0.040s user 0.017s sys 0.007s
I20240502 21:28:49.008633 2713 ts_tablet_manager.cc:1398] T 3c88daaac3734d918955a774829532c6 P e632c553ff4341328acf1cbaca678392: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 21:28:49.008913 2713 tablet_bootstrap.cc:492] T 8f86d1517df74ead9e397ca9bed44c3f P e632c553ff4341328acf1cbaca678392: Bootstrap starting.
I20240502 21:28:49.009193 2722 raft_consensus.cc:348] T 3c88daaac3734d918955a774829532c6 P a62a935ff43d494ca35d0c417943133d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:28:49.009456 2722 raft_consensus.cc:374] T 3c88daaac3734d918955a774829532c6 P a62a935ff43d494ca35d0c417943133d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:49.009560 2722 raft_consensus.cc:729] T 3c88daaac3734d918955a774829532c6 P a62a935ff43d494ca35d0c417943133d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a62a935ff43d494ca35d0c417943133d, State: Initialized, Role: FOLLOWER
I20240502 21:28:49.009819 2722 consensus_queue.cc:260] T 3c88daaac3734d918955a774829532c6 P a62a935ff43d494ca35d0c417943133d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:28:49.010452 2722 ts_tablet_manager.cc:1398] T 3c88daaac3734d918955a774829532c6 P a62a935ff43d494ca35d0c417943133d: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240502 21:28:49.010743 2722 tablet_bootstrap.cc:492] T 13af1b96121c492c9373f9acbb55725a P a62a935ff43d494ca35d0c417943133d: Bootstrap starting.
I20240502 21:28:49.012303 2722 tablet_bootstrap.cc:654] T 13af1b96121c492c9373f9acbb55725a P a62a935ff43d494ca35d0c417943133d: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:49.013872 2730 ts_tablet_manager.cc:1398] T 3c88daaac3734d918955a774829532c6 P 6ab8df3c796f4b6cb3e29d5045ad3ea0: Time spent starting tablet: real 0.035s user 0.013s sys 0.000s
I20240502 21:28:49.014310 2730 tablet_bootstrap.cc:492] T 8f86d1517df74ead9e397ca9bed44c3f P 6ab8df3c796f4b6cb3e29d5045ad3ea0: Bootstrap starting.
I20240502 21:28:49.015931 2730 tablet_bootstrap.cc:654] T 8f86d1517df74ead9e397ca9bed44c3f P 6ab8df3c796f4b6cb3e29d5045ad3ea0: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:49.010607 2713 tablet_bootstrap.cc:654] T 8f86d1517df74ead9e397ca9bed44c3f P e632c553ff4341328acf1cbaca678392: Neither blocks nor log segments found. Creating new log.
W20240502 21:28:49.019879 2139 tablet.cc:2369] T 13af1b96121c492c9373f9acbb55725a P 521bb93035324cecbebed2f6f2a630f8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 21:28:49.022778 2713 tablet_bootstrap.cc:492] T 8f86d1517df74ead9e397ca9bed44c3f P e632c553ff4341328acf1cbaca678392: No bootstrap required, opened a new log
I20240502 21:28:49.023007 2713 ts_tablet_manager.cc:1367] T 8f86d1517df74ead9e397ca9bed44c3f P e632c553ff4341328acf1cbaca678392: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.003s
I20240502 21:28:49.023487 2721 raft_consensus.cc:482] T 3c88daaac3734d918955a774829532c6 P 16b3a337bc5e462f83ba895ad8660deb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:28:49.023746 2713 raft_consensus.cc:348] T 8f86d1517df74ead9e397ca9bed44c3f P e632c553ff4341328acf1cbaca678392 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } }
I20240502 21:28:49.023676 2721 raft_consensus.cc:504] T 3c88daaac3734d918955a774829532c6 P 16b3a337bc5e462f83ba895ad8660deb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:28:49.024025 2713 raft_consensus.cc:374] T 8f86d1517df74ead9e397ca9bed44c3f P e632c553ff4341328acf1cbaca678392 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:49.024084 2713 raft_consensus.cc:729] T 8f86d1517df74ead9e397ca9bed44c3f P e632c553ff4341328acf1cbaca678392 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e632c553ff4341328acf1cbaca678392, State: Initialized, Role: FOLLOWER
I20240502 21:28:49.024313 2713 consensus_queue.cc:260] T 8f86d1517df74ead9e397ca9bed44c3f P e632c553ff4341328acf1cbaca678392 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } }
I20240502 21:28:49.025673 2713 ts_tablet_manager.cc:1398] T 8f86d1517df74ead9e397ca9bed44c3f P e632c553ff4341328acf1cbaca678392: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 21:28:49.026712 2721 leader_election.cc:288] T 3c88daaac3734d918955a774829532c6 P 16b3a337bc5e462f83ba895ad8660deb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e632c553ff4341328acf1cbaca678392 (127.15.255.133:37439), 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132:44947), 521bb93035324cecbebed2f6f2a630f8 (127.15.255.134:40821), 6ab8df3c796f4b6cb3e29d5045ad3ea0 (127.15.255.135:34701), 7d1e5c1b047443cbb28d7b0a9744babf (127.15.255.129:37897), a62a935ff43d494ca35d0c417943133d (127.15.255.130:37991)
I20240502 21:28:49.047451 1820 raft_consensus.cc:2454] T 8f86d1517df74ead9e397ca9bed44c3f P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 521bb93035324cecbebed2f6f2a630f8 in term 1.
I20240502 21:28:49.048074 2024 leader_election.cc:302] T 8f86d1517df74ead9e397ca9bed44c3f P 521bb93035324cecbebed2f6f2a630f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 5 responses out of 7 voters: 4 yes votes; 1 no votes. yes voters: 153026dd7f8943cc93fedf6f48450fb4, 16b3a337bc5e462f83ba895ad8660deb, 521bb93035324cecbebed2f6f2a630f8, a62a935ff43d494ca35d0c417943133d; no voters: 7d1e5c1b047443cbb28d7b0a9744babf
I20240502 21:28:49.053330 1955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f86d1517df74ead9e397ca9bed44c3f" candidate_uuid: "521bb93035324cecbebed2f6f2a630f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e632c553ff4341328acf1cbaca678392"
I20240502 21:28:49.053602 1955 raft_consensus.cc:3037] T 8f86d1517df74ead9e397ca9bed44c3f P e632c553ff4341328acf1cbaca678392 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:49.056766 2729 raft_consensus.cc:482] T 3c88daaac3734d918955a774829532c6 P a62a935ff43d494ca35d0c417943133d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:28:49.056983 2720 leader_election.cc:288] T 8f86d1517df74ead9e397ca9bed44c3f P 521bb93035324cecbebed2f6f2a630f8 [CANDIDATE]: Term 1 election: Requested vote from peers 7d1e5c1b047443cbb28d7b0a9744babf (127.15.255.129:37897), a62a935ff43d494ca35d0c417943133d (127.15.255.130:37991), 16b3a337bc5e462f83ba895ad8660deb (127.15.255.131:36033), 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132:44947), e632c553ff4341328acf1cbaca678392 (127.15.255.133:37439), 6ab8df3c796f4b6cb3e29d5045ad3ea0 (127.15.255.135:34701)
I20240502 21:28:49.057186 2720 raft_consensus.cc:2781] T 8f86d1517df74ead9e397ca9bed44c3f P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:28:49.056926 2729 raft_consensus.cc:504] T 3c88daaac3734d918955a774829532c6 P a62a935ff43d494ca35d0c417943133d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:28:49.057314 2720 raft_consensus.cc:686] T 8f86d1517df74ead9e397ca9bed44c3f P 521bb93035324cecbebed2f6f2a630f8 [term 1 LEADER]: Becoming Leader. State: Replica: 521bb93035324cecbebed2f6f2a630f8, State: Running, Role: LEADER
I20240502 21:28:49.057864 2720 consensus_queue.cc:237] T 8f86d1517df74ead9e397ca9bed44c3f P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } }
I20240502 21:28:49.058676 1955 raft_consensus.cc:2454] T 8f86d1517df74ead9e397ca9bed44c3f P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 521bb93035324cecbebed2f6f2a630f8 in term 1.
I20240502 21:28:49.059624 2729 leader_election.cc:288] T 3c88daaac3734d918955a774829532c6 P a62a935ff43d494ca35d0c417943133d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e632c553ff4341328acf1cbaca678392 (127.15.255.133:37439), 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132:44947), 521bb93035324cecbebed2f6f2a630f8 (127.15.255.134:40821), 6ab8df3c796f4b6cb3e29d5045ad3ea0 (127.15.255.135:34701), 7d1e5c1b047443cbb28d7b0a9744babf (127.15.255.129:37897), 16b3a337bc5e462f83ba895ad8660deb (127.15.255.131:36033)
W20240502 21:28:49.078574 1461 tablet.cc:2369] T 13af1b96121c492c9373f9acbb55725a P 7d1e5c1b047443cbb28d7b0a9744babf: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240502 21:28:49.079154 2002 tablet.cc:2369] T 13af1b96121c492c9373f9acbb55725a P e632c553ff4341328acf1cbaca678392: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 21:28:49.136107 2227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f86d1517df74ead9e397ca9bed44c3f" candidate_uuid: "521bb93035324cecbebed2f6f2a630f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" is_pre_election: true
I20240502 21:28:49.137228 2226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f86d1517df74ead9e397ca9bed44c3f" candidate_uuid: "521bb93035324cecbebed2f6f2a630f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0"
W20240502 21:28:49.137888 2023 leader_election.cc:341] T 8f86d1517df74ead9e397ca9bed44c3f P 521bb93035324cecbebed2f6f2a630f8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6ab8df3c796f4b6cb3e29d5045ad3ea0 (127.15.255.135:34701): Illegal state: must be running to vote when last-logged opid is not known
W20240502 21:28:49.138617 2023 leader_election.cc:341] T 8f86d1517df74ead9e397ca9bed44c3f P 521bb93035324cecbebed2f6f2a630f8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 6ab8df3c796f4b6cb3e29d5045ad3ea0 (127.15.255.135:34701): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:28:49.147428 2716 raft_consensus.cc:482] T 13af1b96121c492c9373f9acbb55725a P 153026dd7f8943cc93fedf6f48450fb4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:28:49.147652 2716 raft_consensus.cc:504] T 13af1b96121c492c9373f9acbb55725a P 153026dd7f8943cc93fedf6f48450fb4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } }
I20240502 21:28:49.151336 2716 leader_election.cc:288] T 13af1b96121c492c9373f9acbb55725a P 153026dd7f8943cc93fedf6f48450fb4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7d1e5c1b047443cbb28d7b0a9744babf (127.15.255.129:37897), a62a935ff43d494ca35d0c417943133d (127.15.255.130:37991), 16b3a337bc5e462f83ba895ad8660deb (127.15.255.131:36033), 6ab8df3c796f4b6cb3e29d5045ad3ea0 (127.15.255.135:34701), e632c553ff4341328acf1cbaca678392 (127.15.255.133:37439), 521bb93035324cecbebed2f6f2a630f8 (127.15.255.134:40821)
W20240502 21:28:49.194794 2024 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.255.132:44947, user_credentials={real_user=slave}} blocked reactor thread for 146763us
W20240502 21:28:49.218467 2275 tablet.cc:2369] T 3c88daaac3734d918955a774829532c6 P 6ab8df3c796f4b6cb3e29d5045ad3ea0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 21:28:49.228293 2716 raft_consensus.cc:482] T 3c88daaac3734d918955a774829532c6 P 153026dd7f8943cc93fedf6f48450fb4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:28:49.228555 2716 raft_consensus.cc:504] T 3c88daaac3734d918955a774829532c6 P 153026dd7f8943cc93fedf6f48450fb4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:28:49.229804 2716 leader_election.cc:288] T 3c88daaac3734d918955a774829532c6 P 153026dd7f8943cc93fedf6f48450fb4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e632c553ff4341328acf1cbaca678392 (127.15.255.133:37439), 521bb93035324cecbebed2f6f2a630f8 (127.15.255.134:40821), 6ab8df3c796f4b6cb3e29d5045ad3ea0 (127.15.255.135:34701), 7d1e5c1b047443cbb28d7b0a9744babf (127.15.255.129:37897), a62a935ff43d494ca35d0c417943133d (127.15.255.130:37991), 16b3a337bc5e462f83ba895ad8660deb (127.15.255.131:36033)
I20240502 21:28:49.206750 2720 raft_consensus.cc:482] T 13af1b96121c492c9373f9acbb55725a P 521bb93035324cecbebed2f6f2a630f8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:28:49.231788 2720 raft_consensus.cc:504] T 13af1b96121c492c9373f9acbb55725a P 521bb93035324cecbebed2f6f2a630f8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } }
I20240502 21:28:49.233356 1955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c88daaac3734d918955a774829532c6" candidate_uuid: "a62a935ff43d494ca35d0c417943133d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e632c553ff4341328acf1cbaca678392" is_pre_election: true
I20240502 21:28:49.233597 1955 raft_consensus.cc:2454] T 3c88daaac3734d918955a774829532c6 P e632c553ff4341328acf1cbaca678392 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a62a935ff43d494ca35d0c417943133d in term 0.
I20240502 21:28:49.235278 1955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13af1b96121c492c9373f9acbb55725a" candidate_uuid: "521bb93035324cecbebed2f6f2a630f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e632c553ff4341328acf1cbaca678392" is_pre_election: true
I20240502 21:28:49.235320 1686 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13af1b96121c492c9373f9acbb55725a" candidate_uuid: "521bb93035324cecbebed2f6f2a630f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16b3a337bc5e462f83ba895ad8660deb" is_pre_election: true
I20240502 21:28:49.235482 1955 raft_consensus.cc:2454] T 13af1b96121c492c9373f9acbb55725a P e632c553ff4341328acf1cbaca678392 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 521bb93035324cecbebed2f6f2a630f8 in term 0.
I20240502 21:28:49.235546 1686 raft_consensus.cc:2454] T 13af1b96121c492c9373f9acbb55725a P 16b3a337bc5e462f83ba895ad8660deb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 521bb93035324cecbebed2f6f2a630f8 in term 0.
I20240502 21:28:49.253428 2089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13af1b96121c492c9373f9acbb55725a" candidate_uuid: "153026dd7f8943cc93fedf6f48450fb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "521bb93035324cecbebed2f6f2a630f8" is_pre_election: true
I20240502 21:28:49.253855 2089 raft_consensus.cc:2454] T 13af1b96121c492c9373f9acbb55725a P 521bb93035324cecbebed2f6f2a630f8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 153026dd7f8943cc93fedf6f48450fb4 in term 0.
I20240502 21:28:49.254304 2088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c88daaac3734d918955a774829532c6" candidate_uuid: "153026dd7f8943cc93fedf6f48450fb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "521bb93035324cecbebed2f6f2a630f8" is_pre_election: true
I20240502 21:28:49.254565 2088 raft_consensus.cc:2454] T 3c88daaac3734d918955a774829532c6 P 521bb93035324cecbebed2f6f2a630f8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 153026dd7f8943cc93fedf6f48450fb4 in term 0.
I20240502 21:28:49.265489 2720 leader_election.cc:288] T 13af1b96121c492c9373f9acbb55725a P 521bb93035324cecbebed2f6f2a630f8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7d1e5c1b047443cbb28d7b0a9744babf (127.15.255.129:37897), a62a935ff43d494ca35d0c417943133d (127.15.255.130:37991), 16b3a337bc5e462f83ba895ad8660deb (127.15.255.131:36033), 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132:44947), 6ab8df3c796f4b6cb3e29d5045ad3ea0 (127.15.255.135:34701), e632c553ff4341328acf1cbaca678392 (127.15.255.133:37439)
I20240502 21:28:49.267989 1955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13af1b96121c492c9373f9acbb55725a" candidate_uuid: "153026dd7f8943cc93fedf6f48450fb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e632c553ff4341328acf1cbaca678392" is_pre_election: true
I20240502 21:28:49.268210 1955 raft_consensus.cc:2454] T 13af1b96121c492c9373f9acbb55725a P e632c553ff4341328acf1cbaca678392 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 153026dd7f8943cc93fedf6f48450fb4 in term 0.
I20240502 21:28:49.268692 1955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c88daaac3734d918955a774829532c6" candidate_uuid: "153026dd7f8943cc93fedf6f48450fb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e632c553ff4341328acf1cbaca678392" is_pre_election: true
I20240502 21:28:49.268877 1955 raft_consensus.cc:2454] T 3c88daaac3734d918955a774829532c6 P e632c553ff4341328acf1cbaca678392 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 153026dd7f8943cc93fedf6f48450fb4 in term 0.
I20240502 21:28:49.280827 1820 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13af1b96121c492c9373f9acbb55725a" candidate_uuid: "521bb93035324cecbebed2f6f2a630f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "153026dd7f8943cc93fedf6f48450fb4" is_pre_election: true
I20240502 21:28:49.281098 1820 raft_consensus.cc:2454] T 13af1b96121c492c9373f9acbb55725a P 153026dd7f8943cc93fedf6f48450fb4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 521bb93035324cecbebed2f6f2a630f8 in term 0.
I20240502 21:28:49.283082 1270 catalog_manager.cc:5484] T 8f86d1517df74ead9e397ca9bed44c3f P 521bb93035324cecbebed2f6f2a630f8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 521bb93035324cecbebed2f6f2a630f8 (127.15.255.134). New cstate: current_term: 1 leader_uuid: "521bb93035324cecbebed2f6f2a630f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } health_report { overall_health: UNKNOWN } } }
I20240502 21:28:49.299129 1415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13af1b96121c492c9373f9acbb55725a" candidate_uuid: "521bb93035324cecbebed2f6f2a630f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" is_pre_election: true
I20240502 21:28:49.299531 1415 raft_consensus.cc:2454] T 13af1b96121c492c9373f9acbb55725a P 7d1e5c1b047443cbb28d7b0a9744babf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 521bb93035324cecbebed2f6f2a630f8 in term 0.
I20240502 21:28:49.300681 2226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13af1b96121c492c9373f9acbb55725a" candidate_uuid: "521bb93035324cecbebed2f6f2a630f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" is_pre_election: true
I20240502 21:28:49.302678 1415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c88daaac3734d918955a774829532c6" candidate_uuid: "a62a935ff43d494ca35d0c417943133d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" is_pre_election: true
I20240502 21:28:49.302417 2024 leader_election.cc:302] T 13af1b96121c492c9373f9acbb55725a P 521bb93035324cecbebed2f6f2a630f8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 153026dd7f8943cc93fedf6f48450fb4, 16b3a337bc5e462f83ba895ad8660deb, 521bb93035324cecbebed2f6f2a630f8, e632c553ff4341328acf1cbaca678392; no voters:
I20240502 21:28:49.318835 2769 raft_consensus.cc:2781] T 13af1b96121c492c9373f9acbb55725a P 521bb93035324cecbebed2f6f2a630f8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:28:49.319000 2769 raft_consensus.cc:482] T 13af1b96121c492c9373f9acbb55725a P 521bb93035324cecbebed2f6f2a630f8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:28:49.319208 2769 raft_consensus.cc:3037] T 13af1b96121c492c9373f9acbb55725a P 521bb93035324cecbebed2f6f2a630f8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:49.321820 2088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c88daaac3734d918955a774829532c6" candidate_uuid: "a62a935ff43d494ca35d0c417943133d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "521bb93035324cecbebed2f6f2a630f8" is_pre_election: true
I20240502 21:28:49.322044 2088 raft_consensus.cc:2454] T 3c88daaac3734d918955a774829532c6 P 521bb93035324cecbebed2f6f2a630f8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a62a935ff43d494ca35d0c417943133d in term 0.
W20240502 21:28:49.322417 2023 leader_election.cc:341] T 13af1b96121c492c9373f9acbb55725a P 521bb93035324cecbebed2f6f2a630f8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6ab8df3c796f4b6cb3e29d5045ad3ea0 (127.15.255.135:34701): Illegal state: must be running to vote when last-logged opid is not known
W20240502 21:28:49.323258 1481 leader_election.cc:341] T 3c88daaac3734d918955a774829532c6 P a62a935ff43d494ca35d0c417943133d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7d1e5c1b047443cbb28d7b0a9744babf (127.15.255.129:37897): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:28:49.324762 1415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13af1b96121c492c9373f9acbb55725a" candidate_uuid: "153026dd7f8943cc93fedf6f48450fb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" is_pre_election: true
I20240502 21:28:49.325023 1415 raft_consensus.cc:2454] T 13af1b96121c492c9373f9acbb55725a P 7d1e5c1b047443cbb28d7b0a9744babf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 153026dd7f8943cc93fedf6f48450fb4 in term 0.
I20240502 21:28:49.325759 1415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c88daaac3734d918955a774829532c6" candidate_uuid: "153026dd7f8943cc93fedf6f48450fb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" is_pre_election: true
I20240502 21:28:49.326503 1752 leader_election.cc:302] T 13af1b96121c492c9373f9acbb55725a P 153026dd7f8943cc93fedf6f48450fb4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 153026dd7f8943cc93fedf6f48450fb4, 521bb93035324cecbebed2f6f2a630f8, 7d1e5c1b047443cbb28d7b0a9744babf, e632c553ff4341328acf1cbaca678392; no voters:
I20240502 21:28:49.338526 1686 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13af1b96121c492c9373f9acbb55725a" candidate_uuid: "153026dd7f8943cc93fedf6f48450fb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16b3a337bc5e462f83ba895ad8660deb" is_pre_election: true
I20240502 21:28:49.338810 1686 raft_consensus.cc:2454] T 13af1b96121c492c9373f9acbb55725a P 16b3a337bc5e462f83ba895ad8660deb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 153026dd7f8943cc93fedf6f48450fb4 in term 0.
I20240502 21:28:49.339583 1686 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c88daaac3734d918955a774829532c6" candidate_uuid: "153026dd7f8943cc93fedf6f48450fb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16b3a337bc5e462f83ba895ad8660deb" is_pre_election: true
I20240502 21:28:49.339799 1686 raft_consensus.cc:2454] T 3c88daaac3734d918955a774829532c6 P 16b3a337bc5e462f83ba895ad8660deb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 153026dd7f8943cc93fedf6f48450fb4 in term 0.
I20240502 21:28:49.344349 2226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c88daaac3734d918955a774829532c6" candidate_uuid: "a62a935ff43d494ca35d0c417943133d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" is_pre_election: true
I20240502 21:28:49.344771 2226 raft_consensus.cc:2454] T 3c88daaac3734d918955a774829532c6 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a62a935ff43d494ca35d0c417943133d in term 0.
I20240502 21:28:49.346083 1481 leader_election.cc:302] T 3c88daaac3734d918955a774829532c6 P a62a935ff43d494ca35d0c417943133d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 5 responses out of 7 voters: 4 yes votes; 1 no votes. yes voters: 521bb93035324cecbebed2f6f2a630f8, 6ab8df3c796f4b6cb3e29d5045ad3ea0, a62a935ff43d494ca35d0c417943133d, e632c553ff4341328acf1cbaca678392; no voters: 7d1e5c1b047443cbb28d7b0a9744babf
I20240502 21:28:49.348543 2729 raft_consensus.cc:2781] T 3c88daaac3734d918955a774829532c6 P a62a935ff43d494ca35d0c417943133d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:28:49.348664 2729 raft_consensus.cc:482] T 3c88daaac3734d918955a774829532c6 P a62a935ff43d494ca35d0c417943133d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:28:49.348738 2729 raft_consensus.cc:3037] T 3c88daaac3734d918955a774829532c6 P a62a935ff43d494ca35d0c417943133d [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:49.351668 2729 raft_consensus.cc:504] T 3c88daaac3734d918955a774829532c6 P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:28:49.353080 1955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c88daaac3734d918955a774829532c6" candidate_uuid: "a62a935ff43d494ca35d0c417943133d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e632c553ff4341328acf1cbaca678392"
I20240502 21:28:49.353294 1955 raft_consensus.cc:3037] T 3c88daaac3734d918955a774829532c6 P e632c553ff4341328acf1cbaca678392 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:49.356091 1955 raft_consensus.cc:2454] T 3c88daaac3734d918955a774829532c6 P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a62a935ff43d494ca35d0c417943133d in term 1.
I20240502 21:28:49.356935 2729 leader_election.cc:288] T 3c88daaac3734d918955a774829532c6 P a62a935ff43d494ca35d0c417943133d [CANDIDATE]: Term 1 election: Requested vote from peers e632c553ff4341328acf1cbaca678392 (127.15.255.133:37439), 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132:44947), 521bb93035324cecbebed2f6f2a630f8 (127.15.255.134:40821), 6ab8df3c796f4b6cb3e29d5045ad3ea0 (127.15.255.135:34701), 7d1e5c1b047443cbb28d7b0a9744babf (127.15.255.129:37897), 16b3a337bc5e462f83ba895ad8660deb (127.15.255.131:36033)
I20240502 21:28:49.357069 2088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c88daaac3734d918955a774829532c6" candidate_uuid: "a62a935ff43d494ca35d0c417943133d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "521bb93035324cecbebed2f6f2a630f8"
I20240502 21:28:49.357293 2088 raft_consensus.cc:3037] T 3c88daaac3734d918955a774829532c6 P 521bb93035324cecbebed2f6f2a630f8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:49.360311 2088 raft_consensus.cc:2454] T 3c88daaac3734d918955a774829532c6 P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a62a935ff43d494ca35d0c417943133d in term 1.
I20240502 21:28:49.361672 2730 tablet_bootstrap.cc:492] T 8f86d1517df74ead9e397ca9bed44c3f P 6ab8df3c796f4b6cb3e29d5045ad3ea0: No bootstrap required, opened a new log
I20240502 21:28:49.361964 2730 ts_tablet_manager.cc:1367] T 8f86d1517df74ead9e397ca9bed44c3f P 6ab8df3c796f4b6cb3e29d5045ad3ea0: Time spent bootstrapping tablet: real 0.348s user 0.095s sys 0.145s
I20240502 21:28:49.361902 2226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c88daaac3734d918955a774829532c6" candidate_uuid: "a62a935ff43d494ca35d0c417943133d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0"
I20240502 21:28:49.362185 2226 raft_consensus.cc:3037] T 3c88daaac3734d918955a774829532c6 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:49.363575 1415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c88daaac3734d918955a774829532c6" candidate_uuid: "a62a935ff43d494ca35d0c417943133d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d1e5c1b047443cbb28d7b0a9744babf"
W20240502 21:28:49.364197 1481 leader_election.cc:341] T 3c88daaac3734d918955a774829532c6 P a62a935ff43d494ca35d0c417943133d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7d1e5c1b047443cbb28d7b0a9744babf (127.15.255.129:37897): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:28:49.365643 2226 raft_consensus.cc:2454] T 3c88daaac3734d918955a774829532c6 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a62a935ff43d494ca35d0c417943133d in term 1.
I20240502 21:28:49.367333 1754 leader_election.cc:302] T 3c88daaac3734d918955a774829532c6 P 153026dd7f8943cc93fedf6f48450fb4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 153026dd7f8943cc93fedf6f48450fb4, 16b3a337bc5e462f83ba895ad8660deb, 521bb93035324cecbebed2f6f2a630f8, e632c553ff4341328acf1cbaca678392; no voters:
I20240502 21:28:49.367713 2716 raft_consensus.cc:2781] T 3c88daaac3734d918955a774829532c6 P 153026dd7f8943cc93fedf6f48450fb4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:28:49.367827 2716 raft_consensus.cc:482] T 3c88daaac3734d918955a774829532c6 P 153026dd7f8943cc93fedf6f48450fb4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:28:49.367938 2716 raft_consensus.cc:3037] T 3c88daaac3734d918955a774829532c6 P 153026dd7f8943cc93fedf6f48450fb4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:49.368857 2730 raft_consensus.cc:348] T 8f86d1517df74ead9e397ca9bed44c3f P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } }
I20240502 21:28:49.369381 1481 leader_election.cc:302] T 3c88daaac3734d918955a774829532c6 P a62a935ff43d494ca35d0c417943133d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 5 responses out of 7 voters: 4 yes votes; 1 no votes. yes voters: 521bb93035324cecbebed2f6f2a630f8, 6ab8df3c796f4b6cb3e29d5045ad3ea0, a62a935ff43d494ca35d0c417943133d, e632c553ff4341328acf1cbaca678392; no voters: 7d1e5c1b047443cbb28d7b0a9744babf
I20240502 21:28:49.369478 2722 tablet_bootstrap.cc:492] T 13af1b96121c492c9373f9acbb55725a P a62a935ff43d494ca35d0c417943133d: No bootstrap required, opened a new log
I20240502 21:28:49.369711 2722 ts_tablet_manager.cc:1367] T 13af1b96121c492c9373f9acbb55725a P a62a935ff43d494ca35d0c417943133d: Time spent bootstrapping tablet: real 0.359s user 0.062s sys 0.133s
I20240502 21:28:49.369755 2730 raft_consensus.cc:374] T 8f86d1517df74ead9e397ca9bed44c3f P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:49.369933 2730 raft_consensus.cc:729] T 8f86d1517df74ead9e397ca9bed44c3f P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ab8df3c796f4b6cb3e29d5045ad3ea0, State: Initialized, Role: FOLLOWER
I20240502 21:28:49.370297 2730 consensus_queue.cc:260] T 8f86d1517df74ead9e397ca9bed44c3f P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } }
I20240502 21:28:49.370488 2722 raft_consensus.cc:348] T 13af1b96121c492c9373f9acbb55725a P a62a935ff43d494ca35d0c417943133d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } }
I20240502 21:28:49.370589 2716 raft_consensus.cc:504] T 3c88daaac3734d918955a774829532c6 P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:28:49.370800 2722 raft_consensus.cc:374] T 13af1b96121c492c9373f9acbb55725a P a62a935ff43d494ca35d0c417943133d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:49.370888 2722 raft_consensus.cc:729] T 13af1b96121c492c9373f9acbb55725a P a62a935ff43d494ca35d0c417943133d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a62a935ff43d494ca35d0c417943133d, State: Initialized, Role: FOLLOWER
I20240502 21:28:49.371270 2730 ts_tablet_manager.cc:1398] T 8f86d1517df74ead9e397ca9bed44c3f P 6ab8df3c796f4b6cb3e29d5045ad3ea0: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 21:28:49.371145 2722 consensus_queue.cc:260] T 13af1b96121c492c9373f9acbb55725a P a62a935ff43d494ca35d0c417943133d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } }
I20240502 21:28:49.371621 2729 raft_consensus.cc:2781] T 3c88daaac3734d918955a774829532c6 P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:28:49.372365 2716 leader_election.cc:288] T 3c88daaac3734d918955a774829532c6 P 153026dd7f8943cc93fedf6f48450fb4 [CANDIDATE]: Term 1 election: Requested vote from peers e632c553ff4341328acf1cbaca678392 (127.15.255.133:37439), 521bb93035324cecbebed2f6f2a630f8 (127.15.255.134:40821), 6ab8df3c796f4b6cb3e29d5045ad3ea0 (127.15.255.135:34701), 7d1e5c1b047443cbb28d7b0a9744babf (127.15.255.129:37897), a62a935ff43d494ca35d0c417943133d (127.15.255.130:37991), 16b3a337bc5e462f83ba895ad8660deb (127.15.255.131:36033)
I20240502 21:28:49.372409 2088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c88daaac3734d918955a774829532c6" candidate_uuid: "153026dd7f8943cc93fedf6f48450fb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "521bb93035324cecbebed2f6f2a630f8"
I20240502 21:28:49.372516 2730 tablet_bootstrap.cc:492] T 13af1b96121c492c9373f9acbb55725a P 6ab8df3c796f4b6cb3e29d5045ad3ea0: Bootstrap starting.
I20240502 21:28:49.372563 2716 raft_consensus.cc:2781] T 13af1b96121c492c9373f9acbb55725a P 153026dd7f8943cc93fedf6f48450fb4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:28:49.372663 2716 raft_consensus.cc:482] T 13af1b96121c492c9373f9acbb55725a P 153026dd7f8943cc93fedf6f48450fb4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:28:49.372751 2088 raft_consensus.cc:2379] T 3c88daaac3734d918955a774829532c6 P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 153026dd7f8943cc93fedf6f48450fb4 in current term 1: Already voted for candidate a62a935ff43d494ca35d0c417943133d in this term.
I20240502 21:28:49.372750 2716 raft_consensus.cc:3037] T 13af1b96121c492c9373f9acbb55725a P 153026dd7f8943cc93fedf6f48450fb4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:49.374320 2730 tablet_bootstrap.cc:654] T 13af1b96121c492c9373f9acbb55725a P 6ab8df3c796f4b6cb3e29d5045ad3ea0: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:49.375202 2716 raft_consensus.cc:504] T 13af1b96121c492c9373f9acbb55725a P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } }
I20240502 21:28:49.376183 2716 leader_election.cc:288] T 13af1b96121c492c9373f9acbb55725a P 153026dd7f8943cc93fedf6f48450fb4 [CANDIDATE]: Term 1 election: Requested vote from peers 7d1e5c1b047443cbb28d7b0a9744babf (127.15.255.129:37897), a62a935ff43d494ca35d0c417943133d (127.15.255.130:37991), 16b3a337bc5e462f83ba895ad8660deb (127.15.255.131:36033), 6ab8df3c796f4b6cb3e29d5045ad3ea0 (127.15.255.135:34701), e632c553ff4341328acf1cbaca678392 (127.15.255.133:37439), 521bb93035324cecbebed2f6f2a630f8 (127.15.255.134:40821)
I20240502 21:28:49.376812 1686 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c88daaac3734d918955a774829532c6" candidate_uuid: "153026dd7f8943cc93fedf6f48450fb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16b3a337bc5e462f83ba895ad8660deb"
I20240502 21:28:49.377041 1686 raft_consensus.cc:3037] T 3c88daaac3734d918955a774829532c6 P 16b3a337bc5e462f83ba895ad8660deb [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:49.378173 2088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13af1b96121c492c9373f9acbb55725a" candidate_uuid: "153026dd7f8943cc93fedf6f48450fb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "521bb93035324cecbebed2f6f2a630f8"
I20240502 21:28:49.379855 1686 raft_consensus.cc:2454] T 3c88daaac3734d918955a774829532c6 P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 153026dd7f8943cc93fedf6f48450fb4 in term 1.
I20240502 21:28:49.380649 1686 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13af1b96121c492c9373f9acbb55725a" candidate_uuid: "153026dd7f8943cc93fedf6f48450fb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16b3a337bc5e462f83ba895ad8660deb"
I20240502 21:28:49.380821 1686 raft_consensus.cc:3037] T 13af1b96121c492c9373f9acbb55725a P 16b3a337bc5e462f83ba895ad8660deb [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:49.383517 1686 raft_consensus.cc:2454] T 13af1b96121c492c9373f9acbb55725a P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 153026dd7f8943cc93fedf6f48450fb4 in term 1.
I20240502 21:28:49.384392 1955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c88daaac3734d918955a774829532c6" candidate_uuid: "153026dd7f8943cc93fedf6f48450fb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e632c553ff4341328acf1cbaca678392"
I20240502 21:28:49.384744 1955 raft_consensus.cc:2379] T 3c88daaac3734d918955a774829532c6 P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 153026dd7f8943cc93fedf6f48450fb4 in current term 1: Already voted for candidate a62a935ff43d494ca35d0c417943133d in this term.
I20240502 21:28:49.385320 1955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13af1b96121c492c9373f9acbb55725a" candidate_uuid: "153026dd7f8943cc93fedf6f48450fb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e632c553ff4341328acf1cbaca678392"
I20240502 21:28:49.385515 1955 raft_consensus.cc:3037] T 13af1b96121c492c9373f9acbb55725a P e632c553ff4341328acf1cbaca678392 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:49.387187 2730 tablet_bootstrap.cc:492] T 13af1b96121c492c9373f9acbb55725a P 6ab8df3c796f4b6cb3e29d5045ad3ea0: No bootstrap required, opened a new log
I20240502 21:28:49.387396 2730 ts_tablet_manager.cc:1367] T 13af1b96121c492c9373f9acbb55725a P 6ab8df3c796f4b6cb3e29d5045ad3ea0: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.001s
I20240502 21:28:49.388072 1955 raft_consensus.cc:2454] T 13af1b96121c492c9373f9acbb55725a P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 153026dd7f8943cc93fedf6f48450fb4 in term 1.
I20240502 21:28:49.388101 2730 raft_consensus.cc:348] T 13af1b96121c492c9373f9acbb55725a P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } }
I20240502 21:28:49.388497 2730 raft_consensus.cc:374] T 13af1b96121c492c9373f9acbb55725a P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:49.388628 2730 raft_consensus.cc:729] T 13af1b96121c492c9373f9acbb55725a P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ab8df3c796f4b6cb3e29d5045ad3ea0, State: Initialized, Role: FOLLOWER
I20240502 21:28:49.388960 2730 consensus_queue.cc:260] T 13af1b96121c492c9373f9acbb55725a P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } }
I20240502 21:28:49.390877 2730 ts_tablet_manager.cc:1398] T 13af1b96121c492c9373f9acbb55725a P 6ab8df3c796f4b6cb3e29d5045ad3ea0: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240502 21:28:49.397246 2745 raft_consensus.cc:482] T 8f86d1517df74ead9e397ca9bed44c3f P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:28:49.397424 2745 raft_consensus.cc:504] T 8f86d1517df74ead9e397ca9bed44c3f P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } }
I20240502 21:28:49.398588 1686 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c88daaac3734d918955a774829532c6" candidate_uuid: "a62a935ff43d494ca35d0c417943133d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16b3a337bc5e462f83ba895ad8660deb" is_pre_election: true
I20240502 21:28:49.398864 1686 raft_consensus.cc:2379] T 3c88daaac3734d918955a774829532c6 P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a62a935ff43d494ca35d0c417943133d in current term 1: Already voted for candidate 153026dd7f8943cc93fedf6f48450fb4 in this term.
I20240502 21:28:49.400148 2745 leader_election.cc:288] T 8f86d1517df74ead9e397ca9bed44c3f P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7d1e5c1b047443cbb28d7b0a9744babf (127.15.255.129:37897), a62a935ff43d494ca35d0c417943133d (127.15.255.130:37991), 16b3a337bc5e462f83ba895ad8660deb (127.15.255.131:36033), 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132:44947), e632c553ff4341328acf1cbaca678392 (127.15.255.133:37439), 521bb93035324cecbebed2f6f2a630f8 (127.15.255.134:40821)
I20240502 21:28:49.402698 2729 raft_consensus.cc:686] T 3c88daaac3734d918955a774829532c6 P a62a935ff43d494ca35d0c417943133d [term 1 LEADER]: Becoming Leader. State: Replica: a62a935ff43d494ca35d0c417943133d, State: Running, Role: LEADER
I20240502 21:28:49.403292 2729 consensus_queue.cc:237] T 3c88daaac3734d918955a774829532c6 P a62a935ff43d494ca35d0c417943133d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:28:49.406713 1550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13af1b96121c492c9373f9acbb55725a" candidate_uuid: "521bb93035324cecbebed2f6f2a630f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a62a935ff43d494ca35d0c417943133d" is_pre_election: true
I20240502 21:28:49.406927 1550 raft_consensus.cc:2454] T 13af1b96121c492c9373f9acbb55725a P a62a935ff43d494ca35d0c417943133d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 521bb93035324cecbebed2f6f2a630f8 in term 0.
I20240502 21:28:49.407886 1686 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c88daaac3734d918955a774829532c6" candidate_uuid: "a62a935ff43d494ca35d0c417943133d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16b3a337bc5e462f83ba895ad8660deb"
I20240502 21:28:49.408145 1686 raft_consensus.cc:2379] T 3c88daaac3734d918955a774829532c6 P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate a62a935ff43d494ca35d0c417943133d in current term 1: Already voted for candidate 153026dd7f8943cc93fedf6f48450fb4 in this term.
I20240502 21:28:49.408547 2722 ts_tablet_manager.cc:1398] T 13af1b96121c492c9373f9acbb55725a P a62a935ff43d494ca35d0c417943133d: Time spent starting tablet: real 0.039s user 0.004s sys 0.008s
I20240502 21:28:49.408846 1955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c88daaac3734d918955a774829532c6" candidate_uuid: "16b3a337bc5e462f83ba895ad8660deb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e632c553ff4341328acf1cbaca678392" is_pre_election: true
I20240502 21:28:49.409065 1955 raft_consensus.cc:2379] T 3c88daaac3734d918955a774829532c6 P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 16b3a337bc5e462f83ba895ad8660deb in current term 1: Already voted for candidate a62a935ff43d494ca35d0c417943133d in this term.
I20240502 21:28:49.413009 1415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f86d1517df74ead9e397ca9bed44c3f" candidate_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" is_pre_election: true
W20240502 21:28:49.413805 2159 leader_election.cc:341] T 8f86d1517df74ead9e397ca9bed44c3f P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7d1e5c1b047443cbb28d7b0a9744babf (127.15.255.129:37897): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:28:49.426000 1268 catalog_manager.cc:5484] T 3c88daaac3734d918955a774829532c6 P a62a935ff43d494ca35d0c417943133d reported cstate change: term changed from 0 to 1, leader changed from <none> to a62a935ff43d494ca35d0c417943133d (127.15.255.130). New cstate: current_term: 1 leader_uuid: "a62a935ff43d494ca35d0c417943133d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } health_report { overall_health: UNKNOWN } } }
I20240502 21:28:49.429221 2769 raft_consensus.cc:504] T 13af1b96121c492c9373f9acbb55725a P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } }
I20240502 21:28:49.429906 2088 raft_consensus.cc:2379] T 13af1b96121c492c9373f9acbb55725a P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 153026dd7f8943cc93fedf6f48450fb4 in current term 1: Already voted for candidate 521bb93035324cecbebed2f6f2a630f8 in this term.
W20240502 21:28:49.432866 1752 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.15.255.129:37897, user_credentials={real_user=slave}} blocked reactor thread for 106460us
W20240502 21:28:49.433396 1752 leader_election.cc:341] T 3c88daaac3734d918955a774829532c6 P 153026dd7f8943cc93fedf6f48450fb4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7d1e5c1b047443cbb28d7b0a9744babf (127.15.255.129:37897): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:28:49.435137 1820 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c88daaac3734d918955a774829532c6" candidate_uuid: "a62a935ff43d494ca35d0c417943133d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "153026dd7f8943cc93fedf6f48450fb4" is_pre_election: true
I20240502 21:28:49.435452 1820 raft_consensus.cc:2379] T 3c88daaac3734d918955a774829532c6 P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a62a935ff43d494ca35d0c417943133d in current term 1: Already voted for candidate 153026dd7f8943cc93fedf6f48450fb4 in this term.
I20240502 21:28:49.436367 1550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13af1b96121c492c9373f9acbb55725a" candidate_uuid: "153026dd7f8943cc93fedf6f48450fb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a62a935ff43d494ca35d0c417943133d" is_pre_election: true
I20240502 21:28:49.436630 1550 raft_consensus.cc:2454] T 13af1b96121c492c9373f9acbb55725a P a62a935ff43d494ca35d0c417943133d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 153026dd7f8943cc93fedf6f48450fb4 in term 0.
I20240502 21:28:49.437203 1550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c88daaac3734d918955a774829532c6" candidate_uuid: "153026dd7f8943cc93fedf6f48450fb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a62a935ff43d494ca35d0c417943133d" is_pre_election: true
I20240502 21:28:49.437949 1550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c88daaac3734d918955a774829532c6" candidate_uuid: "153026dd7f8943cc93fedf6f48450fb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a62a935ff43d494ca35d0c417943133d"
I20240502 21:28:49.438515 1550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13af1b96121c492c9373f9acbb55725a" candidate_uuid: "153026dd7f8943cc93fedf6f48450fb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a62a935ff43d494ca35d0c417943133d"
I20240502 21:28:49.438699 1550 raft_consensus.cc:3037] T 13af1b96121c492c9373f9acbb55725a P a62a935ff43d494ca35d0c417943133d [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:49.441567 1550 raft_consensus.cc:2454] T 13af1b96121c492c9373f9acbb55725a P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 153026dd7f8943cc93fedf6f48450fb4 in term 1.
I20240502 21:28:49.445830 1686 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13af1b96121c492c9373f9acbb55725a" candidate_uuid: "521bb93035324cecbebed2f6f2a630f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16b3a337bc5e462f83ba895ad8660deb"
I20240502 21:28:49.446085 1686 raft_consensus.cc:2379] T 13af1b96121c492c9373f9acbb55725a P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 521bb93035324cecbebed2f6f2a630f8 in current term 1: Already voted for candidate 153026dd7f8943cc93fedf6f48450fb4 in this term.
I20240502 21:28:49.446501 1820 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c88daaac3734d918955a774829532c6" candidate_uuid: "a62a935ff43d494ca35d0c417943133d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "153026dd7f8943cc93fedf6f48450fb4"
I20240502 21:28:49.446749 1820 raft_consensus.cc:2379] T 3c88daaac3734d918955a774829532c6 P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate a62a935ff43d494ca35d0c417943133d in current term 1: Already voted for candidate 153026dd7f8943cc93fedf6f48450fb4 in this term.
I20240502 21:28:49.446831 1752 leader_election.cc:302] T 13af1b96121c492c9373f9acbb55725a P 153026dd7f8943cc93fedf6f48450fb4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 5 responses out of 7 voters: 4 yes votes; 1 no votes. yes voters: 153026dd7f8943cc93fedf6f48450fb4, 16b3a337bc5e462f83ba895ad8660deb, a62a935ff43d494ca35d0c417943133d, e632c553ff4341328acf1cbaca678392; no voters: 521bb93035324cecbebed2f6f2a630f8
I20240502 21:28:49.447070 1819 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c88daaac3734d918955a774829532c6" candidate_uuid: "16b3a337bc5e462f83ba895ad8660deb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "153026dd7f8943cc93fedf6f48450fb4" is_pre_election: true
I20240502 21:28:49.447279 1819 raft_consensus.cc:2379] T 3c88daaac3734d918955a774829532c6 P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 16b3a337bc5e462f83ba895ad8660deb in current term 1: Already voted for candidate 153026dd7f8943cc93fedf6f48450fb4 in this term.
I20240502 21:28:49.447589 1415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c88daaac3734d918955a774829532c6" candidate_uuid: "153026dd7f8943cc93fedf6f48450fb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d1e5c1b047443cbb28d7b0a9744babf"
I20240502 21:28:49.449805 1550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c88daaac3734d918955a774829532c6" candidate_uuid: "16b3a337bc5e462f83ba895ad8660deb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a62a935ff43d494ca35d0c417943133d" is_pre_election: true
I20240502 21:28:49.456215 2088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c88daaac3734d918955a774829532c6" candidate_uuid: "16b3a337bc5e462f83ba895ad8660deb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "521bb93035324cecbebed2f6f2a630f8" is_pre_election: true
I20240502 21:28:49.456507 2088 raft_consensus.cc:2379] T 3c88daaac3734d918955a774829532c6 P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 16b3a337bc5e462f83ba895ad8660deb in current term 1: Already voted for candidate a62a935ff43d494ca35d0c417943133d in this term.
I20240502 21:28:49.467165 1955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f86d1517df74ead9e397ca9bed44c3f" candidate_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e632c553ff4341328acf1cbaca678392" is_pre_election: true
I20240502 21:28:49.467463 1955 raft_consensus.cc:2379] T 8f86d1517df74ead9e397ca9bed44c3f P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6ab8df3c796f4b6cb3e29d5045ad3ea0 in current term 1: Already voted for candidate 521bb93035324cecbebed2f6f2a630f8 in this term.
I20240502 21:28:49.468192 1686 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f86d1517df74ead9e397ca9bed44c3f" candidate_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16b3a337bc5e462f83ba895ad8660deb" is_pre_election: true
I20240502 21:28:49.468487 1686 raft_consensus.cc:2379] T 8f86d1517df74ead9e397ca9bed44c3f P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6ab8df3c796f4b6cb3e29d5045ad3ea0 in current term 1: Already voted for candidate 521bb93035324cecbebed2f6f2a630f8 in this term.
I20240502 21:28:49.468657 2226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13af1b96121c492c9373f9acbb55725a" candidate_uuid: "153026dd7f8943cc93fedf6f48450fb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" is_pre_election: true
I20240502 21:28:49.468909 2226 raft_consensus.cc:2454] T 13af1b96121c492c9373f9acbb55725a P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 153026dd7f8943cc93fedf6f48450fb4 in term 0.
I20240502 21:28:49.469347 2226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c88daaac3734d918955a774829532c6" candidate_uuid: "153026dd7f8943cc93fedf6f48450fb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" is_pre_election: true
I20240502 21:28:49.469628 2226 raft_consensus.cc:2379] T 3c88daaac3734d918955a774829532c6 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 153026dd7f8943cc93fedf6f48450fb4 in current term 1: Already voted for candidate a62a935ff43d494ca35d0c417943133d in this term.
I20240502 21:28:49.471203 2226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c88daaac3734d918955a774829532c6" candidate_uuid: "16b3a337bc5e462f83ba895ad8660deb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" is_pre_election: true
I20240502 21:28:49.471450 2226 raft_consensus.cc:2379] T 3c88daaac3734d918955a774829532c6 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 16b3a337bc5e462f83ba895ad8660deb in current term 1: Already voted for candidate a62a935ff43d494ca35d0c417943133d in this term.
I20240502 21:28:49.474702 2226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c88daaac3734d918955a774829532c6" candidate_uuid: "153026dd7f8943cc93fedf6f48450fb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0"
I20240502 21:28:49.474931 2226 raft_consensus.cc:2379] T 3c88daaac3734d918955a774829532c6 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 153026dd7f8943cc93fedf6f48450fb4 in current term 1: Already voted for candidate a62a935ff43d494ca35d0c417943133d in this term.
I20240502 21:28:49.475342 2226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13af1b96121c492c9373f9acbb55725a" candidate_uuid: "153026dd7f8943cc93fedf6f48450fb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0"
I20240502 21:28:49.475523 2226 raft_consensus.cc:3037] T 13af1b96121c492c9373f9acbb55725a P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:49.478622 2226 raft_consensus.cc:2454] T 13af1b96121c492c9373f9acbb55725a P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 153026dd7f8943cc93fedf6f48450fb4 in term 1.
I20240502 21:28:49.480269 1819 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f86d1517df74ead9e397ca9bed44c3f" candidate_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "153026dd7f8943cc93fedf6f48450fb4" is_pre_election: true
I20240502 21:28:49.480533 1819 raft_consensus.cc:2379] T 8f86d1517df74ead9e397ca9bed44c3f P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6ab8df3c796f4b6cb3e29d5045ad3ea0 in current term 1: Already voted for candidate 521bb93035324cecbebed2f6f2a630f8 in this term.
I20240502 21:28:49.480716 1617 leader_election.cc:302] T 3c88daaac3734d918955a774829532c6 P 16b3a337bc5e462f83ba895ad8660deb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 5 responses out of 7 voters: 1 yes votes; 4 no votes. yes voters: 16b3a337bc5e462f83ba895ad8660deb; no voters: 521bb93035324cecbebed2f6f2a630f8, 6ab8df3c796f4b6cb3e29d5045ad3ea0, a62a935ff43d494ca35d0c417943133d, e632c553ff4341328acf1cbaca678392
I20240502 21:28:49.481293 2721 raft_consensus.cc:2726] T 3c88daaac3734d918955a774829532c6 P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 21:28:49.482285 1550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13af1b96121c492c9373f9acbb55725a" candidate_uuid: "521bb93035324cecbebed2f6f2a630f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a62a935ff43d494ca35d0c417943133d"
I20240502 21:28:49.482371 2781 raft_consensus.cc:2781] T 13af1b96121c492c9373f9acbb55725a P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:28:49.482538 2781 raft_consensus.cc:686] T 13af1b96121c492c9373f9acbb55725a P 153026dd7f8943cc93fedf6f48450fb4 [term 1 LEADER]: Becoming Leader. State: Replica: 153026dd7f8943cc93fedf6f48450fb4, State: Running, Role: LEADER
I20240502 21:28:49.482617 1550 raft_consensus.cc:2379] T 13af1b96121c492c9373f9acbb55725a P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 521bb93035324cecbebed2f6f2a630f8 in current term 1: Already voted for candidate 153026dd7f8943cc93fedf6f48450fb4 in this term.
I20240502 21:28:49.482479 1549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f86d1517df74ead9e397ca9bed44c3f" candidate_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a62a935ff43d494ca35d0c417943133d" is_pre_election: true
I20240502 21:28:49.482848 1549 raft_consensus.cc:2379] T 8f86d1517df74ead9e397ca9bed44c3f P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6ab8df3c796f4b6cb3e29d5045ad3ea0 in current term 1: Already voted for candidate 521bb93035324cecbebed2f6f2a630f8 in this term.
I20240502 21:28:49.483150 2781 consensus_queue.cc:237] T 13af1b96121c492c9373f9acbb55725a P 153026dd7f8943cc93fedf6f48450fb4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } }
I20240502 21:28:49.483659 2159 leader_election.cc:302] T 8f86d1517df74ead9e397ca9bed44c3f P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 5 responses out of 7 voters: 1 yes votes; 4 no votes. yes voters: 6ab8df3c796f4b6cb3e29d5045ad3ea0; no voters: 153026dd7f8943cc93fedf6f48450fb4, 16b3a337bc5e462f83ba895ad8660deb, 7d1e5c1b047443cbb28d7b0a9744babf, a62a935ff43d494ca35d0c417943133d
I20240502 21:28:49.484091 2745 raft_consensus.cc:3037] T 8f86d1517df74ead9e397ca9bed44c3f P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:49.486783 2745 raft_consensus.cc:2726] T 8f86d1517df74ead9e397ca9bed44c3f P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 21:28:49.487058 2769 leader_election.cc:288] T 13af1b96121c492c9373f9acbb55725a P 521bb93035324cecbebed2f6f2a630f8 [CANDIDATE]: Term 1 election: Requested vote from peers 7d1e5c1b047443cbb28d7b0a9744babf (127.15.255.129:37897), a62a935ff43d494ca35d0c417943133d (127.15.255.130:37991), 16b3a337bc5e462f83ba895ad8660deb (127.15.255.131:36033), 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132:44947), 6ab8df3c796f4b6cb3e29d5045ad3ea0 (127.15.255.135:34701), e632c553ff4341328acf1cbaca678392 (127.15.255.133:37439)
I20240502 21:28:49.487147 1955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13af1b96121c492c9373f9acbb55725a" candidate_uuid: "521bb93035324cecbebed2f6f2a630f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e632c553ff4341328acf1cbaca678392"
I20240502 21:28:49.487390 1955 raft_consensus.cc:2379] T 13af1b96121c492c9373f9acbb55725a P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 521bb93035324cecbebed2f6f2a630f8 in current term 1: Already voted for candidate 153026dd7f8943cc93fedf6f48450fb4 in this term.
I20240502 21:28:49.487623 1819 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13af1b96121c492c9373f9acbb55725a" candidate_uuid: "521bb93035324cecbebed2f6f2a630f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "153026dd7f8943cc93fedf6f48450fb4"
I20240502 21:28:49.487861 2226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13af1b96121c492c9373f9acbb55725a" candidate_uuid: "521bb93035324cecbebed2f6f2a630f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0"
I20240502 21:28:49.488085 2226 raft_consensus.cc:2379] T 13af1b96121c492c9373f9acbb55725a P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 521bb93035324cecbebed2f6f2a630f8 in current term 1: Already voted for candidate 153026dd7f8943cc93fedf6f48450fb4 in this term.
I20240502 21:28:49.488179 2024 leader_election.cc:302] T 13af1b96121c492c9373f9acbb55725a P 521bb93035324cecbebed2f6f2a630f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 5 responses out of 7 voters: 1 yes votes; 4 no votes. yes voters: 521bb93035324cecbebed2f6f2a630f8; no voters: 153026dd7f8943cc93fedf6f48450fb4, 16b3a337bc5e462f83ba895ad8660deb, a62a935ff43d494ca35d0c417943133d, e632c553ff4341328acf1cbaca678392
I20240502 21:28:49.488344 2088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f86d1517df74ead9e397ca9bed44c3f" candidate_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "521bb93035324cecbebed2f6f2a630f8" is_pre_election: true
I20240502 21:28:49.488512 2769 raft_consensus.cc:2726] T 13af1b96121c492c9373f9acbb55725a P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240502 21:28:49.488584 1752 leader_election.cc:302] T 3c88daaac3734d918955a774829532c6 P 153026dd7f8943cc93fedf6f48450fb4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 6 responses out of 7 voters: 2 yes votes; 4 no votes. yes voters: 153026dd7f8943cc93fedf6f48450fb4, 16b3a337bc5e462f83ba895ad8660deb; no voters: 521bb93035324cecbebed2f6f2a630f8, 6ab8df3c796f4b6cb3e29d5045ad3ea0, a62a935ff43d494ca35d0c417943133d, e632c553ff4341328acf1cbaca678392
I20240502 21:28:49.488867 2716 raft_consensus.cc:2726] T 3c88daaac3734d918955a774829532c6 P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
W20240502 21:28:49.488956 1752 leader_election.cc:341] T 3c88daaac3734d918955a774829532c6 P 153026dd7f8943cc93fedf6f48450fb4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7d1e5c1b047443cbb28d7b0a9744babf (127.15.255.129:37897): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:28:49.491488 1269 catalog_manager.cc:5484] T 13af1b96121c492c9373f9acbb55725a P 153026dd7f8943cc93fedf6f48450fb4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132). New cstate: current_term: 1 leader_uuid: "153026dd7f8943cc93fedf6f48450fb4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } health_report { overall_health: UNKNOWN } } }
I20240502 21:28:49.493865 1415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13af1b96121c492c9373f9acbb55725a" candidate_uuid: "153026dd7f8943cc93fedf6f48450fb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d1e5c1b047443cbb28d7b0a9744babf"
I20240502 21:28:49.494469 1415 raft_consensus.cc:3037] T 13af1b96121c492c9373f9acbb55725a P 7d1e5c1b047443cbb28d7b0a9744babf [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:49.494997 2769 consensus_queue.cc:1035] T 8f86d1517df74ead9e397ca9bed44c3f P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240502 21:28:49.495503 1413 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c88daaac3734d918955a774829532c6" candidate_uuid: "16b3a337bc5e462f83ba895ad8660deb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" is_pre_election: true
W20240502 21:28:49.496563 1617 leader_election.cc:341] T 3c88daaac3734d918955a774829532c6 P 16b3a337bc5e462f83ba895ad8660deb [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7d1e5c1b047443cbb28d7b0a9744babf (127.15.255.129:37897): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:28:49.497629 1414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13af1b96121c492c9373f9acbb55725a" candidate_uuid: "521bb93035324cecbebed2f6f2a630f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d1e5c1b047443cbb28d7b0a9744babf"
I20240502 21:28:49.501535 1415 raft_consensus.cc:2454] T 13af1b96121c492c9373f9acbb55725a P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 153026dd7f8943cc93fedf6f48450fb4 in term 1.
I20240502 21:28:49.503957 2728 tablet_bootstrap.cc:492] T 3c88daaac3734d918955a774829532c6 P 7d1e5c1b047443cbb28d7b0a9744babf: No bootstrap required, opened a new log
I20240502 21:28:49.504143 2728 ts_tablet_manager.cc:1367] T 3c88daaac3734d918955a774829532c6 P 7d1e5c1b047443cbb28d7b0a9744babf: Time spent bootstrapping tablet: real 0.500s user 0.213s sys 0.245s
I20240502 21:28:49.505450 2728 raft_consensus.cc:348] T 3c88daaac3734d918955a774829532c6 P 7d1e5c1b047443cbb28d7b0a9744babf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:28:49.505728 2728 raft_consensus.cc:374] T 3c88daaac3734d918955a774829532c6 P 7d1e5c1b047443cbb28d7b0a9744babf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:49.505821 2728 raft_consensus.cc:729] T 3c88daaac3734d918955a774829532c6 P 7d1e5c1b047443cbb28d7b0a9744babf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d1e5c1b047443cbb28d7b0a9744babf, State: Initialized, Role: FOLLOWER
I20240502 21:28:49.506134 2728 consensus_queue.cc:260] T 3c88daaac3734d918955a774829532c6 P 7d1e5c1b047443cbb28d7b0a9744babf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:28:49.514405 2728 ts_tablet_manager.cc:1398] T 3c88daaac3734d918955a774829532c6 P 7d1e5c1b047443cbb28d7b0a9744babf: Time spent starting tablet: real 0.010s user 0.010s sys 0.000s
I20240502 21:28:49.514730 2728 tablet_bootstrap.cc:492] T 8f86d1517df74ead9e397ca9bed44c3f P 7d1e5c1b047443cbb28d7b0a9744babf: Bootstrap starting.
I20240502 21:28:49.516698 2728 tablet_bootstrap.cc:654] T 8f86d1517df74ead9e397ca9bed44c3f P 7d1e5c1b047443cbb28d7b0a9744babf: Neither blocks nor log segments found. Creating new log.
W20240502 21:28:49.516759 2023 consensus_peers.cc:485] T 8f86d1517df74ead9e397ca9bed44c3f P 521bb93035324cecbebed2f6f2a630f8 -> Peer 7d1e5c1b047443cbb28d7b0a9744babf (127.15.255.129:37897): Couldn't send request to peer 7d1e5c1b047443cbb28d7b0a9744babf. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:28:49.518580 2728 tablet_bootstrap.cc:492] T 8f86d1517df74ead9e397ca9bed44c3f P 7d1e5c1b047443cbb28d7b0a9744babf: No bootstrap required, opened a new log
I20240502 21:28:49.518740 2728 ts_tablet_manager.cc:1367] T 8f86d1517df74ead9e397ca9bed44c3f P 7d1e5c1b047443cbb28d7b0a9744babf: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20240502 21:28:49.519415 2728 raft_consensus.cc:348] T 8f86d1517df74ead9e397ca9bed44c3f P 7d1e5c1b047443cbb28d7b0a9744babf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } }
I20240502 21:28:49.519640 2728 raft_consensus.cc:374] T 8f86d1517df74ead9e397ca9bed44c3f P 7d1e5c1b047443cbb28d7b0a9744babf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:49.519711 2728 raft_consensus.cc:729] T 8f86d1517df74ead9e397ca9bed44c3f P 7d1e5c1b047443cbb28d7b0a9744babf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d1e5c1b047443cbb28d7b0a9744babf, State: Initialized, Role: FOLLOWER
I20240502 21:28:49.519934 2728 consensus_queue.cc:260] T 8f86d1517df74ead9e397ca9bed44c3f P 7d1e5c1b047443cbb28d7b0a9744babf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } }
I20240502 21:28:49.520366 2728 ts_tablet_manager.cc:1398] T 8f86d1517df74ead9e397ca9bed44c3f P 7d1e5c1b047443cbb28d7b0a9744babf: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 21:28:49.551859 2796 raft_consensus.cc:482] T 8f86d1517df74ead9e397ca9bed44c3f P 7d1e5c1b047443cbb28d7b0a9744babf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:28:49.552016 2796 raft_consensus.cc:504] T 8f86d1517df74ead9e397ca9bed44c3f P 7d1e5c1b047443cbb28d7b0a9744babf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } }
I20240502 21:28:49.560717 2796 leader_election.cc:288] T 8f86d1517df74ead9e397ca9bed44c3f P 7d1e5c1b047443cbb28d7b0a9744babf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a62a935ff43d494ca35d0c417943133d (127.15.255.130:37991), 16b3a337bc5e462f83ba895ad8660deb (127.15.255.131:36033), 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132:44947), e632c553ff4341328acf1cbaca678392 (127.15.255.133:37439), 521bb93035324cecbebed2f6f2a630f8 (127.15.255.134:40821), 6ab8df3c796f4b6cb3e29d5045ad3ea0 (127.15.255.135:34701)
I20240502 21:28:49.586970 1819 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f86d1517df74ead9e397ca9bed44c3f" candidate_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "153026dd7f8943cc93fedf6f48450fb4" is_pre_election: true
I20240502 21:28:49.587338 1819 raft_consensus.cc:2379] T 8f86d1517df74ead9e397ca9bed44c3f P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7d1e5c1b047443cbb28d7b0a9744babf in current term 1: Already voted for candidate 521bb93035324cecbebed2f6f2a630f8 in this term.
I20240502 21:28:49.588275 1549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f86d1517df74ead9e397ca9bed44c3f" candidate_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a62a935ff43d494ca35d0c417943133d" is_pre_election: true
I20240502 21:28:49.591708 1685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f86d1517df74ead9e397ca9bed44c3f" candidate_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16b3a337bc5e462f83ba895ad8660deb" is_pre_election: true
I20240502 21:28:49.591837 2769 consensus_queue.cc:1035] T 8f86d1517df74ead9e397ca9bed44c3f P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:28:49.591969 1685 raft_consensus.cc:2379] T 8f86d1517df74ead9e397ca9bed44c3f P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7d1e5c1b047443cbb28d7b0a9744babf in current term 1: Already voted for candidate 521bb93035324cecbebed2f6f2a630f8 in this term.
I20240502 21:28:49.592886 2226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f86d1517df74ead9e397ca9bed44c3f" candidate_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" is_pre_election: true
I20240502 21:28:49.593129 2226 raft_consensus.cc:2454] T 8f86d1517df74ead9e397ca9bed44c3f P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d1e5c1b047443cbb28d7b0a9744babf in term 1.
I20240502 21:28:49.595800 1955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f86d1517df74ead9e397ca9bed44c3f" candidate_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e632c553ff4341328acf1cbaca678392" is_pre_election: true
I20240502 21:28:49.596057 1955 raft_consensus.cc:2379] T 8f86d1517df74ead9e397ca9bed44c3f P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7d1e5c1b047443cbb28d7b0a9744babf in current term 1: Already voted for candidate 521bb93035324cecbebed2f6f2a630f8 in this term.
I20240502 21:28:49.596731 1347 leader_election.cc:302] T 8f86d1517df74ead9e397ca9bed44c3f P 7d1e5c1b047443cbb28d7b0a9744babf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 6 responses out of 7 voters: 2 yes votes; 4 no votes. yes voters: 6ab8df3c796f4b6cb3e29d5045ad3ea0, 7d1e5c1b047443cbb28d7b0a9744babf; no voters: 153026dd7f8943cc93fedf6f48450fb4, 16b3a337bc5e462f83ba895ad8660deb, a62a935ff43d494ca35d0c417943133d, e632c553ff4341328acf1cbaca678392
I20240502 21:28:49.597239 2796 raft_consensus.cc:3037] T 8f86d1517df74ead9e397ca9bed44c3f P 7d1e5c1b047443cbb28d7b0a9744babf [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:49.599396 2796 raft_consensus.cc:2726] T 8f86d1517df74ead9e397ca9bed44c3f P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 21:28:49.599771 2088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f86d1517df74ead9e397ca9bed44c3f" candidate_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "521bb93035324cecbebed2f6f2a630f8" is_pre_election: true
I20240502 21:28:49.608342 2769 consensus_queue.cc:1035] T 8f86d1517df74ead9e397ca9bed44c3f P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:28:49.615538 2808 consensus_queue.cc:1035] T 8f86d1517df74ead9e397ca9bed44c3f P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:28:49.616214 2808 consensus_queue.cc:1035] T 8f86d1517df74ead9e397ca9bed44c3f P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:28:49.645728 2796 raft_consensus.cc:482] T 3c88daaac3734d918955a774829532c6 P 7d1e5c1b047443cbb28d7b0a9744babf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:28:49.645918 2796 raft_consensus.cc:504] T 3c88daaac3734d918955a774829532c6 P 7d1e5c1b047443cbb28d7b0a9744babf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:28:49.646909 2796 leader_election.cc:288] T 3c88daaac3734d918955a774829532c6 P 7d1e5c1b047443cbb28d7b0a9744babf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e632c553ff4341328acf1cbaca678392 (127.15.255.133:37439), 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132:44947), 521bb93035324cecbebed2f6f2a630f8 (127.15.255.134:40821), 6ab8df3c796f4b6cb3e29d5045ad3ea0 (127.15.255.135:34701), a62a935ff43d494ca35d0c417943133d (127.15.255.130:37991), 16b3a337bc5e462f83ba895ad8660deb (127.15.255.131:36033)
I20240502 21:28:49.646962 1955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c88daaac3734d918955a774829532c6" candidate_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e632c553ff4341328acf1cbaca678392" is_pre_election: true
I20240502 21:28:49.647064 1819 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c88daaac3734d918955a774829532c6" candidate_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "153026dd7f8943cc93fedf6f48450fb4" is_pre_election: true
I20240502 21:28:49.647198 1955 raft_consensus.cc:2379] T 3c88daaac3734d918955a774829532c6 P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7d1e5c1b047443cbb28d7b0a9744babf in current term 1: Already voted for candidate a62a935ff43d494ca35d0c417943133d in this term.
I20240502 21:28:49.647325 1819 raft_consensus.cc:2379] T 3c88daaac3734d918955a774829532c6 P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7d1e5c1b047443cbb28d7b0a9744babf in current term 1: Already voted for candidate 153026dd7f8943cc93fedf6f48450fb4 in this term.
I20240502 21:28:49.647313 2088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c88daaac3734d918955a774829532c6" candidate_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "521bb93035324cecbebed2f6f2a630f8" is_pre_election: true
I20240502 21:28:49.647454 2226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c88daaac3734d918955a774829532c6" candidate_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" is_pre_election: true
I20240502 21:28:49.647565 2088 raft_consensus.cc:2379] T 3c88daaac3734d918955a774829532c6 P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7d1e5c1b047443cbb28d7b0a9744babf in current term 1: Already voted for candidate a62a935ff43d494ca35d0c417943133d in this term.
I20240502 21:28:49.647692 2226 raft_consensus.cc:2379] T 3c88daaac3734d918955a774829532c6 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7d1e5c1b047443cbb28d7b0a9744babf in current term 1: Already voted for candidate a62a935ff43d494ca35d0c417943133d in this term.
I20240502 21:28:49.648026 1685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c88daaac3734d918955a774829532c6" candidate_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16b3a337bc5e462f83ba895ad8660deb" is_pre_election: true
I20240502 21:28:49.648074 1549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c88daaac3734d918955a774829532c6" candidate_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a62a935ff43d494ca35d0c417943133d" is_pre_election: true
I20240502 21:28:49.648247 1685 raft_consensus.cc:2379] T 3c88daaac3734d918955a774829532c6 P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7d1e5c1b047443cbb28d7b0a9744babf in current term 1: Already voted for candidate 153026dd7f8943cc93fedf6f48450fb4 in this term.
I20240502 21:28:49.648206 1347 leader_election.cc:302] T 3c88daaac3734d918955a774829532c6 P 7d1e5c1b047443cbb28d7b0a9744babf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 5 responses out of 7 voters: 1 yes votes; 4 no votes. yes voters: 7d1e5c1b047443cbb28d7b0a9744babf; no voters: 153026dd7f8943cc93fedf6f48450fb4, 521bb93035324cecbebed2f6f2a630f8, 6ab8df3c796f4b6cb3e29d5045ad3ea0, e632c553ff4341328acf1cbaca678392
I20240502 21:28:49.648593 2796 raft_consensus.cc:3037] T 3c88daaac3734d918955a774829532c6 P 7d1e5c1b047443cbb28d7b0a9744babf [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:49.650373 2796 raft_consensus.cc:2726] T 3c88daaac3734d918955a774829532c6 P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 21:28:49.855227 2729 consensus_queue.cc:1035] T 3c88daaac3734d918955a774829532c6 P a62a935ff43d494ca35d0c417943133d [LEADER]: Connected to new peer: Peer: permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:28:49.888516 2781 consensus_queue.cc:1035] T 13af1b96121c492c9373f9acbb55725a P 153026dd7f8943cc93fedf6f48450fb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:28:49.888952 2716 consensus_queue.cc:1035] T 13af1b96121c492c9373f9acbb55725a P 153026dd7f8943cc93fedf6f48450fb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:28:49.932401 2729 consensus_queue.cc:1035] T 3c88daaac3734d918955a774829532c6 P a62a935ff43d494ca35d0c417943133d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:28:49.943208 2716 consensus_queue.cc:1035] T 13af1b96121c492c9373f9acbb55725a P 153026dd7f8943cc93fedf6f48450fb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:28:49.947795 2729 consensus_queue.cc:1035] T 3c88daaac3734d918955a774829532c6 P a62a935ff43d494ca35d0c417943133d [LEADER]: Connected to new peer: Peer: permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:28:49.948601 2781 consensus_queue.cc:1035] T 13af1b96121c492c9373f9acbb55725a P 153026dd7f8943cc93fedf6f48450fb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:28:49.952551 2823 consensus_queue.cc:1035] T 13af1b96121c492c9373f9acbb55725a P 153026dd7f8943cc93fedf6f48450fb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:28:49.956818 2781 consensus_queue.cc:1035] T 13af1b96121c492c9373f9acbb55725a P 153026dd7f8943cc93fedf6f48450fb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:28:49.980715 2729 consensus_queue.cc:1035] T 3c88daaac3734d918955a774829532c6 P a62a935ff43d494ca35d0c417943133d [LEADER]: Connected to new peer: Peer: permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:28:49.986526 2729 consensus_queue.cc:1035] T 3c88daaac3734d918955a774829532c6 P a62a935ff43d494ca35d0c417943133d [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:28:49.992760 2833 consensus_queue.cc:1035] T 3c88daaac3734d918955a774829532c6 P a62a935ff43d494ca35d0c417943133d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 21:28:50.031754 2817 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:28:50.032073 2817 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:28:50.039638 2817 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_0'
INSERT report
rows total: 2000
time total: 531.349 ms
time per row: 0.265675 ms
-----------------------------------------------------
Suppressions used:
count bytes template
4 1192 libcrypto.so.1.1
-----------------------------------------------------
I20240502 21:28:51.587366 1267 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44934:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 7
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
W20240502 21:28:51.588101 1267 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table rebalance_test_table_1 in case of a server failure: 8 tablet servers would be needed, 7 are available. Consider bringing up more tablet servers.
I20240502 21:28:51.596882 2067 tablet_service.cc:1467] Processing CreateTablet for tablet e44bf56d64b24e328803ebd0a3a54ceb (DEFAULT_TABLE table=rebalance_test_table_1 [id=f2592b36bf814d51bd833a6ad315e513]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:51.597352 2067 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e44bf56d64b24e328803ebd0a3a54ceb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:51.597576 2207 tablet_service.cc:1467] Processing CreateTablet for tablet e44bf56d64b24e328803ebd0a3a54ceb (DEFAULT_TABLE table=rebalance_test_table_1 [id=f2592b36bf814d51bd833a6ad315e513]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:51.597921 2207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e44bf56d64b24e328803ebd0a3a54ceb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:51.597858 1526 tablet_service.cc:1467] Processing CreateTablet for tablet e44bf56d64b24e328803ebd0a3a54ceb (DEFAULT_TABLE table=rebalance_test_table_1 [id=f2592b36bf814d51bd833a6ad315e513]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:51.598289 1526 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e44bf56d64b24e328803ebd0a3a54ceb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:51.602391 1931 tablet_service.cc:1467] Processing CreateTablet for tablet e44bf56d64b24e328803ebd0a3a54ceb (DEFAULT_TABLE table=rebalance_test_table_1 [id=f2592b36bf814d51bd833a6ad315e513]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:51.602836 1931 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e44bf56d64b24e328803ebd0a3a54ceb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:51.603400 1798 tablet_service.cc:1467] Processing CreateTablet for tablet e44bf56d64b24e328803ebd0a3a54ceb (DEFAULT_TABLE table=rebalance_test_table_1 [id=f2592b36bf814d51bd833a6ad315e513]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:51.603807 1798 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e44bf56d64b24e328803ebd0a3a54ceb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:51.607160 2919 tablet_bootstrap.cc:492] T e44bf56d64b24e328803ebd0a3a54ceb P 521bb93035324cecbebed2f6f2a630f8: Bootstrap starting.
I20240502 21:28:51.610225 1800 tablet_service.cc:1467] Processing CreateTablet for tablet 444c1cba2a6f4729aa01b75d06b6b517 (DEFAULT_TABLE table=rebalance_test_table_1 [id=f2592b36bf814d51bd833a6ad315e513]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:51.610467 1933 tablet_service.cc:1467] Processing CreateTablet for tablet 444c1cba2a6f4729aa01b75d06b6b517 (DEFAULT_TABLE table=rebalance_test_table_1 [id=f2592b36bf814d51bd833a6ad315e513]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:51.610635 1800 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 444c1cba2a6f4729aa01b75d06b6b517. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:51.610641 2919 tablet_bootstrap.cc:654] T e44bf56d64b24e328803ebd0a3a54ceb P 521bb93035324cecbebed2f6f2a630f8: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:51.610819 1933 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 444c1cba2a6f4729aa01b75d06b6b517. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:51.613508 2920 tablet_bootstrap.cc:492] T e44bf56d64b24e328803ebd0a3a54ceb P a62a935ff43d494ca35d0c417943133d: Bootstrap starting.
I20240502 21:28:51.614990 2920 tablet_bootstrap.cc:654] T e44bf56d64b24e328803ebd0a3a54ceb P a62a935ff43d494ca35d0c417943133d: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:51.616945 2920 tablet_bootstrap.cc:492] T e44bf56d64b24e328803ebd0a3a54ceb P a62a935ff43d494ca35d0c417943133d: No bootstrap required, opened a new log
I20240502 21:28:51.617158 2920 ts_tablet_manager.cc:1367] T e44bf56d64b24e328803ebd0a3a54ceb P a62a935ff43d494ca35d0c417943133d: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240502 21:28:51.617601 1394 tablet_service.cc:1467] Processing CreateTablet for tablet e44bf56d64b24e328803ebd0a3a54ceb (DEFAULT_TABLE table=rebalance_test_table_1 [id=f2592b36bf814d51bd833a6ad315e513]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:51.618063 1394 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e44bf56d64b24e328803ebd0a3a54ceb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:51.617949 2920 raft_consensus.cc:348] T e44bf56d64b24e328803ebd0a3a54ceb P a62a935ff43d494ca35d0c417943133d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:28:51.618306 2920 raft_consensus.cc:374] T e44bf56d64b24e328803ebd0a3a54ceb P a62a935ff43d494ca35d0c417943133d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:51.618394 2920 raft_consensus.cc:729] T e44bf56d64b24e328803ebd0a3a54ceb P a62a935ff43d494ca35d0c417943133d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a62a935ff43d494ca35d0c417943133d, State: Initialized, Role: FOLLOWER
I20240502 21:28:51.618414 1664 tablet_service.cc:1467] Processing CreateTablet for tablet e44bf56d64b24e328803ebd0a3a54ceb (DEFAULT_TABLE table=rebalance_test_table_1 [id=f2592b36bf814d51bd833a6ad315e513]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:51.618645 2920 consensus_queue.cc:260] T e44bf56d64b24e328803ebd0a3a54ceb P a62a935ff43d494ca35d0c417943133d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:28:51.618839 1664 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e44bf56d64b24e328803ebd0a3a54ceb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:51.619719 2920 ts_tablet_manager.cc:1398] T e44bf56d64b24e328803ebd0a3a54ceb P a62a935ff43d494ca35d0c417943133d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:28:51.621042 2923 tablet_bootstrap.cc:492] T 444c1cba2a6f4729aa01b75d06b6b517 P 153026dd7f8943cc93fedf6f48450fb4: Bootstrap starting.
I20240502 21:28:51.623245 2923 tablet_bootstrap.cc:654] T 444c1cba2a6f4729aa01b75d06b6b517 P 153026dd7f8943cc93fedf6f48450fb4: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:51.624744 2918 tablet_bootstrap.cc:492] T e44bf56d64b24e328803ebd0a3a54ceb P 6ab8df3c796f4b6cb3e29d5045ad3ea0: Bootstrap starting.
I20240502 21:28:51.626801 2918 tablet_bootstrap.cc:654] T e44bf56d64b24e328803ebd0a3a54ceb P 6ab8df3c796f4b6cb3e29d5045ad3ea0: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:51.627895 2067 tablet_service.cc:1467] Processing CreateTablet for tablet 444c1cba2a6f4729aa01b75d06b6b517 (DEFAULT_TABLE table=rebalance_test_table_1 [id=f2592b36bf814d51bd833a6ad315e513]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:51.628295 2067 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 444c1cba2a6f4729aa01b75d06b6b517. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:51.634707 2207 tablet_service.cc:1467] Processing CreateTablet for tablet 444c1cba2a6f4729aa01b75d06b6b517 (DEFAULT_TABLE table=rebalance_test_table_1 [id=f2592b36bf814d51bd833a6ad315e513]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:51.635102 2207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 444c1cba2a6f4729aa01b75d06b6b517. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:51.635378 2918 tablet_bootstrap.cc:492] T e44bf56d64b24e328803ebd0a3a54ceb P 6ab8df3c796f4b6cb3e29d5045ad3ea0: No bootstrap required, opened a new log
I20240502 21:28:51.635298 1526 tablet_service.cc:1467] Processing CreateTablet for tablet 444c1cba2a6f4729aa01b75d06b6b517 (DEFAULT_TABLE table=rebalance_test_table_1 [id=f2592b36bf814d51bd833a6ad315e513]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:51.635542 2918 ts_tablet_manager.cc:1367] T e44bf56d64b24e328803ebd0a3a54ceb P 6ab8df3c796f4b6cb3e29d5045ad3ea0: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.004s
I20240502 21:28:51.635651 1526 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 444c1cba2a6f4729aa01b75d06b6b517. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:51.636435 2918 raft_consensus.cc:348] T e44bf56d64b24e328803ebd0a3a54ceb P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:28:51.636744 2918 raft_consensus.cc:374] T e44bf56d64b24e328803ebd0a3a54ceb P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:51.637117 2918 raft_consensus.cc:729] T e44bf56d64b24e328803ebd0a3a54ceb P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ab8df3c796f4b6cb3e29d5045ad3ea0, State: Initialized, Role: FOLLOWER
I20240502 21:28:51.637746 2918 consensus_queue.cc:260] T e44bf56d64b24e328803ebd0a3a54ceb P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:28:51.642798 2923 tablet_bootstrap.cc:492] T 444c1cba2a6f4729aa01b75d06b6b517 P 153026dd7f8943cc93fedf6f48450fb4: No bootstrap required, opened a new log
I20240502 21:28:51.642956 2923 ts_tablet_manager.cc:1367] T 444c1cba2a6f4729aa01b75d06b6b517 P 153026dd7f8943cc93fedf6f48450fb4: Time spent bootstrapping tablet: real 0.022s user 0.000s sys 0.008s
I20240502 21:28:51.643700 2923 raft_consensus.cc:348] T 444c1cba2a6f4729aa01b75d06b6b517 P 153026dd7f8943cc93fedf6f48450fb4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:28:51.643977 2923 raft_consensus.cc:374] T 444c1cba2a6f4729aa01b75d06b6b517 P 153026dd7f8943cc93fedf6f48450fb4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:51.644055 2923 raft_consensus.cc:729] T 444c1cba2a6f4729aa01b75d06b6b517 P 153026dd7f8943cc93fedf6f48450fb4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 153026dd7f8943cc93fedf6f48450fb4, State: Initialized, Role: FOLLOWER
I20240502 21:28:51.644465 2923 consensus_queue.cc:260] T 444c1cba2a6f4729aa01b75d06b6b517 P 153026dd7f8943cc93fedf6f48450fb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:28:51.645150 1393 tablet_service.cc:1467] Processing CreateTablet for tablet 444c1cba2a6f4729aa01b75d06b6b517 (DEFAULT_TABLE table=rebalance_test_table_1 [id=f2592b36bf814d51bd833a6ad315e513]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:51.645558 1393 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 444c1cba2a6f4729aa01b75d06b6b517. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:51.645648 1800 tablet_service.cc:1467] Processing CreateTablet for tablet 20a7af42e83c4639bba20f22dee69f2a (DEFAULT_TABLE table=rebalance_test_table_1 [id=f2592b36bf814d51bd833a6ad315e513]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:51.645887 1932 tablet_service.cc:1467] Processing CreateTablet for tablet 20a7af42e83c4639bba20f22dee69f2a (DEFAULT_TABLE table=rebalance_test_table_1 [id=f2592b36bf814d51bd833a6ad315e513]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:51.646028 1800 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20a7af42e83c4639bba20f22dee69f2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:51.646282 1932 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20a7af42e83c4639bba20f22dee69f2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:51.651330 2923 ts_tablet_manager.cc:1398] T 444c1cba2a6f4729aa01b75d06b6b517 P 153026dd7f8943cc93fedf6f48450fb4: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20240502 21:28:51.652312 2923 tablet_bootstrap.cc:492] T e44bf56d64b24e328803ebd0a3a54ceb P 153026dd7f8943cc93fedf6f48450fb4: Bootstrap starting.
I20240502 21:28:51.653934 2923 tablet_bootstrap.cc:654] T e44bf56d64b24e328803ebd0a3a54ceb P 153026dd7f8943cc93fedf6f48450fb4: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:51.654559 1662 tablet_service.cc:1467] Processing CreateTablet for tablet 444c1cba2a6f4729aa01b75d06b6b517 (DEFAULT_TABLE table=rebalance_test_table_1 [id=f2592b36bf814d51bd833a6ad315e513]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:51.654999 1662 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 444c1cba2a6f4729aa01b75d06b6b517. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:51.662145 2930 tablet_bootstrap.cc:492] T 444c1cba2a6f4729aa01b75d06b6b517 P 16b3a337bc5e462f83ba895ad8660deb: Bootstrap starting.
I20240502 21:28:51.663107 2920 tablet_bootstrap.cc:492] T 444c1cba2a6f4729aa01b75d06b6b517 P a62a935ff43d494ca35d0c417943133d: Bootstrap starting.
I20240502 21:28:51.664697 2920 tablet_bootstrap.cc:654] T 444c1cba2a6f4729aa01b75d06b6b517 P a62a935ff43d494ca35d0c417943133d: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:51.665431 2930 tablet_bootstrap.cc:654] T 444c1cba2a6f4729aa01b75d06b6b517 P 16b3a337bc5e462f83ba895ad8660deb: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:51.668184 1393 tablet_service.cc:1467] Processing CreateTablet for tablet 20a7af42e83c4639bba20f22dee69f2a (DEFAULT_TABLE table=rebalance_test_table_1 [id=f2592b36bf814d51bd833a6ad315e513]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:51.668642 1393 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20a7af42e83c4639bba20f22dee69f2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:51.669301 2924 tablet_bootstrap.cc:492] T 444c1cba2a6f4729aa01b75d06b6b517 P e632c553ff4341328acf1cbaca678392: Bootstrap starting.
I20240502 21:28:51.669272 1526 tablet_service.cc:1467] Processing CreateTablet for tablet 20a7af42e83c4639bba20f22dee69f2a (DEFAULT_TABLE table=rebalance_test_table_1 [id=f2592b36bf814d51bd833a6ad315e513]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:51.669637 1526 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20a7af42e83c4639bba20f22dee69f2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:51.670490 2923 tablet_bootstrap.cc:492] T e44bf56d64b24e328803ebd0a3a54ceb P 153026dd7f8943cc93fedf6f48450fb4: No bootstrap required, opened a new log
I20240502 21:28:51.670675 2923 ts_tablet_manager.cc:1367] T e44bf56d64b24e328803ebd0a3a54ceb P 153026dd7f8943cc93fedf6f48450fb4: Time spent bootstrapping tablet: real 0.018s user 0.004s sys 0.002s
I20240502 21:28:51.670814 2924 tablet_bootstrap.cc:654] T 444c1cba2a6f4729aa01b75d06b6b517 P e632c553ff4341328acf1cbaca678392: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:51.671394 2923 raft_consensus.cc:348] T e44bf56d64b24e328803ebd0a3a54ceb P 153026dd7f8943cc93fedf6f48450fb4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:28:51.671669 2923 raft_consensus.cc:374] T e44bf56d64b24e328803ebd0a3a54ceb P 153026dd7f8943cc93fedf6f48450fb4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:51.672814 1664 tablet_service.cc:1467] Processing CreateTablet for tablet 20a7af42e83c4639bba20f22dee69f2a (DEFAULT_TABLE table=rebalance_test_table_1 [id=f2592b36bf814d51bd833a6ad315e513]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:51.671741 2923 raft_consensus.cc:729] T e44bf56d64b24e328803ebd0a3a54ceb P 153026dd7f8943cc93fedf6f48450fb4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 153026dd7f8943cc93fedf6f48450fb4, State: Initialized, Role: FOLLOWER
I20240502 21:28:51.673197 1664 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20a7af42e83c4639bba20f22dee69f2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:51.673359 2923 consensus_queue.cc:260] T e44bf56d64b24e328803ebd0a3a54ceb P 153026dd7f8943cc93fedf6f48450fb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:28:51.674155 2923 ts_tablet_manager.cc:1398] T e44bf56d64b24e328803ebd0a3a54ceb P 153026dd7f8943cc93fedf6f48450fb4: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20240502 21:28:51.667579 2207 tablet_service.cc:1467] Processing CreateTablet for tablet 20a7af42e83c4639bba20f22dee69f2a (DEFAULT_TABLE table=rebalance_test_table_1 [id=f2592b36bf814d51bd833a6ad315e513]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:51.674731 2207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20a7af42e83c4639bba20f22dee69f2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:51.675786 2067 tablet_service.cc:1467] Processing CreateTablet for tablet 20a7af42e83c4639bba20f22dee69f2a (DEFAULT_TABLE table=rebalance_test_table_1 [id=f2592b36bf814d51bd833a6ad315e513]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:51.676205 2923 tablet_bootstrap.cc:492] T 20a7af42e83c4639bba20f22dee69f2a P 153026dd7f8943cc93fedf6f48450fb4: Bootstrap starting.
I20240502 21:28:51.676224 2067 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20a7af42e83c4639bba20f22dee69f2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:51.679157 2923 tablet_bootstrap.cc:654] T 20a7af42e83c4639bba20f22dee69f2a P 153026dd7f8943cc93fedf6f48450fb4: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:51.680595 2924 tablet_bootstrap.cc:492] T 444c1cba2a6f4729aa01b75d06b6b517 P e632c553ff4341328acf1cbaca678392: No bootstrap required, opened a new log
I20240502 21:28:51.680748 2924 ts_tablet_manager.cc:1367] T 444c1cba2a6f4729aa01b75d06b6b517 P e632c553ff4341328acf1cbaca678392: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.002s
I20240502 21:28:51.681100 2918 ts_tablet_manager.cc:1398] T e44bf56d64b24e328803ebd0a3a54ceb P 6ab8df3c796f4b6cb3e29d5045ad3ea0: Time spent starting tablet: real 0.045s user 0.004s sys 0.004s
I20240502 21:28:51.681403 2918 tablet_bootstrap.cc:492] T 444c1cba2a6f4729aa01b75d06b6b517 P 6ab8df3c796f4b6cb3e29d5045ad3ea0: Bootstrap starting.
I20240502 21:28:51.681495 2924 raft_consensus.cc:348] T 444c1cba2a6f4729aa01b75d06b6b517 P e632c553ff4341328acf1cbaca678392 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:28:51.681752 2924 raft_consensus.cc:374] T 444c1cba2a6f4729aa01b75d06b6b517 P e632c553ff4341328acf1cbaca678392 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:51.681852 2924 raft_consensus.cc:729] T 444c1cba2a6f4729aa01b75d06b6b517 P e632c553ff4341328acf1cbaca678392 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e632c553ff4341328acf1cbaca678392, State: Initialized, Role: FOLLOWER
I20240502 21:28:51.682906 2918 tablet_bootstrap.cc:654] T 444c1cba2a6f4729aa01b75d06b6b517 P 6ab8df3c796f4b6cb3e29d5045ad3ea0: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:51.684255 2920 tablet_bootstrap.cc:492] T 444c1cba2a6f4729aa01b75d06b6b517 P a62a935ff43d494ca35d0c417943133d: No bootstrap required, opened a new log
I20240502 21:28:51.684441 2920 ts_tablet_manager.cc:1367] T 444c1cba2a6f4729aa01b75d06b6b517 P a62a935ff43d494ca35d0c417943133d: Time spent bootstrapping tablet: real 0.021s user 0.012s sys 0.000s
I20240502 21:28:51.684486 2823 raft_consensus.cc:482] T e44bf56d64b24e328803ebd0a3a54ceb P 153026dd7f8943cc93fedf6f48450fb4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:28:51.684643 2823 raft_consensus.cc:504] T e44bf56d64b24e328803ebd0a3a54ceb P 153026dd7f8943cc93fedf6f48450fb4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:28:51.685194 2920 raft_consensus.cc:348] T 444c1cba2a6f4729aa01b75d06b6b517 P a62a935ff43d494ca35d0c417943133d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:28:51.685487 2920 raft_consensus.cc:374] T 444c1cba2a6f4729aa01b75d06b6b517 P a62a935ff43d494ca35d0c417943133d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:51.685582 2920 raft_consensus.cc:729] T 444c1cba2a6f4729aa01b75d06b6b517 P a62a935ff43d494ca35d0c417943133d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a62a935ff43d494ca35d0c417943133d, State: Initialized, Role: FOLLOWER
I20240502 21:28:51.685842 2823 leader_election.cc:288] T e44bf56d64b24e328803ebd0a3a54ceb P 153026dd7f8943cc93fedf6f48450fb4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 521bb93035324cecbebed2f6f2a630f8 (127.15.255.134:40821), e632c553ff4341328acf1cbaca678392 (127.15.255.133:37439), 6ab8df3c796f4b6cb3e29d5045ad3ea0 (127.15.255.135:34701), a62a935ff43d494ca35d0c417943133d (127.15.255.130:37991), 7d1e5c1b047443cbb28d7b0a9744babf (127.15.255.129:37897), 16b3a337bc5e462f83ba895ad8660deb (127.15.255.131:36033)
I20240502 21:28:51.685906 2920 consensus_queue.cc:260] T 444c1cba2a6f4729aa01b75d06b6b517 P a62a935ff43d494ca35d0c417943133d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:28:51.686491 2920 ts_tablet_manager.cc:1398] T 444c1cba2a6f4729aa01b75d06b6b517 P a62a935ff43d494ca35d0c417943133d: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 21:28:51.686844 2924 consensus_queue.cc:260] T 444c1cba2a6f4729aa01b75d06b6b517 P e632c553ff4341328acf1cbaca678392 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:28:51.687841 2088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e44bf56d64b24e328803ebd0a3a54ceb" candidate_uuid: "153026dd7f8943cc93fedf6f48450fb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "521bb93035324cecbebed2f6f2a630f8" is_pre_election: true
I20240502 21:28:51.688721 1953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e44bf56d64b24e328803ebd0a3a54ceb" candidate_uuid: "153026dd7f8943cc93fedf6f48450fb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e632c553ff4341328acf1cbaca678392" is_pre_election: true
I20240502 21:28:51.689195 2227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e44bf56d64b24e328803ebd0a3a54ceb" candidate_uuid: "153026dd7f8943cc93fedf6f48450fb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" is_pre_election: true
I20240502 21:28:51.689441 2227 raft_consensus.cc:2454] T e44bf56d64b24e328803ebd0a3a54ceb P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 153026dd7f8943cc93fedf6f48450fb4 in term 0.
I20240502 21:28:51.689832 1415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e44bf56d64b24e328803ebd0a3a54ceb" candidate_uuid: "153026dd7f8943cc93fedf6f48450fb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" is_pre_election: true
I20240502 21:28:51.690591 1549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e44bf56d64b24e328803ebd0a3a54ceb" candidate_uuid: "153026dd7f8943cc93fedf6f48450fb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a62a935ff43d494ca35d0c417943133d" is_pre_election: true
I20240502 21:28:51.690800 1549 raft_consensus.cc:2454] T e44bf56d64b24e328803ebd0a3a54ceb P a62a935ff43d494ca35d0c417943133d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 153026dd7f8943cc93fedf6f48450fb4 in term 0.
I20240502 21:28:51.691031 2924 ts_tablet_manager.cc:1398] T 444c1cba2a6f4729aa01b75d06b6b517 P e632c553ff4341328acf1cbaca678392: Time spent starting tablet: real 0.010s user 0.005s sys 0.000s
I20240502 21:28:51.691056 2926 tablet_bootstrap.cc:492] T e44bf56d64b24e328803ebd0a3a54ceb P 7d1e5c1b047443cbb28d7b0a9744babf: Bootstrap starting.
I20240502 21:28:51.691362 2924 tablet_bootstrap.cc:492] T 20a7af42e83c4639bba20f22dee69f2a P e632c553ff4341328acf1cbaca678392: Bootstrap starting.
I20240502 21:28:51.692137 1684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e44bf56d64b24e328803ebd0a3a54ceb" candidate_uuid: "153026dd7f8943cc93fedf6f48450fb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16b3a337bc5e462f83ba895ad8660deb" is_pre_election: true
I20240502 21:28:51.692698 2926 tablet_bootstrap.cc:654] T e44bf56d64b24e328803ebd0a3a54ceb P 7d1e5c1b047443cbb28d7b0a9744babf: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:51.692878 2924 tablet_bootstrap.cc:654] T 20a7af42e83c4639bba20f22dee69f2a P e632c553ff4341328acf1cbaca678392: Neither blocks nor log segments found. Creating new log.
W20240502 21:28:51.693097 1754 leader_election.cc:341] T e44bf56d64b24e328803ebd0a3a54ceb P 153026dd7f8943cc93fedf6f48450fb4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 16b3a337bc5e462f83ba895ad8660deb (127.15.255.131:36033): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:28:51.693730 2920 tablet_bootstrap.cc:492] T 20a7af42e83c4639bba20f22dee69f2a P a62a935ff43d494ca35d0c417943133d: Bootstrap starting.
I20240502 21:28:51.693840 2935 raft_consensus.cc:482] T 444c1cba2a6f4729aa01b75d06b6b517 P e632c553ff4341328acf1cbaca678392 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240502 21:28:51.693863 1752 leader_election.cc:341] T e44bf56d64b24e328803ebd0a3a54ceb P 153026dd7f8943cc93fedf6f48450fb4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7d1e5c1b047443cbb28d7b0a9744babf (127.15.255.129:37897): Illegal state: must be running to vote when last-logged opid is not known
W20240502 21:28:51.694145 1751 leader_election.cc:341] T e44bf56d64b24e328803ebd0a3a54ceb P 153026dd7f8943cc93fedf6f48450fb4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 521bb93035324cecbebed2f6f2a630f8 (127.15.255.134:40821): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:28:51.694012 2935 raft_consensus.cc:504] T 444c1cba2a6f4729aa01b75d06b6b517 P e632c553ff4341328acf1cbaca678392 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
W20240502 21:28:51.694330 1751 leader_election.cc:341] T e44bf56d64b24e328803ebd0a3a54ceb P 153026dd7f8943cc93fedf6f48450fb4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e632c553ff4341328acf1cbaca678392 (127.15.255.133:37439): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:28:51.694440 1751 leader_election.cc:302] T e44bf56d64b24e328803ebd0a3a54ceb P 153026dd7f8943cc93fedf6f48450fb4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 6 responses out of 7 voters: 2 yes votes; 4 no votes. yes voters: 153026dd7f8943cc93fedf6f48450fb4, 6ab8df3c796f4b6cb3e29d5045ad3ea0; no voters: 16b3a337bc5e462f83ba895ad8660deb, 521bb93035324cecbebed2f6f2a630f8, 7d1e5c1b047443cbb28d7b0a9744babf, e632c553ff4341328acf1cbaca678392
I20240502 21:28:51.695217 2920 tablet_bootstrap.cc:654] T 20a7af42e83c4639bba20f22dee69f2a P a62a935ff43d494ca35d0c417943133d: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:51.697602 2823 raft_consensus.cc:2726] T e44bf56d64b24e328803ebd0a3a54ceb P 153026dd7f8943cc93fedf6f48450fb4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 21:28:51.698978 2919 tablet_bootstrap.cc:492] T e44bf56d64b24e328803ebd0a3a54ceb P 521bb93035324cecbebed2f6f2a630f8: No bootstrap required, opened a new log
I20240502 21:28:51.699193 2919 ts_tablet_manager.cc:1367] T e44bf56d64b24e328803ebd0a3a54ceb P 521bb93035324cecbebed2f6f2a630f8: Time spent bootstrapping tablet: real 0.092s user 0.013s sys 0.007s
I20240502 21:28:51.699999 2919 raft_consensus.cc:348] T e44bf56d64b24e328803ebd0a3a54ceb P 521bb93035324cecbebed2f6f2a630f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:28:51.700512 2919 raft_consensus.cc:374] T e44bf56d64b24e328803ebd0a3a54ceb P 521bb93035324cecbebed2f6f2a630f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:51.700662 2919 raft_consensus.cc:729] T e44bf56d64b24e328803ebd0a3a54ceb P 521bb93035324cecbebed2f6f2a630f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 521bb93035324cecbebed2f6f2a630f8, State: Initialized, Role: FOLLOWER
I20240502 21:28:51.700973 2919 consensus_queue.cc:260] T e44bf56d64b24e328803ebd0a3a54ceb P 521bb93035324cecbebed2f6f2a630f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:28:51.703111 2919 ts_tablet_manager.cc:1398] T e44bf56d64b24e328803ebd0a3a54ceb P 521bb93035324cecbebed2f6f2a630f8: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240502 21:28:51.703444 2919 tablet_bootstrap.cc:492] T 444c1cba2a6f4729aa01b75d06b6b517 P 521bb93035324cecbebed2f6f2a630f8: Bootstrap starting.
I20240502 21:28:51.704957 2930 tablet_bootstrap.cc:492] T 444c1cba2a6f4729aa01b75d06b6b517 P 16b3a337bc5e462f83ba895ad8660deb: No bootstrap required, opened a new log
I20240502 21:28:51.705168 2930 ts_tablet_manager.cc:1367] T 444c1cba2a6f4729aa01b75d06b6b517 P 16b3a337bc5e462f83ba895ad8660deb: Time spent bootstrapping tablet: real 0.043s user 0.003s sys 0.004s
I20240502 21:28:51.706104 2930 raft_consensus.cc:348] T 444c1cba2a6f4729aa01b75d06b6b517 P 16b3a337bc5e462f83ba895ad8660deb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:28:51.706362 2923 tablet_bootstrap.cc:492] T 20a7af42e83c4639bba20f22dee69f2a P 153026dd7f8943cc93fedf6f48450fb4: No bootstrap required, opened a new log
I20240502 21:28:51.706475 2930 raft_consensus.cc:374] T 444c1cba2a6f4729aa01b75d06b6b517 P 16b3a337bc5e462f83ba895ad8660deb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:51.706522 2923 ts_tablet_manager.cc:1367] T 20a7af42e83c4639bba20f22dee69f2a P 153026dd7f8943cc93fedf6f48450fb4: Time spent bootstrapping tablet: real 0.030s user 0.003s sys 0.000s
I20240502 21:28:51.706640 2930 raft_consensus.cc:729] T 444c1cba2a6f4729aa01b75d06b6b517 P 16b3a337bc5e462f83ba895ad8660deb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16b3a337bc5e462f83ba895ad8660deb, State: Initialized, Role: FOLLOWER
I20240502 21:28:51.706936 2930 consensus_queue.cc:260] T 444c1cba2a6f4729aa01b75d06b6b517 P 16b3a337bc5e462f83ba895ad8660deb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:28:51.707248 2923 raft_consensus.cc:348] T 20a7af42e83c4639bba20f22dee69f2a P 153026dd7f8943cc93fedf6f48450fb4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:28:51.707676 2923 raft_consensus.cc:374] T 20a7af42e83c4639bba20f22dee69f2a P 153026dd7f8943cc93fedf6f48450fb4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:51.707792 2923 raft_consensus.cc:729] T 20a7af42e83c4639bba20f22dee69f2a P 153026dd7f8943cc93fedf6f48450fb4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 153026dd7f8943cc93fedf6f48450fb4, State: Initialized, Role: FOLLOWER
I20240502 21:28:51.708066 2923 consensus_queue.cc:260] T 20a7af42e83c4639bba20f22dee69f2a P 153026dd7f8943cc93fedf6f48450fb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:28:51.708822 2923 ts_tablet_manager.cc:1398] T 20a7af42e83c4639bba20f22dee69f2a P 153026dd7f8943cc93fedf6f48450fb4: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 21:28:51.709117 2919 tablet_bootstrap.cc:654] T 444c1cba2a6f4729aa01b75d06b6b517 P 521bb93035324cecbebed2f6f2a630f8: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:51.713265 2918 tablet_bootstrap.cc:492] T 444c1cba2a6f4729aa01b75d06b6b517 P 6ab8df3c796f4b6cb3e29d5045ad3ea0: No bootstrap required, opened a new log
I20240502 21:28:51.713462 2918 ts_tablet_manager.cc:1367] T 444c1cba2a6f4729aa01b75d06b6b517 P 6ab8df3c796f4b6cb3e29d5045ad3ea0: Time spent bootstrapping tablet: real 0.032s user 0.001s sys 0.002s
I20240502 21:28:51.714164 2918 raft_consensus.cc:348] T 444c1cba2a6f4729aa01b75d06b6b517 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:28:51.714463 2918 raft_consensus.cc:374] T 444c1cba2a6f4729aa01b75d06b6b517 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:51.714535 2918 raft_consensus.cc:729] T 444c1cba2a6f4729aa01b75d06b6b517 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ab8df3c796f4b6cb3e29d5045ad3ea0, State: Initialized, Role: FOLLOWER
I20240502 21:28:51.714931 2919 tablet_bootstrap.cc:492] T 444c1cba2a6f4729aa01b75d06b6b517 P 521bb93035324cecbebed2f6f2a630f8: No bootstrap required, opened a new log
I20240502 21:28:51.714767 2918 consensus_queue.cc:260] T 444c1cba2a6f4729aa01b75d06b6b517 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:28:51.715298 2919 ts_tablet_manager.cc:1367] T 444c1cba2a6f4729aa01b75d06b6b517 P 521bb93035324cecbebed2f6f2a630f8: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.000s
I20240502 21:28:51.715306 2918 ts_tablet_manager.cc:1398] T 444c1cba2a6f4729aa01b75d06b6b517 P 6ab8df3c796f4b6cb3e29d5045ad3ea0: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20240502 21:28:51.715620 2918 tablet_bootstrap.cc:492] T 20a7af42e83c4639bba20f22dee69f2a P 6ab8df3c796f4b6cb3e29d5045ad3ea0: Bootstrap starting.
I20240502 21:28:51.718029 2918 tablet_bootstrap.cc:654] T 20a7af42e83c4639bba20f22dee69f2a P 6ab8df3c796f4b6cb3e29d5045ad3ea0: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:51.722528 2930 ts_tablet_manager.cc:1398] T 444c1cba2a6f4729aa01b75d06b6b517 P 16b3a337bc5e462f83ba895ad8660deb: Time spent starting tablet: real 0.017s user 0.000s sys 0.005s
I20240502 21:28:51.722680 2919 raft_consensus.cc:348] T 444c1cba2a6f4729aa01b75d06b6b517 P 521bb93035324cecbebed2f6f2a630f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:28:51.724880 2930 tablet_bootstrap.cc:492] T e44bf56d64b24e328803ebd0a3a54ceb P 16b3a337bc5e462f83ba895ad8660deb: Bootstrap starting.
I20240502 21:28:51.728041 2930 tablet_bootstrap.cc:654] T e44bf56d64b24e328803ebd0a3a54ceb P 16b3a337bc5e462f83ba895ad8660deb: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:51.728113 2920 tablet_bootstrap.cc:492] T 20a7af42e83c4639bba20f22dee69f2a P a62a935ff43d494ca35d0c417943133d: No bootstrap required, opened a new log
I20240502 21:28:51.728281 2920 ts_tablet_manager.cc:1367] T 20a7af42e83c4639bba20f22dee69f2a P a62a935ff43d494ca35d0c417943133d: Time spent bootstrapping tablet: real 0.035s user 0.004s sys 0.014s
I20240502 21:28:51.729151 2919 raft_consensus.cc:374] T 444c1cba2a6f4729aa01b75d06b6b517 P 521bb93035324cecbebed2f6f2a630f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:51.729068 2920 raft_consensus.cc:348] T 20a7af42e83c4639bba20f22dee69f2a P a62a935ff43d494ca35d0c417943133d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:28:51.729348 2920 raft_consensus.cc:374] T 20a7af42e83c4639bba20f22dee69f2a P a62a935ff43d494ca35d0c417943133d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:51.729308 2919 raft_consensus.cc:729] T 444c1cba2a6f4729aa01b75d06b6b517 P 521bb93035324cecbebed2f6f2a630f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 521bb93035324cecbebed2f6f2a630f8, State: Initialized, Role: FOLLOWER
I20240502 21:28:51.729441 2920 raft_consensus.cc:729] T 20a7af42e83c4639bba20f22dee69f2a P a62a935ff43d494ca35d0c417943133d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a62a935ff43d494ca35d0c417943133d, State: Initialized, Role: FOLLOWER
I20240502 21:28:51.729718 2920 consensus_queue.cc:260] T 20a7af42e83c4639bba20f22dee69f2a P a62a935ff43d494ca35d0c417943133d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:28:51.729772 2919 consensus_queue.cc:260] T 444c1cba2a6f4729aa01b75d06b6b517 P 521bb93035324cecbebed2f6f2a630f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:28:51.730449 2920 ts_tablet_manager.cc:1398] T 20a7af42e83c4639bba20f22dee69f2a P a62a935ff43d494ca35d0c417943133d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:28:51.733570 2919 ts_tablet_manager.cc:1398] T 444c1cba2a6f4729aa01b75d06b6b517 P 521bb93035324cecbebed2f6f2a630f8: Time spent starting tablet: real 0.013s user 0.002s sys 0.000s
I20240502 21:28:51.733918 2919 tablet_bootstrap.cc:492] T 20a7af42e83c4639bba20f22dee69f2a P 521bb93035324cecbebed2f6f2a630f8: Bootstrap starting.
I20240502 21:28:51.734266 2930 tablet_bootstrap.cc:492] T e44bf56d64b24e328803ebd0a3a54ceb P 16b3a337bc5e462f83ba895ad8660deb: No bootstrap required, opened a new log
I20240502 21:28:51.734432 2930 ts_tablet_manager.cc:1367] T e44bf56d64b24e328803ebd0a3a54ceb P 16b3a337bc5e462f83ba895ad8660deb: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.002s
I20240502 21:28:51.735108 2930 raft_consensus.cc:348] T e44bf56d64b24e328803ebd0a3a54ceb P 16b3a337bc5e462f83ba895ad8660deb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:28:51.735381 2930 raft_consensus.cc:374] T e44bf56d64b24e328803ebd0a3a54ceb P 16b3a337bc5e462f83ba895ad8660deb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:51.735466 2930 raft_consensus.cc:729] T e44bf56d64b24e328803ebd0a3a54ceb P 16b3a337bc5e462f83ba895ad8660deb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16b3a337bc5e462f83ba895ad8660deb, State: Initialized, Role: FOLLOWER
I20240502 21:28:51.735643 2919 tablet_bootstrap.cc:654] T 20a7af42e83c4639bba20f22dee69f2a P 521bb93035324cecbebed2f6f2a630f8: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:51.735749 2930 consensus_queue.cc:260] T e44bf56d64b24e328803ebd0a3a54ceb P 16b3a337bc5e462f83ba895ad8660deb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:28:51.736285 2930 ts_tablet_manager.cc:1398] T e44bf56d64b24e328803ebd0a3a54ceb P 16b3a337bc5e462f83ba895ad8660deb: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 21:28:51.736652 2930 tablet_bootstrap.cc:492] T 20a7af42e83c4639bba20f22dee69f2a P 16b3a337bc5e462f83ba895ad8660deb: Bootstrap starting.
I20240502 21:28:51.738263 2930 tablet_bootstrap.cc:654] T 20a7af42e83c4639bba20f22dee69f2a P 16b3a337bc5e462f83ba895ad8660deb: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:51.740130 2926 tablet_bootstrap.cc:492] T e44bf56d64b24e328803ebd0a3a54ceb P 7d1e5c1b047443cbb28d7b0a9744babf: No bootstrap required, opened a new log
I20240502 21:28:51.740304 2926 ts_tablet_manager.cc:1367] T e44bf56d64b24e328803ebd0a3a54ceb P 7d1e5c1b047443cbb28d7b0a9744babf: Time spent bootstrapping tablet: real 0.049s user 0.005s sys 0.017s
I20240502 21:28:51.741137 2926 raft_consensus.cc:348] T e44bf56d64b24e328803ebd0a3a54ceb P 7d1e5c1b047443cbb28d7b0a9744babf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:28:51.741425 2926 raft_consensus.cc:374] T e44bf56d64b24e328803ebd0a3a54ceb P 7d1e5c1b047443cbb28d7b0a9744babf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:51.741510 2926 raft_consensus.cc:729] T e44bf56d64b24e328803ebd0a3a54ceb P 7d1e5c1b047443cbb28d7b0a9744babf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d1e5c1b047443cbb28d7b0a9744babf, State: Initialized, Role: FOLLOWER
I20240502 21:28:51.741783 2926 consensus_queue.cc:260] T e44bf56d64b24e328803ebd0a3a54ceb P 7d1e5c1b047443cbb28d7b0a9744babf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:28:51.743271 2935 leader_election.cc:288] T 444c1cba2a6f4729aa01b75d06b6b517 P e632c553ff4341328acf1cbaca678392 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132:44947), 521bb93035324cecbebed2f6f2a630f8 (127.15.255.134:40821), 6ab8df3c796f4b6cb3e29d5045ad3ea0 (127.15.255.135:34701), a62a935ff43d494ca35d0c417943133d (127.15.255.130:37991), 7d1e5c1b047443cbb28d7b0a9744babf (127.15.255.129:37897), 16b3a337bc5e462f83ba895ad8660deb (127.15.255.131:36033)
I20240502 21:28:51.746879 2729 raft_consensus.cc:482] T 20a7af42e83c4639bba20f22dee69f2a P a62a935ff43d494ca35d0c417943133d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:28:51.747064 2729 raft_consensus.cc:504] T 20a7af42e83c4639bba20f22dee69f2a P a62a935ff43d494ca35d0c417943133d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:28:51.748328 2729 leader_election.cc:288] T 20a7af42e83c4639bba20f22dee69f2a P a62a935ff43d494ca35d0c417943133d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132:44947), e632c553ff4341328acf1cbaca678392 (127.15.255.133:37439), 521bb93035324cecbebed2f6f2a630f8 (127.15.255.134:40821), 6ab8df3c796f4b6cb3e29d5045ad3ea0 (127.15.255.135:34701), 7d1e5c1b047443cbb28d7b0a9744babf (127.15.255.129:37897), 16b3a337bc5e462f83ba895ad8660deb (127.15.255.131:36033)
I20240502 21:28:51.748555 2088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20a7af42e83c4639bba20f22dee69f2a" candidate_uuid: "a62a935ff43d494ca35d0c417943133d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "521bb93035324cecbebed2f6f2a630f8" is_pre_election: true
I20240502 21:28:51.748595 1818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20a7af42e83c4639bba20f22dee69f2a" candidate_uuid: "a62a935ff43d494ca35d0c417943133d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "153026dd7f8943cc93fedf6f48450fb4" is_pre_election: true
I20240502 21:28:51.748811 1818 raft_consensus.cc:2454] T 20a7af42e83c4639bba20f22dee69f2a P 153026dd7f8943cc93fedf6f48450fb4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a62a935ff43d494ca35d0c417943133d in term 0.
W20240502 21:28:51.749145 1480 leader_election.cc:341] T 20a7af42e83c4639bba20f22dee69f2a P a62a935ff43d494ca35d0c417943133d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 521bb93035324cecbebed2f6f2a630f8 (127.15.255.134:40821): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:28:51.749187 1684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20a7af42e83c4639bba20f22dee69f2a" candidate_uuid: "a62a935ff43d494ca35d0c417943133d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16b3a337bc5e462f83ba895ad8660deb" is_pre_election: true
W20240502 21:28:51.749737 1483 leader_election.cc:341] T 20a7af42e83c4639bba20f22dee69f2a P a62a935ff43d494ca35d0c417943133d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 16b3a337bc5e462f83ba895ad8660deb (127.15.255.131:36033): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:28:51.750528 2924 tablet_bootstrap.cc:492] T 20a7af42e83c4639bba20f22dee69f2a P e632c553ff4341328acf1cbaca678392: No bootstrap required, opened a new log
I20240502 21:28:51.750659 1415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20a7af42e83c4639bba20f22dee69f2a" candidate_uuid: "a62a935ff43d494ca35d0c417943133d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" is_pre_election: true
I20240502 21:28:51.750732 2924 ts_tablet_manager.cc:1367] T 20a7af42e83c4639bba20f22dee69f2a P e632c553ff4341328acf1cbaca678392: Time spent bootstrapping tablet: real 0.059s user 0.007s sys 0.000s
I20240502 21:28:51.751291 2227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20a7af42e83c4639bba20f22dee69f2a" candidate_uuid: "a62a935ff43d494ca35d0c417943133d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" is_pre_election: true
I20240502 21:28:51.751447 2924 raft_consensus.cc:348] T 20a7af42e83c4639bba20f22dee69f2a P e632c553ff4341328acf1cbaca678392 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:28:51.751616 2926 ts_tablet_manager.cc:1398] T e44bf56d64b24e328803ebd0a3a54ceb P 7d1e5c1b047443cbb28d7b0a9744babf: Time spent starting tablet: real 0.011s user 0.003s sys 0.003s
I20240502 21:28:51.751701 2924 raft_consensus.cc:374] T 20a7af42e83c4639bba20f22dee69f2a P e632c553ff4341328acf1cbaca678392 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:51.751782 2924 raft_consensus.cc:729] T 20a7af42e83c4639bba20f22dee69f2a P e632c553ff4341328acf1cbaca678392 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e632c553ff4341328acf1cbaca678392, State: Initialized, Role: FOLLOWER
I20240502 21:28:51.751897 2926 tablet_bootstrap.cc:492] T 444c1cba2a6f4729aa01b75d06b6b517 P 7d1e5c1b047443cbb28d7b0a9744babf: Bootstrap starting.
W20240502 21:28:51.752182 1481 leader_election.cc:341] T 20a7af42e83c4639bba20f22dee69f2a P a62a935ff43d494ca35d0c417943133d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6ab8df3c796f4b6cb3e29d5045ad3ea0 (127.15.255.135:34701): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:28:51.752019 2924 consensus_queue.cc:260] T 20a7af42e83c4639bba20f22dee69f2a P e632c553ff4341328acf1cbaca678392 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
W20240502 21:28:51.752384 1481 leader_election.cc:341] T 20a7af42e83c4639bba20f22dee69f2a P a62a935ff43d494ca35d0c417943133d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7d1e5c1b047443cbb28d7b0a9744babf (127.15.255.129:37897): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:28:51.752848 1481 leader_election.cc:302] T 20a7af42e83c4639bba20f22dee69f2a P a62a935ff43d494ca35d0c417943133d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 5 responses out of 7 voters: 1 yes votes; 4 no votes. yes voters: a62a935ff43d494ca35d0c417943133d; no voters: 16b3a337bc5e462f83ba895ad8660deb, 521bb93035324cecbebed2f6f2a630f8, 6ab8df3c796f4b6cb3e29d5045ad3ea0, 7d1e5c1b047443cbb28d7b0a9744babf
I20240502 21:28:51.753157 2729 raft_consensus.cc:2726] T 20a7af42e83c4639bba20f22dee69f2a P a62a935ff43d494ca35d0c417943133d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 21:28:51.753505 2926 tablet_bootstrap.cc:654] T 444c1cba2a6f4729aa01b75d06b6b517 P 7d1e5c1b047443cbb28d7b0a9744babf: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:51.755504 1818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "444c1cba2a6f4729aa01b75d06b6b517" candidate_uuid: "e632c553ff4341328acf1cbaca678392" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "153026dd7f8943cc93fedf6f48450fb4" is_pre_election: true
I20240502 21:28:51.755715 1818 raft_consensus.cc:2454] T 444c1cba2a6f4729aa01b75d06b6b517 P 153026dd7f8943cc93fedf6f48450fb4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e632c553ff4341328acf1cbaca678392 in term 0.
I20240502 21:28:51.755761 2930 tablet_bootstrap.cc:492] T 20a7af42e83c4639bba20f22dee69f2a P 16b3a337bc5e462f83ba895ad8660deb: No bootstrap required, opened a new log
I20240502 21:28:51.755894 2930 ts_tablet_manager.cc:1367] T 20a7af42e83c4639bba20f22dee69f2a P 16b3a337bc5e462f83ba895ad8660deb: Time spent bootstrapping tablet: real 0.019s user 0.004s sys 0.003s
I20240502 21:28:51.755941 2919 tablet_bootstrap.cc:492] T 20a7af42e83c4639bba20f22dee69f2a P 521bb93035324cecbebed2f6f2a630f8: No bootstrap required, opened a new log
I20240502 21:28:51.756165 2919 ts_tablet_manager.cc:1367] T 20a7af42e83c4639bba20f22dee69f2a P 521bb93035324cecbebed2f6f2a630f8: Time spent bootstrapping tablet: real 0.022s user 0.016s sys 0.001s
I20240502 21:28:51.756662 2930 raft_consensus.cc:348] T 20a7af42e83c4639bba20f22dee69f2a P 16b3a337bc5e462f83ba895ad8660deb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:28:51.756925 2930 raft_consensus.cc:374] T 20a7af42e83c4639bba20f22dee69f2a P 16b3a337bc5e462f83ba895ad8660deb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:51.757025 2930 raft_consensus.cc:729] T 20a7af42e83c4639bba20f22dee69f2a P 16b3a337bc5e462f83ba895ad8660deb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16b3a337bc5e462f83ba895ad8660deb, State: Initialized, Role: FOLLOWER
I20240502 21:28:51.757025 2919 raft_consensus.cc:348] T 20a7af42e83c4639bba20f22dee69f2a P 521bb93035324cecbebed2f6f2a630f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:28:51.757385 2919 raft_consensus.cc:374] T 20a7af42e83c4639bba20f22dee69f2a P 521bb93035324cecbebed2f6f2a630f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:51.757252 2930 consensus_queue.cc:260] T 20a7af42e83c4639bba20f22dee69f2a P 16b3a337bc5e462f83ba895ad8660deb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:28:51.757516 2919 raft_consensus.cc:729] T 20a7af42e83c4639bba20f22dee69f2a P 521bb93035324cecbebed2f6f2a630f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 521bb93035324cecbebed2f6f2a630f8, State: Initialized, Role: FOLLOWER
I20240502 21:28:51.757896 2930 ts_tablet_manager.cc:1398] T 20a7af42e83c4639bba20f22dee69f2a P 16b3a337bc5e462f83ba895ad8660deb: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:28:51.758028 2919 consensus_queue.cc:260] T 20a7af42e83c4639bba20f22dee69f2a P 521bb93035324cecbebed2f6f2a630f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:28:51.759341 2919 ts_tablet_manager.cc:1398] T 20a7af42e83c4639bba20f22dee69f2a P 521bb93035324cecbebed2f6f2a630f8: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 21:28:51.760038 2924 ts_tablet_manager.cc:1398] T 20a7af42e83c4639bba20f22dee69f2a P e632c553ff4341328acf1cbaca678392: Time spent starting tablet: real 0.009s user 0.002s sys 0.001s
I20240502 21:28:51.760373 2924 tablet_bootstrap.cc:492] T e44bf56d64b24e328803ebd0a3a54ceb P e632c553ff4341328acf1cbaca678392: Bootstrap starting.
I20240502 21:28:51.760779 2926 tablet_bootstrap.cc:492] T 444c1cba2a6f4729aa01b75d06b6b517 P 7d1e5c1b047443cbb28d7b0a9744babf: No bootstrap required, opened a new log
I20240502 21:28:51.760994 2926 ts_tablet_manager.cc:1367] T 444c1cba2a6f4729aa01b75d06b6b517 P 7d1e5c1b047443cbb28d7b0a9744babf: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.001s
I20240502 21:28:51.761794 2926 raft_consensus.cc:348] T 444c1cba2a6f4729aa01b75d06b6b517 P 7d1e5c1b047443cbb28d7b0a9744babf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:28:51.761966 2924 tablet_bootstrap.cc:654] T e44bf56d64b24e328803ebd0a3a54ceb P e632c553ff4341328acf1cbaca678392: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:51.762118 2926 raft_consensus.cc:374] T 444c1cba2a6f4729aa01b75d06b6b517 P 7d1e5c1b047443cbb28d7b0a9744babf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:51.762243 2926 raft_consensus.cc:729] T 444c1cba2a6f4729aa01b75d06b6b517 P 7d1e5c1b047443cbb28d7b0a9744babf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d1e5c1b047443cbb28d7b0a9744babf, State: Initialized, Role: FOLLOWER
I20240502 21:28:51.763161 2926 consensus_queue.cc:260] T 444c1cba2a6f4729aa01b75d06b6b517 P 7d1e5c1b047443cbb28d7b0a9744babf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:28:51.763943 2926 ts_tablet_manager.cc:1398] T 444c1cba2a6f4729aa01b75d06b6b517 P 7d1e5c1b047443cbb28d7b0a9744babf: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240502 21:28:51.764369 2926 tablet_bootstrap.cc:492] T 20a7af42e83c4639bba20f22dee69f2a P 7d1e5c1b047443cbb28d7b0a9744babf: Bootstrap starting.
I20240502 21:28:51.766112 2926 tablet_bootstrap.cc:654] T 20a7af42e83c4639bba20f22dee69f2a P 7d1e5c1b047443cbb28d7b0a9744babf: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:51.768298 2924 tablet_bootstrap.cc:492] T e44bf56d64b24e328803ebd0a3a54ceb P e632c553ff4341328acf1cbaca678392: No bootstrap required, opened a new log
I20240502 21:28:51.768538 2924 ts_tablet_manager.cc:1367] T e44bf56d64b24e328803ebd0a3a54ceb P e632c553ff4341328acf1cbaca678392: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.007s
I20240502 21:28:51.770000 2924 raft_consensus.cc:348] T e44bf56d64b24e328803ebd0a3a54ceb P e632c553ff4341328acf1cbaca678392 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:28:51.770344 2924 raft_consensus.cc:374] T e44bf56d64b24e328803ebd0a3a54ceb P e632c553ff4341328acf1cbaca678392 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:51.770480 2924 raft_consensus.cc:729] T e44bf56d64b24e328803ebd0a3a54ceb P e632c553ff4341328acf1cbaca678392 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e632c553ff4341328acf1cbaca678392, State: Initialized, Role: FOLLOWER
I20240502 21:28:51.770790 2924 consensus_queue.cc:260] T e44bf56d64b24e328803ebd0a3a54ceb P e632c553ff4341328acf1cbaca678392 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:28:51.771601 2924 ts_tablet_manager.cc:1398] T e44bf56d64b24e328803ebd0a3a54ceb P e632c553ff4341328acf1cbaca678392: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 21:28:51.772668 2926 tablet_bootstrap.cc:492] T 20a7af42e83c4639bba20f22dee69f2a P 7d1e5c1b047443cbb28d7b0a9744babf: No bootstrap required, opened a new log
I20240502 21:28:51.772863 2926 ts_tablet_manager.cc:1367] T 20a7af42e83c4639bba20f22dee69f2a P 7d1e5c1b047443cbb28d7b0a9744babf: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.002s
I20240502 21:28:51.773612 2926 raft_consensus.cc:348] T 20a7af42e83c4639bba20f22dee69f2a P 7d1e5c1b047443cbb28d7b0a9744babf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:28:51.773715 2088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "444c1cba2a6f4729aa01b75d06b6b517" candidate_uuid: "e632c553ff4341328acf1cbaca678392" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "521bb93035324cecbebed2f6f2a630f8" is_pre_election: true
I20240502 21:28:51.773790 2918 tablet_bootstrap.cc:492] T 20a7af42e83c4639bba20f22dee69f2a P 6ab8df3c796f4b6cb3e29d5045ad3ea0: No bootstrap required, opened a new log
I20240502 21:28:51.773921 2926 raft_consensus.cc:374] T 20a7af42e83c4639bba20f22dee69f2a P 7d1e5c1b047443cbb28d7b0a9744babf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:51.773912 2088 raft_consensus.cc:2454] T 444c1cba2a6f4729aa01b75d06b6b517 P 521bb93035324cecbebed2f6f2a630f8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e632c553ff4341328acf1cbaca678392 in term 0.
I20240502 21:28:51.773948 2918 ts_tablet_manager.cc:1367] T 20a7af42e83c4639bba20f22dee69f2a P 6ab8df3c796f4b6cb3e29d5045ad3ea0: Time spent bootstrapping tablet: real 0.058s user 0.010s sys 0.004s
I20240502 21:28:51.774071 2926 raft_consensus.cc:729] T 20a7af42e83c4639bba20f22dee69f2a P 7d1e5c1b047443cbb28d7b0a9744babf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d1e5c1b047443cbb28d7b0a9744babf, State: Initialized, Role: FOLLOWER
I20240502 21:28:51.774379 2926 consensus_queue.cc:260] T 20a7af42e83c4639bba20f22dee69f2a P 7d1e5c1b047443cbb28d7b0a9744babf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:28:51.775239 2918 raft_consensus.cc:348] T 20a7af42e83c4639bba20f22dee69f2a P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:28:51.775542 2918 raft_consensus.cc:374] T 20a7af42e83c4639bba20f22dee69f2a P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:51.775640 2918 raft_consensus.cc:729] T 20a7af42e83c4639bba20f22dee69f2a P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ab8df3c796f4b6cb3e29d5045ad3ea0, State: Initialized, Role: FOLLOWER
I20240502 21:28:51.775938 2918 consensus_queue.cc:260] T 20a7af42e83c4639bba20f22dee69f2a P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:28:51.779011 2918 ts_tablet_manager.cc:1398] T 20a7af42e83c4639bba20f22dee69f2a P 6ab8df3c796f4b6cb3e29d5045ad3ea0: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20240502 21:28:51.780738 2926 ts_tablet_manager.cc:1398] T 20a7af42e83c4639bba20f22dee69f2a P 7d1e5c1b047443cbb28d7b0a9744babf: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20240502 21:28:51.785552 2227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "444c1cba2a6f4729aa01b75d06b6b517" candidate_uuid: "e632c553ff4341328acf1cbaca678392" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" is_pre_election: true
I20240502 21:28:51.785602 1953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20a7af42e83c4639bba20f22dee69f2a" candidate_uuid: "a62a935ff43d494ca35d0c417943133d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e632c553ff4341328acf1cbaca678392" is_pre_election: true
I20240502 21:28:51.785759 2227 raft_consensus.cc:2454] T 444c1cba2a6f4729aa01b75d06b6b517 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e632c553ff4341328acf1cbaca678392 in term 0.
I20240502 21:28:51.785794 1953 raft_consensus.cc:2454] T 20a7af42e83c4639bba20f22dee69f2a P e632c553ff4341328acf1cbaca678392 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a62a935ff43d494ca35d0c417943133d in term 0.
I20240502 21:28:51.786294 1885 leader_election.cc:302] T 444c1cba2a6f4729aa01b75d06b6b517 P e632c553ff4341328acf1cbaca678392 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 153026dd7f8943cc93fedf6f48450fb4, 521bb93035324cecbebed2f6f2a630f8, 6ab8df3c796f4b6cb3e29d5045ad3ea0, e632c553ff4341328acf1cbaca678392; no voters:
I20240502 21:28:51.786741 2935 raft_consensus.cc:2781] T 444c1cba2a6f4729aa01b75d06b6b517 P e632c553ff4341328acf1cbaca678392 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:28:51.786862 2935 raft_consensus.cc:482] T 444c1cba2a6f4729aa01b75d06b6b517 P e632c553ff4341328acf1cbaca678392 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:28:51.786958 2935 raft_consensus.cc:3037] T 444c1cba2a6f4729aa01b75d06b6b517 P e632c553ff4341328acf1cbaca678392 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:51.787328 1684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "444c1cba2a6f4729aa01b75d06b6b517" candidate_uuid: "e632c553ff4341328acf1cbaca678392" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16b3a337bc5e462f83ba895ad8660deb" is_pre_election: true
I20240502 21:28:51.787549 1684 raft_consensus.cc:2454] T 444c1cba2a6f4729aa01b75d06b6b517 P 16b3a337bc5e462f83ba895ad8660deb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e632c553ff4341328acf1cbaca678392 in term 0.
I20240502 21:28:51.789696 1415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "444c1cba2a6f4729aa01b75d06b6b517" candidate_uuid: "e632c553ff4341328acf1cbaca678392" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" is_pre_election: true
I20240502 21:28:51.789917 1415 raft_consensus.cc:2454] T 444c1cba2a6f4729aa01b75d06b6b517 P 7d1e5c1b047443cbb28d7b0a9744babf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e632c553ff4341328acf1cbaca678392 in term 0.
I20240502 21:28:51.789865 2935 raft_consensus.cc:504] T 444c1cba2a6f4729aa01b75d06b6b517 P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:28:51.791078 1818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "444c1cba2a6f4729aa01b75d06b6b517" candidate_uuid: "e632c553ff4341328acf1cbaca678392" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "153026dd7f8943cc93fedf6f48450fb4"
I20240502 21:28:51.791131 2088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "444c1cba2a6f4729aa01b75d06b6b517" candidate_uuid: "e632c553ff4341328acf1cbaca678392" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "521bb93035324cecbebed2f6f2a630f8"
I20240502 21:28:51.791281 1818 raft_consensus.cc:3037] T 444c1cba2a6f4729aa01b75d06b6b517 P 153026dd7f8943cc93fedf6f48450fb4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:51.791337 2088 raft_consensus.cc:3037] T 444c1cba2a6f4729aa01b75d06b6b517 P 521bb93035324cecbebed2f6f2a630f8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:51.792837 2935 leader_election.cc:288] T 444c1cba2a6f4729aa01b75d06b6b517 P e632c553ff4341328acf1cbaca678392 [CANDIDATE]: Term 1 election: Requested vote from peers 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132:44947), 521bb93035324cecbebed2f6f2a630f8 (127.15.255.134:40821), 6ab8df3c796f4b6cb3e29d5045ad3ea0 (127.15.255.135:34701), a62a935ff43d494ca35d0c417943133d (127.15.255.130:37991), 7d1e5c1b047443cbb28d7b0a9744babf (127.15.255.129:37897), 16b3a337bc5e462f83ba895ad8660deb (127.15.255.131:36033)
I20240502 21:28:51.793493 1818 raft_consensus.cc:2454] T 444c1cba2a6f4729aa01b75d06b6b517 P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e632c553ff4341328acf1cbaca678392 in term 1.
I20240502 21:28:51.793556 2227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "444c1cba2a6f4729aa01b75d06b6b517" candidate_uuid: "e632c553ff4341328acf1cbaca678392" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0"
I20240502 21:28:51.793711 2227 raft_consensus.cc:3037] T 444c1cba2a6f4729aa01b75d06b6b517 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:51.794020 2088 raft_consensus.cc:2454] T 444c1cba2a6f4729aa01b75d06b6b517 P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e632c553ff4341328acf1cbaca678392 in term 1.
I20240502 21:28:51.794538 1684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "444c1cba2a6f4729aa01b75d06b6b517" candidate_uuid: "e632c553ff4341328acf1cbaca678392" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16b3a337bc5e462f83ba895ad8660deb"
I20240502 21:28:51.794679 1684 raft_consensus.cc:3037] T 444c1cba2a6f4729aa01b75d06b6b517 P 16b3a337bc5e462f83ba895ad8660deb [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:51.795563 2227 raft_consensus.cc:2454] T 444c1cba2a6f4729aa01b75d06b6b517 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e632c553ff4341328acf1cbaca678392 in term 1.
I20240502 21:28:51.796154 1885 leader_election.cc:302] T 444c1cba2a6f4729aa01b75d06b6b517 P e632c553ff4341328acf1cbaca678392 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 153026dd7f8943cc93fedf6f48450fb4, 521bb93035324cecbebed2f6f2a630f8, 6ab8df3c796f4b6cb3e29d5045ad3ea0, e632c553ff4341328acf1cbaca678392; no voters:
I20240502 21:28:51.796315 1415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "444c1cba2a6f4729aa01b75d06b6b517" candidate_uuid: "e632c553ff4341328acf1cbaca678392" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d1e5c1b047443cbb28d7b0a9744babf"
I20240502 21:28:51.796484 2935 raft_consensus.cc:2781] T 444c1cba2a6f4729aa01b75d06b6b517 P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:28:51.796403 1549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "444c1cba2a6f4729aa01b75d06b6b517" candidate_uuid: "e632c553ff4341328acf1cbaca678392" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a62a935ff43d494ca35d0c417943133d" is_pre_election: true
I20240502 21:28:51.796519 1415 raft_consensus.cc:3037] T 444c1cba2a6f4729aa01b75d06b6b517 P 7d1e5c1b047443cbb28d7b0a9744babf [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:51.796640 1549 raft_consensus.cc:2454] T 444c1cba2a6f4729aa01b75d06b6b517 P a62a935ff43d494ca35d0c417943133d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e632c553ff4341328acf1cbaca678392 in term 0.
I20240502 21:28:51.796656 1684 raft_consensus.cc:2454] T 444c1cba2a6f4729aa01b75d06b6b517 P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e632c553ff4341328acf1cbaca678392 in term 1.
I20240502 21:28:51.797124 1549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "444c1cba2a6f4729aa01b75d06b6b517" candidate_uuid: "e632c553ff4341328acf1cbaca678392" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a62a935ff43d494ca35d0c417943133d"
I20240502 21:28:51.797276 1549 raft_consensus.cc:3037] T 444c1cba2a6f4729aa01b75d06b6b517 P a62a935ff43d494ca35d0c417943133d [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:51.799079 1549 raft_consensus.cc:2454] T 444c1cba2a6f4729aa01b75d06b6b517 P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e632c553ff4341328acf1cbaca678392 in term 1.
I20240502 21:28:51.799299 1415 raft_consensus.cc:2454] T 444c1cba2a6f4729aa01b75d06b6b517 P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e632c553ff4341328acf1cbaca678392 in term 1.
I20240502 21:28:51.799839 2935 raft_consensus.cc:686] T 444c1cba2a6f4729aa01b75d06b6b517 P e632c553ff4341328acf1cbaca678392 [term 1 LEADER]: Becoming Leader. State: Replica: e632c553ff4341328acf1cbaca678392, State: Running, Role: LEADER
I20240502 21:28:51.800688 2935 consensus_queue.cc:237] T 444c1cba2a6f4729aa01b75d06b6b517 P e632c553ff4341328acf1cbaca678392 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:28:51.804437 1268 catalog_manager.cc:5484] T 444c1cba2a6f4729aa01b75d06b6b517 P e632c553ff4341328acf1cbaca678392 reported cstate change: term changed from 0 to 1, leader changed from <none> to e632c553ff4341328acf1cbaca678392 (127.15.255.133). New cstate: current_term: 1 leader_uuid: "e632c553ff4341328acf1cbaca678392" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } health_report { overall_health: UNKNOWN } } }
I20240502 21:28:51.844825 2808 raft_consensus.cc:482] T e44bf56d64b24e328803ebd0a3a54ceb P 521bb93035324cecbebed2f6f2a630f8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:28:51.844971 2808 raft_consensus.cc:504] T e44bf56d64b24e328803ebd0a3a54ceb P 521bb93035324cecbebed2f6f2a630f8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:28:51.845984 2808 leader_election.cc:288] T e44bf56d64b24e328803ebd0a3a54ceb P 521bb93035324cecbebed2f6f2a630f8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132:44947), e632c553ff4341328acf1cbaca678392 (127.15.255.133:37439), 6ab8df3c796f4b6cb3e29d5045ad3ea0 (127.15.255.135:34701), a62a935ff43d494ca35d0c417943133d (127.15.255.130:37991), 7d1e5c1b047443cbb28d7b0a9744babf (127.15.255.129:37897), 16b3a337bc5e462f83ba895ad8660deb (127.15.255.131:36033)
I20240502 21:28:51.846041 1818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e44bf56d64b24e328803ebd0a3a54ceb" candidate_uuid: "521bb93035324cecbebed2f6f2a630f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "153026dd7f8943cc93fedf6f48450fb4" is_pre_election: true
I20240502 21:28:51.846124 1953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e44bf56d64b24e328803ebd0a3a54ceb" candidate_uuid: "521bb93035324cecbebed2f6f2a630f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e632c553ff4341328acf1cbaca678392" is_pre_election: true
I20240502 21:28:51.846251 1818 raft_consensus.cc:2454] T e44bf56d64b24e328803ebd0a3a54ceb P 153026dd7f8943cc93fedf6f48450fb4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 521bb93035324cecbebed2f6f2a630f8 in term 0.
I20240502 21:28:51.846350 1953 raft_consensus.cc:2454] T e44bf56d64b24e328803ebd0a3a54ceb P e632c553ff4341328acf1cbaca678392 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 521bb93035324cecbebed2f6f2a630f8 in term 0.
I20240502 21:28:51.846830 2227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e44bf56d64b24e328803ebd0a3a54ceb" candidate_uuid: "521bb93035324cecbebed2f6f2a630f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" is_pre_election: true
I20240502 21:28:51.846824 1549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e44bf56d64b24e328803ebd0a3a54ceb" candidate_uuid: "521bb93035324cecbebed2f6f2a630f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a62a935ff43d494ca35d0c417943133d" is_pre_election: true
I20240502 21:28:51.846908 1684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e44bf56d64b24e328803ebd0a3a54ceb" candidate_uuid: "521bb93035324cecbebed2f6f2a630f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16b3a337bc5e462f83ba895ad8660deb" is_pre_election: true
I20240502 21:28:51.847035 2227 raft_consensus.cc:2454] T e44bf56d64b24e328803ebd0a3a54ceb P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 521bb93035324cecbebed2f6f2a630f8 in term 0.
I20240502 21:28:51.847046 1549 raft_consensus.cc:2454] T e44bf56d64b24e328803ebd0a3a54ceb P a62a935ff43d494ca35d0c417943133d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 521bb93035324cecbebed2f6f2a630f8 in term 0.
I20240502 21:28:51.847126 1684 raft_consensus.cc:2454] T e44bf56d64b24e328803ebd0a3a54ceb P 16b3a337bc5e462f83ba895ad8660deb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 521bb93035324cecbebed2f6f2a630f8 in term 0.
I20240502 21:28:51.847469 2023 leader_election.cc:302] T e44bf56d64b24e328803ebd0a3a54ceb P 521bb93035324cecbebed2f6f2a630f8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 5 responses out of 7 voters: 5 yes votes; 0 no votes. yes voters: 153026dd7f8943cc93fedf6f48450fb4, 16b3a337bc5e462f83ba895ad8660deb, 521bb93035324cecbebed2f6f2a630f8, a62a935ff43d494ca35d0c417943133d, e632c553ff4341328acf1cbaca678392; no voters:
I20240502 21:28:51.847836 2808 raft_consensus.cc:2781] T e44bf56d64b24e328803ebd0a3a54ceb P 521bb93035324cecbebed2f6f2a630f8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:28:51.847815 1415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e44bf56d64b24e328803ebd0a3a54ceb" candidate_uuid: "521bb93035324cecbebed2f6f2a630f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" is_pre_election: true
I20240502 21:28:51.847950 2808 raft_consensus.cc:482] T e44bf56d64b24e328803ebd0a3a54ceb P 521bb93035324cecbebed2f6f2a630f8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:28:51.848037 2808 raft_consensus.cc:3037] T e44bf56d64b24e328803ebd0a3a54ceb P 521bb93035324cecbebed2f6f2a630f8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:51.848012 1415 raft_consensus.cc:2454] T e44bf56d64b24e328803ebd0a3a54ceb P 7d1e5c1b047443cbb28d7b0a9744babf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 521bb93035324cecbebed2f6f2a630f8 in term 0.
I20240502 21:28:51.850891 2808 raft_consensus.cc:504] T e44bf56d64b24e328803ebd0a3a54ceb P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:28:51.851981 1818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e44bf56d64b24e328803ebd0a3a54ceb" candidate_uuid: "521bb93035324cecbebed2f6f2a630f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "153026dd7f8943cc93fedf6f48450fb4"
I20240502 21:28:51.852084 1953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e44bf56d64b24e328803ebd0a3a54ceb" candidate_uuid: "521bb93035324cecbebed2f6f2a630f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e632c553ff4341328acf1cbaca678392"
I20240502 21:28:51.852172 1818 raft_consensus.cc:3037] T e44bf56d64b24e328803ebd0a3a54ceb P 153026dd7f8943cc93fedf6f48450fb4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:51.852249 1953 raft_consensus.cc:3037] T e44bf56d64b24e328803ebd0a3a54ceb P e632c553ff4341328acf1cbaca678392 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:51.852666 1684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e44bf56d64b24e328803ebd0a3a54ceb" candidate_uuid: "521bb93035324cecbebed2f6f2a630f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16b3a337bc5e462f83ba895ad8660deb"
I20240502 21:28:51.852860 1684 raft_consensus.cc:3037] T e44bf56d64b24e328803ebd0a3a54ceb P 16b3a337bc5e462f83ba895ad8660deb [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:51.854837 1818 raft_consensus.cc:2454] T e44bf56d64b24e328803ebd0a3a54ceb P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 521bb93035324cecbebed2f6f2a630f8 in term 1.
I20240502 21:28:51.854837 1953 raft_consensus.cc:2454] T e44bf56d64b24e328803ebd0a3a54ceb P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 521bb93035324cecbebed2f6f2a630f8 in term 1.
I20240502 21:28:51.855702 1684 raft_consensus.cc:2454] T e44bf56d64b24e328803ebd0a3a54ceb P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 521bb93035324cecbebed2f6f2a630f8 in term 1.
I20240502 21:28:51.855679 1549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e44bf56d64b24e328803ebd0a3a54ceb" candidate_uuid: "521bb93035324cecbebed2f6f2a630f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a62a935ff43d494ca35d0c417943133d"
I20240502 21:28:51.855737 2227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e44bf56d64b24e328803ebd0a3a54ceb" candidate_uuid: "521bb93035324cecbebed2f6f2a630f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0"
I20240502 21:28:51.855878 1549 raft_consensus.cc:3037] T e44bf56d64b24e328803ebd0a3a54ceb P a62a935ff43d494ca35d0c417943133d [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:51.855901 2227 raft_consensus.cc:3037] T e44bf56d64b24e328803ebd0a3a54ceb P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:51.856106 2025 leader_election.cc:302] T e44bf56d64b24e328803ebd0a3a54ceb P 521bb93035324cecbebed2f6f2a630f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 153026dd7f8943cc93fedf6f48450fb4, 16b3a337bc5e462f83ba895ad8660deb, 521bb93035324cecbebed2f6f2a630f8, e632c553ff4341328acf1cbaca678392; no voters:
I20240502 21:28:51.857805 2808 leader_election.cc:288] T e44bf56d64b24e328803ebd0a3a54ceb P 521bb93035324cecbebed2f6f2a630f8 [CANDIDATE]: Term 1 election: Requested vote from peers 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132:44947), e632c553ff4341328acf1cbaca678392 (127.15.255.133:37439), 6ab8df3c796f4b6cb3e29d5045ad3ea0 (127.15.255.135:34701), a62a935ff43d494ca35d0c417943133d (127.15.255.130:37991), 7d1e5c1b047443cbb28d7b0a9744babf (127.15.255.129:37897), 16b3a337bc5e462f83ba895ad8660deb (127.15.255.131:36033)
I20240502 21:28:51.857995 2808 raft_consensus.cc:2781] T e44bf56d64b24e328803ebd0a3a54ceb P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:28:51.858122 2808 raft_consensus.cc:686] T e44bf56d64b24e328803ebd0a3a54ceb P 521bb93035324cecbebed2f6f2a630f8 [term 1 LEADER]: Becoming Leader. State: Replica: 521bb93035324cecbebed2f6f2a630f8, State: Running, Role: LEADER
I20240502 21:28:51.858541 2227 raft_consensus.cc:2454] T e44bf56d64b24e328803ebd0a3a54ceb P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 521bb93035324cecbebed2f6f2a630f8 in term 1.
I20240502 21:28:51.858448 2808 consensus_queue.cc:237] T e44bf56d64b24e328803ebd0a3a54ceb P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:28:51.858966 1549 raft_consensus.cc:2454] T e44bf56d64b24e328803ebd0a3a54ceb P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 521bb93035324cecbebed2f6f2a630f8 in term 1.
I20240502 21:28:51.859221 1415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e44bf56d64b24e328803ebd0a3a54ceb" candidate_uuid: "521bb93035324cecbebed2f6f2a630f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d1e5c1b047443cbb28d7b0a9744babf"
I20240502 21:28:51.859441 1415 raft_consensus.cc:3037] T e44bf56d64b24e328803ebd0a3a54ceb P 7d1e5c1b047443cbb28d7b0a9744babf [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:51.861383 1415 raft_consensus.cc:2454] T e44bf56d64b24e328803ebd0a3a54ceb P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 521bb93035324cecbebed2f6f2a630f8 in term 1.
I20240502 21:28:51.863819 1268 catalog_manager.cc:5484] T e44bf56d64b24e328803ebd0a3a54ceb P 521bb93035324cecbebed2f6f2a630f8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 521bb93035324cecbebed2f6f2a630f8 (127.15.255.134). New cstate: current_term: 1 leader_uuid: "521bb93035324cecbebed2f6f2a630f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } health_report { overall_health: UNKNOWN } } }
I20240502 21:28:51.875720 2955 raft_consensus.cc:482] T 20a7af42e83c4639bba20f22dee69f2a P 7d1e5c1b047443cbb28d7b0a9744babf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:28:51.875859 2955 raft_consensus.cc:504] T 20a7af42e83c4639bba20f22dee69f2a P 7d1e5c1b047443cbb28d7b0a9744babf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:28:51.876917 1818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20a7af42e83c4639bba20f22dee69f2a" candidate_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "153026dd7f8943cc93fedf6f48450fb4" is_pre_election: true
I20240502 21:28:51.876978 2955 leader_election.cc:288] T 20a7af42e83c4639bba20f22dee69f2a P 7d1e5c1b047443cbb28d7b0a9744babf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132:44947), e632c553ff4341328acf1cbaca678392 (127.15.255.133:37439), 521bb93035324cecbebed2f6f2a630f8 (127.15.255.134:40821), 6ab8df3c796f4b6cb3e29d5045ad3ea0 (127.15.255.135:34701), a62a935ff43d494ca35d0c417943133d (127.15.255.130:37991), 16b3a337bc5e462f83ba895ad8660deb (127.15.255.131:36033)
I20240502 21:28:51.877120 1818 raft_consensus.cc:2454] T 20a7af42e83c4639bba20f22dee69f2a P 153026dd7f8943cc93fedf6f48450fb4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d1e5c1b047443cbb28d7b0a9744babf in term 0.
I20240502 21:28:51.877357 2227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20a7af42e83c4639bba20f22dee69f2a" candidate_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" is_pre_election: true
I20240502 21:28:51.877547 2227 raft_consensus.cc:2454] T 20a7af42e83c4639bba20f22dee69f2a P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d1e5c1b047443cbb28d7b0a9744babf in term 0.
I20240502 21:28:51.877588 1953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20a7af42e83c4639bba20f22dee69f2a" candidate_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e632c553ff4341328acf1cbaca678392" is_pre_election: true
I20240502 21:28:51.877622 1684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20a7af42e83c4639bba20f22dee69f2a" candidate_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16b3a337bc5e462f83ba895ad8660deb" is_pre_election: true
I20240502 21:28:51.877789 1953 raft_consensus.cc:2454] T 20a7af42e83c4639bba20f22dee69f2a P e632c553ff4341328acf1cbaca678392 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d1e5c1b047443cbb28d7b0a9744babf in term 0.
I20240502 21:28:51.877839 1684 raft_consensus.cc:2454] T 20a7af42e83c4639bba20f22dee69f2a P 16b3a337bc5e462f83ba895ad8660deb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d1e5c1b047443cbb28d7b0a9744babf in term 0.
I20240502 21:28:51.877811 1549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20a7af42e83c4639bba20f22dee69f2a" candidate_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a62a935ff43d494ca35d0c417943133d" is_pre_election: true
I20240502 21:28:51.878154 1347 leader_election.cc:302] T 20a7af42e83c4639bba20f22dee69f2a P 7d1e5c1b047443cbb28d7b0a9744babf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 153026dd7f8943cc93fedf6f48450fb4, 6ab8df3c796f4b6cb3e29d5045ad3ea0, 7d1e5c1b047443cbb28d7b0a9744babf, e632c553ff4341328acf1cbaca678392; no voters:
I20240502 21:28:51.878163 2088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20a7af42e83c4639bba20f22dee69f2a" candidate_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "521bb93035324cecbebed2f6f2a630f8" is_pre_election: true
I20240502 21:28:51.878325 1549 raft_consensus.cc:2454] T 20a7af42e83c4639bba20f22dee69f2a P a62a935ff43d494ca35d0c417943133d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d1e5c1b047443cbb28d7b0a9744babf in term 0.
I20240502 21:28:51.878343 2088 raft_consensus.cc:2454] T 20a7af42e83c4639bba20f22dee69f2a P 521bb93035324cecbebed2f6f2a630f8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d1e5c1b047443cbb28d7b0a9744babf in term 0.
I20240502 21:28:51.878432 2955 raft_consensus.cc:2781] T 20a7af42e83c4639bba20f22dee69f2a P 7d1e5c1b047443cbb28d7b0a9744babf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:28:51.878538 2955 raft_consensus.cc:482] T 20a7af42e83c4639bba20f22dee69f2a P 7d1e5c1b047443cbb28d7b0a9744babf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:28:51.878643 2955 raft_consensus.cc:3037] T 20a7af42e83c4639bba20f22dee69f2a P 7d1e5c1b047443cbb28d7b0a9744babf [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:51.880450 2955 raft_consensus.cc:504] T 20a7af42e83c4639bba20f22dee69f2a P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:28:51.881505 1953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20a7af42e83c4639bba20f22dee69f2a" candidate_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e632c553ff4341328acf1cbaca678392"
I20240502 21:28:51.881614 1818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20a7af42e83c4639bba20f22dee69f2a" candidate_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "153026dd7f8943cc93fedf6f48450fb4"
I20240502 21:28:51.881675 1953 raft_consensus.cc:3037] T 20a7af42e83c4639bba20f22dee69f2a P e632c553ff4341328acf1cbaca678392 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:51.881765 1818 raft_consensus.cc:3037] T 20a7af42e83c4639bba20f22dee69f2a P 153026dd7f8943cc93fedf6f48450fb4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:51.881805 2227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20a7af42e83c4639bba20f22dee69f2a" candidate_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0"
I20240502 21:28:51.881899 1684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20a7af42e83c4639bba20f22dee69f2a" candidate_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16b3a337bc5e462f83ba895ad8660deb"
I20240502 21:28:51.881966 2227 raft_consensus.cc:3037] T 20a7af42e83c4639bba20f22dee69f2a P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:51.882054 1684 raft_consensus.cc:3037] T 20a7af42e83c4639bba20f22dee69f2a P 16b3a337bc5e462f83ba895ad8660deb [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:51.884248 1818 raft_consensus.cc:2454] T 20a7af42e83c4639bba20f22dee69f2a P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d1e5c1b047443cbb28d7b0a9744babf in term 1.
I20240502 21:28:51.884564 2227 raft_consensus.cc:2454] T 20a7af42e83c4639bba20f22dee69f2a P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d1e5c1b047443cbb28d7b0a9744babf in term 1.
I20240502 21:28:51.884821 2088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20a7af42e83c4639bba20f22dee69f2a" candidate_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "521bb93035324cecbebed2f6f2a630f8"
I20240502 21:28:51.884853 1549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20a7af42e83c4639bba20f22dee69f2a" candidate_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a62a935ff43d494ca35d0c417943133d"
I20240502 21:28:51.884989 2955 leader_election.cc:288] T 20a7af42e83c4639bba20f22dee69f2a P 7d1e5c1b047443cbb28d7b0a9744babf [CANDIDATE]: Term 1 election: Requested vote from peers 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132:44947), e632c553ff4341328acf1cbaca678392 (127.15.255.133:37439), 521bb93035324cecbebed2f6f2a630f8 (127.15.255.134:40821), 6ab8df3c796f4b6cb3e29d5045ad3ea0 (127.15.255.135:34701), a62a935ff43d494ca35d0c417943133d (127.15.255.130:37991), 16b3a337bc5e462f83ba895ad8660deb (127.15.255.131:36033)
I20240502 21:28:51.885023 2088 raft_consensus.cc:3037] T 20a7af42e83c4639bba20f22dee69f2a P 521bb93035324cecbebed2f6f2a630f8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:51.885051 1549 raft_consensus.cc:3037] T 20a7af42e83c4639bba20f22dee69f2a P a62a935ff43d494ca35d0c417943133d [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:51.885373 1684 raft_consensus.cc:2454] T 20a7af42e83c4639bba20f22dee69f2a P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d1e5c1b047443cbb28d7b0a9744babf in term 1.
I20240502 21:28:51.885793 1350 leader_election.cc:302] T 20a7af42e83c4639bba20f22dee69f2a P 7d1e5c1b047443cbb28d7b0a9744babf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 153026dd7f8943cc93fedf6f48450fb4, 16b3a337bc5e462f83ba895ad8660deb, 6ab8df3c796f4b6cb3e29d5045ad3ea0, 7d1e5c1b047443cbb28d7b0a9744babf; no voters:
I20240502 21:28:51.886091 2955 raft_consensus.cc:2781] T 20a7af42e83c4639bba20f22dee69f2a P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:28:51.887653 2088 raft_consensus.cc:2454] T 20a7af42e83c4639bba20f22dee69f2a P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d1e5c1b047443cbb28d7b0a9744babf in term 1.
I20240502 21:28:51.887722 1549 raft_consensus.cc:2454] T 20a7af42e83c4639bba20f22dee69f2a P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d1e5c1b047443cbb28d7b0a9744babf in term 1.
I20240502 21:28:51.888620 2955 raft_consensus.cc:686] T 20a7af42e83c4639bba20f22dee69f2a P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 LEADER]: Becoming Leader. State: Replica: 7d1e5c1b047443cbb28d7b0a9744babf, State: Running, Role: LEADER
I20240502 21:28:51.889133 2955 consensus_queue.cc:237] T 20a7af42e83c4639bba20f22dee69f2a P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:28:51.889593 1953 raft_consensus.cc:2454] T 20a7af42e83c4639bba20f22dee69f2a P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d1e5c1b047443cbb28d7b0a9744babf in term 1.
I20240502 21:28:51.893218 1268 catalog_manager.cc:5484] T 20a7af42e83c4639bba20f22dee69f2a P 7d1e5c1b047443cbb28d7b0a9744babf reported cstate change: term changed from 0 to 1, leader changed from <none> to 7d1e5c1b047443cbb28d7b0a9744babf (127.15.255.129). New cstate: current_term: 1 leader_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } health_report { overall_health: UNKNOWN } } }
I20240502 21:28:52.180351 2935 consensus_queue.cc:1035] T 444c1cba2a6f4729aa01b75d06b6b517 P e632c553ff4341328acf1cbaca678392 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:28:52.187216 2935 consensus_queue.cc:1035] T 444c1cba2a6f4729aa01b75d06b6b517 P e632c553ff4341328acf1cbaca678392 [LEADER]: Connected to new peer: Peer: permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:28:52.224789 2935 consensus_queue.cc:1035] T 444c1cba2a6f4729aa01b75d06b6b517 P e632c553ff4341328acf1cbaca678392 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:28:52.231774 2935 consensus_queue.cc:1035] T 444c1cba2a6f4729aa01b75d06b6b517 P e632c553ff4341328acf1cbaca678392 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:28:52.232265 2935 consensus_queue.cc:1035] T 444c1cba2a6f4729aa01b75d06b6b517 P e632c553ff4341328acf1cbaca678392 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:28:52.239624 2808 consensus_queue.cc:1035] T e44bf56d64b24e328803ebd0a3a54ceb P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:28:52.266481 2973 consensus_queue.cc:1035] T 444c1cba2a6f4729aa01b75d06b6b517 P e632c553ff4341328acf1cbaca678392 [LEADER]: Connected to new peer: Peer: permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 21:28:52.300449 2967 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:28:52.300736 2967 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240502 21:28:52.300765 2955 consensus_queue.cc:1035] T 20a7af42e83c4639bba20f22dee69f2a P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 21:28:52.308583 2967 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 21:28:52.312364 2808 consensus_queue.cc:1035] T e44bf56d64b24e328803ebd0a3a54ceb P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:28:52.320817 2808 consensus_queue.cc:1035] T e44bf56d64b24e328803ebd0a3a54ceb P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:28:52.324111 2963 consensus_queue.cc:1035] T e44bf56d64b24e328803ebd0a3a54ceb P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:28:52.363713 3004 consensus_queue.cc:1035] T e44bf56d64b24e328803ebd0a3a54ceb P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240502 21:28:52.368067 2808 consensus_queue.cc:1035] T e44bf56d64b24e328803ebd0a3a54ceb P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_1'
I20240502 21:28:52.387856 2955 consensus_queue.cc:1035] T 20a7af42e83c4639bba20f22dee69f2a P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:28:52.417482 2955 consensus_queue.cc:1035] T 20a7af42e83c4639bba20f22dee69f2a P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:28:52.423169 2955 consensus_queue.cc:1035] T 20a7af42e83c4639bba20f22dee69f2a P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:28:52.461621 3026 consensus_queue.cc:1035] T 20a7af42e83c4639bba20f22dee69f2a P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:28:52.462318 3026 consensus_queue.cc:1035] T 20a7af42e83c4639bba20f22dee69f2a P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 414.728 ms
time per row: 0.207364 ms
I20240502 21:28:52.919771 1261 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44934:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 7
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
W20240502 21:28:52.920572 1261 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table rebalance_test_table_2 in case of a server failure: 8 tablet servers would be needed, 7 are available. Consider bringing up more tablet servers.
I20240502 21:28:52.929044 1526 tablet_service.cc:1467] Processing CreateTablet for tablet 1aa57620996e42b6adf941e9d606f7e2 (DEFAULT_TABLE table=rebalance_test_table_2 [id=900316782c71465f9538d1bbe31708d3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:52.929323 1394 tablet_service.cc:1467] Processing CreateTablet for tablet 1aa57620996e42b6adf941e9d606f7e2 (DEFAULT_TABLE table=rebalance_test_table_2 [id=900316782c71465f9538d1bbe31708d3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:52.929527 1526 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1aa57620996e42b6adf941e9d606f7e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:52.929543 1664 tablet_service.cc:1467] Processing CreateTablet for tablet 1aa57620996e42b6adf941e9d606f7e2 (DEFAULT_TABLE table=rebalance_test_table_2 [id=900316782c71465f9538d1bbe31708d3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:52.929718 1394 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1aa57620996e42b6adf941e9d606f7e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:52.929895 1664 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1aa57620996e42b6adf941e9d606f7e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:52.934931 3074 tablet_bootstrap.cc:492] T 1aa57620996e42b6adf941e9d606f7e2 P 16b3a337bc5e462f83ba895ad8660deb: Bootstrap starting.
I20240502 21:28:52.935240 2067 tablet_service.cc:1467] Processing CreateTablet for tablet 1aa57620996e42b6adf941e9d606f7e2 (DEFAULT_TABLE table=rebalance_test_table_2 [id=900316782c71465f9538d1bbe31708d3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:52.935672 2067 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1aa57620996e42b6adf941e9d606f7e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:52.936619 3074 tablet_bootstrap.cc:654] T 1aa57620996e42b6adf941e9d606f7e2 P 16b3a337bc5e462f83ba895ad8660deb: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:52.941849 3076 tablet_bootstrap.cc:492] T 1aa57620996e42b6adf941e9d606f7e2 P 7d1e5c1b047443cbb28d7b0a9744babf: Bootstrap starting.
I20240502 21:28:52.942188 2207 tablet_service.cc:1467] Processing CreateTablet for tablet 1aa57620996e42b6adf941e9d606f7e2 (DEFAULT_TABLE table=rebalance_test_table_2 [id=900316782c71465f9538d1bbe31708d3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:52.942358 1800 tablet_service.cc:1467] Processing CreateTablet for tablet 1aa57620996e42b6adf941e9d606f7e2 (DEFAULT_TABLE table=rebalance_test_table_2 [id=900316782c71465f9538d1bbe31708d3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:52.942688 2207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1aa57620996e42b6adf941e9d606f7e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:52.942781 1800 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1aa57620996e42b6adf941e9d606f7e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:52.944042 3076 tablet_bootstrap.cc:654] T 1aa57620996e42b6adf941e9d606f7e2 P 7d1e5c1b047443cbb28d7b0a9744babf: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:52.946964 1393 tablet_service.cc:1467] Processing CreateTablet for tablet b300ad3948074c728a25f053512bc37c (DEFAULT_TABLE table=rebalance_test_table_2 [id=900316782c71465f9538d1bbe31708d3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:52.947409 1393 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b300ad3948074c728a25f053512bc37c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:52.948607 1933 tablet_service.cc:1467] Processing CreateTablet for tablet 1aa57620996e42b6adf941e9d606f7e2 (DEFAULT_TABLE table=rebalance_test_table_2 [id=900316782c71465f9538d1bbe31708d3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:52.949075 1933 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1aa57620996e42b6adf941e9d606f7e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:52.953702 1528 tablet_service.cc:1467] Processing CreateTablet for tablet b300ad3948074c728a25f053512bc37c (DEFAULT_TABLE table=rebalance_test_table_2 [id=900316782c71465f9538d1bbe31708d3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:52.954118 1528 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b300ad3948074c728a25f053512bc37c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:52.954250 1662 tablet_service.cc:1467] Processing CreateTablet for tablet b300ad3948074c728a25f053512bc37c (DEFAULT_TABLE table=rebalance_test_table_2 [id=900316782c71465f9538d1bbe31708d3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:52.954718 1662 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b300ad3948074c728a25f053512bc37c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:52.957347 2069 tablet_service.cc:1467] Processing CreateTablet for tablet b300ad3948074c728a25f053512bc37c (DEFAULT_TABLE table=rebalance_test_table_2 [id=900316782c71465f9538d1bbe31708d3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:52.957763 2069 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b300ad3948074c728a25f053512bc37c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:52.961593 1798 tablet_service.cc:1467] Processing CreateTablet for tablet b300ad3948074c728a25f053512bc37c (DEFAULT_TABLE table=rebalance_test_table_2 [id=900316782c71465f9538d1bbe31708d3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:52.961814 1932 tablet_service.cc:1467] Processing CreateTablet for tablet b300ad3948074c728a25f053512bc37c (DEFAULT_TABLE table=rebalance_test_table_2 [id=900316782c71465f9538d1bbe31708d3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:52.961985 1798 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b300ad3948074c728a25f053512bc37c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:52.968258 1932 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b300ad3948074c728a25f053512bc37c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:52.981281 3079 tablet_bootstrap.cc:492] T 1aa57620996e42b6adf941e9d606f7e2 P e632c553ff4341328acf1cbaca678392: Bootstrap starting.
I20240502 21:28:52.983541 3080 tablet_bootstrap.cc:492] T 1aa57620996e42b6adf941e9d606f7e2 P 153026dd7f8943cc93fedf6f48450fb4: Bootstrap starting.
I20240502 21:28:52.986093 3080 tablet_bootstrap.cc:654] T 1aa57620996e42b6adf941e9d606f7e2 P 153026dd7f8943cc93fedf6f48450fb4: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:52.987152 3079 tablet_bootstrap.cc:654] T 1aa57620996e42b6adf941e9d606f7e2 P e632c553ff4341328acf1cbaca678392: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:52.990059 3075 tablet_bootstrap.cc:492] T 1aa57620996e42b6adf941e9d606f7e2 P a62a935ff43d494ca35d0c417943133d: Bootstrap starting.
I20240502 21:28:52.991969 2206 tablet_service.cc:1467] Processing CreateTablet for tablet b300ad3948074c728a25f053512bc37c (DEFAULT_TABLE table=rebalance_test_table_2 [id=900316782c71465f9538d1bbe31708d3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:52.992384 2206 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b300ad3948074c728a25f053512bc37c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:52.993629 2068 tablet_service.cc:1467] Processing CreateTablet for tablet 0658ecbf32754656876452b6f021926b (DEFAULT_TABLE table=rebalance_test_table_2 [id=900316782c71465f9538d1bbe31708d3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:52.994000 2068 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0658ecbf32754656876452b6f021926b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:52.995075 1526 tablet_service.cc:1467] Processing CreateTablet for tablet 0658ecbf32754656876452b6f021926b (DEFAULT_TABLE table=rebalance_test_table_2 [id=900316782c71465f9538d1bbe31708d3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:52.995460 1526 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0658ecbf32754656876452b6f021926b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:52.995748 2205 tablet_service.cc:1467] Processing CreateTablet for tablet 0658ecbf32754656876452b6f021926b (DEFAULT_TABLE table=rebalance_test_table_2 [id=900316782c71465f9538d1bbe31708d3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:52.996228 2205 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0658ecbf32754656876452b6f021926b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:53.001567 3080 tablet_bootstrap.cc:492] T 1aa57620996e42b6adf941e9d606f7e2 P 153026dd7f8943cc93fedf6f48450fb4: No bootstrap required, opened a new log
I20240502 21:28:53.003794 1933 tablet_service.cc:1467] Processing CreateTablet for tablet 0658ecbf32754656876452b6f021926b (DEFAULT_TABLE table=rebalance_test_table_2 [id=900316782c71465f9538d1bbe31708d3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:53.004261 1933 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0658ecbf32754656876452b6f021926b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:53.012610 3080 ts_tablet_manager.cc:1367] T 1aa57620996e42b6adf941e9d606f7e2 P 153026dd7f8943cc93fedf6f48450fb4: Time spent bootstrapping tablet: real 0.030s user 0.004s sys 0.000s
I20240502 21:28:53.015244 3075 tablet_bootstrap.cc:654] T 1aa57620996e42b6adf941e9d606f7e2 P a62a935ff43d494ca35d0c417943133d: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:53.016209 1393 tablet_service.cc:1467] Processing CreateTablet for tablet 0658ecbf32754656876452b6f021926b (DEFAULT_TABLE table=rebalance_test_table_2 [id=900316782c71465f9538d1bbe31708d3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:53.016186 3080 raft_consensus.cc:348] T 1aa57620996e42b6adf941e9d606f7e2 P 153026dd7f8943cc93fedf6f48450fb4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } }
I20240502 21:28:53.017140 1393 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0658ecbf32754656876452b6f021926b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:53.012883 1798 tablet_service.cc:1467] Processing CreateTablet for tablet 0658ecbf32754656876452b6f021926b (DEFAULT_TABLE table=rebalance_test_table_2 [id=900316782c71465f9538d1bbe31708d3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:53.020867 1798 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0658ecbf32754656876452b6f021926b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:53.024834 1662 tablet_service.cc:1467] Processing CreateTablet for tablet 0658ecbf32754656876452b6f021926b (DEFAULT_TABLE table=rebalance_test_table_2 [id=900316782c71465f9538d1bbe31708d3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:53.025375 1662 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0658ecbf32754656876452b6f021926b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:53.026921 3080 raft_consensus.cc:374] T 1aa57620996e42b6adf941e9d606f7e2 P 153026dd7f8943cc93fedf6f48450fb4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:53.027098 3080 raft_consensus.cc:729] T 1aa57620996e42b6adf941e9d606f7e2 P 153026dd7f8943cc93fedf6f48450fb4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 153026dd7f8943cc93fedf6f48450fb4, State: Initialized, Role: FOLLOWER
I20240502 21:28:53.028187 3080 consensus_queue.cc:260] T 1aa57620996e42b6adf941e9d606f7e2 P 153026dd7f8943cc93fedf6f48450fb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } }
I20240502 21:28:53.030226 3080 ts_tablet_manager.cc:1398] T 1aa57620996e42b6adf941e9d606f7e2 P 153026dd7f8943cc93fedf6f48450fb4: Time spent starting tablet: real 0.015s user 0.002s sys 0.000s
I20240502 21:28:53.031965 3080 tablet_bootstrap.cc:492] T b300ad3948074c728a25f053512bc37c P 153026dd7f8943cc93fedf6f48450fb4: Bootstrap starting.
I20240502 21:28:53.032616 3075 tablet_bootstrap.cc:492] T 1aa57620996e42b6adf941e9d606f7e2 P a62a935ff43d494ca35d0c417943133d: No bootstrap required, opened a new log
I20240502 21:28:53.033330 3079 tablet_bootstrap.cc:492] T 1aa57620996e42b6adf941e9d606f7e2 P e632c553ff4341328acf1cbaca678392: No bootstrap required, opened a new log
I20240502 21:28:53.033533 3079 ts_tablet_manager.cc:1367] T 1aa57620996e42b6adf941e9d606f7e2 P e632c553ff4341328acf1cbaca678392: Time spent bootstrapping tablet: real 0.052s user 0.000s sys 0.004s
I20240502 21:28:53.034425 3079 raft_consensus.cc:348] T 1aa57620996e42b6adf941e9d606f7e2 P e632c553ff4341328acf1cbaca678392 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } }
I20240502 21:28:53.034799 3079 raft_consensus.cc:374] T 1aa57620996e42b6adf941e9d606f7e2 P e632c553ff4341328acf1cbaca678392 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:53.034926 3079 raft_consensus.cc:729] T 1aa57620996e42b6adf941e9d606f7e2 P e632c553ff4341328acf1cbaca678392 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e632c553ff4341328acf1cbaca678392, State: Initialized, Role: FOLLOWER
I20240502 21:28:53.035250 3079 consensus_queue.cc:260] T 1aa57620996e42b6adf941e9d606f7e2 P e632c553ff4341328acf1cbaca678392 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } }
I20240502 21:28:53.036103 3079 ts_tablet_manager.cc:1398] T 1aa57620996e42b6adf941e9d606f7e2 P e632c553ff4341328acf1cbaca678392: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 21:28:53.036478 3079 tablet_bootstrap.cc:492] T b300ad3948074c728a25f053512bc37c P e632c553ff4341328acf1cbaca678392: Bootstrap starting.
I20240502 21:28:53.037662 3075 ts_tablet_manager.cc:1367] T 1aa57620996e42b6adf941e9d606f7e2 P a62a935ff43d494ca35d0c417943133d: Time spent bootstrapping tablet: real 0.048s user 0.002s sys 0.002s
I20240502 21:28:53.039148 3080 tablet_bootstrap.cc:654] T b300ad3948074c728a25f053512bc37c P 153026dd7f8943cc93fedf6f48450fb4: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:53.039963 3075 raft_consensus.cc:348] T 1aa57620996e42b6adf941e9d606f7e2 P a62a935ff43d494ca35d0c417943133d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } }
I20240502 21:28:53.041298 3079 tablet_bootstrap.cc:654] T b300ad3948074c728a25f053512bc37c P e632c553ff4341328acf1cbaca678392: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:53.042157 3075 raft_consensus.cc:374] T 1aa57620996e42b6adf941e9d606f7e2 P a62a935ff43d494ca35d0c417943133d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:53.045509 3082 tablet_bootstrap.cc:492] T 1aa57620996e42b6adf941e9d606f7e2 P 6ab8df3c796f4b6cb3e29d5045ad3ea0: Bootstrap starting.
I20240502 21:28:53.047354 3082 tablet_bootstrap.cc:654] T 1aa57620996e42b6adf941e9d606f7e2 P 6ab8df3c796f4b6cb3e29d5045ad3ea0: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:53.048492 3075 raft_consensus.cc:729] T 1aa57620996e42b6adf941e9d606f7e2 P a62a935ff43d494ca35d0c417943133d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a62a935ff43d494ca35d0c417943133d, State: Initialized, Role: FOLLOWER
I20240502 21:28:53.048746 3081 tablet_bootstrap.cc:492] T b300ad3948074c728a25f053512bc37c P 521bb93035324cecbebed2f6f2a630f8: Bootstrap starting.
I20240502 21:28:53.049857 3075 consensus_queue.cc:260] T 1aa57620996e42b6adf941e9d606f7e2 P a62a935ff43d494ca35d0c417943133d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } }
I20240502 21:28:53.050335 3081 tablet_bootstrap.cc:654] T b300ad3948074c728a25f053512bc37c P 521bb93035324cecbebed2f6f2a630f8: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:53.052570 3075 ts_tablet_manager.cc:1398] T 1aa57620996e42b6adf941e9d606f7e2 P a62a935ff43d494ca35d0c417943133d: Time spent starting tablet: real 0.013s user 0.001s sys 0.001s
I20240502 21:28:53.053685 3075 tablet_bootstrap.cc:492] T b300ad3948074c728a25f053512bc37c P a62a935ff43d494ca35d0c417943133d: Bootstrap starting.
I20240502 21:28:53.056329 3075 tablet_bootstrap.cc:654] T b300ad3948074c728a25f053512bc37c P a62a935ff43d494ca35d0c417943133d: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:53.058306 3080 tablet_bootstrap.cc:492] T b300ad3948074c728a25f053512bc37c P 153026dd7f8943cc93fedf6f48450fb4: No bootstrap required, opened a new log
I20240502 21:28:53.058514 3080 ts_tablet_manager.cc:1367] T b300ad3948074c728a25f053512bc37c P 153026dd7f8943cc93fedf6f48450fb4: Time spent bootstrapping tablet: real 0.027s user 0.006s sys 0.000s
I20240502 21:28:53.059278 3080 raft_consensus.cc:348] T b300ad3948074c728a25f053512bc37c P 153026dd7f8943cc93fedf6f48450fb4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } }
I20240502 21:28:53.059605 3080 raft_consensus.cc:374] T b300ad3948074c728a25f053512bc37c P 153026dd7f8943cc93fedf6f48450fb4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:53.059705 3080 raft_consensus.cc:729] T b300ad3948074c728a25f053512bc37c P 153026dd7f8943cc93fedf6f48450fb4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 153026dd7f8943cc93fedf6f48450fb4, State: Initialized, Role: FOLLOWER
I20240502 21:28:53.060178 3081 tablet_bootstrap.cc:492] T b300ad3948074c728a25f053512bc37c P 521bb93035324cecbebed2f6f2a630f8: No bootstrap required, opened a new log
I20240502 21:28:53.059976 3080 consensus_queue.cc:260] T b300ad3948074c728a25f053512bc37c P 153026dd7f8943cc93fedf6f48450fb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } }
I20240502 21:28:53.060406 3081 ts_tablet_manager.cc:1367] T b300ad3948074c728a25f053512bc37c P 521bb93035324cecbebed2f6f2a630f8: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.003s
I20240502 21:28:53.061240 3081 raft_consensus.cc:348] T b300ad3948074c728a25f053512bc37c P 521bb93035324cecbebed2f6f2a630f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } }
I20240502 21:28:53.061574 3081 raft_consensus.cc:374] T b300ad3948074c728a25f053512bc37c P 521bb93035324cecbebed2f6f2a630f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:53.061758 3081 raft_consensus.cc:729] T b300ad3948074c728a25f053512bc37c P 521bb93035324cecbebed2f6f2a630f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 521bb93035324cecbebed2f6f2a630f8, State: Initialized, Role: FOLLOWER
I20240502 21:28:53.062114 3080 ts_tablet_manager.cc:1398] T b300ad3948074c728a25f053512bc37c P 153026dd7f8943cc93fedf6f48450fb4: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 21:28:53.062093 3081 consensus_queue.cc:260] T b300ad3948074c728a25f053512bc37c P 521bb93035324cecbebed2f6f2a630f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } }
I20240502 21:28:53.062916 3081 ts_tablet_manager.cc:1398] T b300ad3948074c728a25f053512bc37c P 521bb93035324cecbebed2f6f2a630f8: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 21:28:53.063298 3081 tablet_bootstrap.cc:492] T 0658ecbf32754656876452b6f021926b P 521bb93035324cecbebed2f6f2a630f8: Bootstrap starting.
I20240502 21:28:53.064776 3080 tablet_bootstrap.cc:492] T 0658ecbf32754656876452b6f021926b P 153026dd7f8943cc93fedf6f48450fb4: Bootstrap starting.
I20240502 21:28:53.064906 3081 tablet_bootstrap.cc:654] T 0658ecbf32754656876452b6f021926b P 521bb93035324cecbebed2f6f2a630f8: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:53.067220 3082 tablet_bootstrap.cc:492] T 1aa57620996e42b6adf941e9d606f7e2 P 6ab8df3c796f4b6cb3e29d5045ad3ea0: No bootstrap required, opened a new log
I20240502 21:28:53.067452 3082 ts_tablet_manager.cc:1367] T 1aa57620996e42b6adf941e9d606f7e2 P 6ab8df3c796f4b6cb3e29d5045ad3ea0: Time spent bootstrapping tablet: real 0.022s user 0.004s sys 0.000s
I20240502 21:28:53.068152 3075 tablet_bootstrap.cc:492] T b300ad3948074c728a25f053512bc37c P a62a935ff43d494ca35d0c417943133d: No bootstrap required, opened a new log
I20240502 21:28:53.068603 3082 raft_consensus.cc:348] T 1aa57620996e42b6adf941e9d606f7e2 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } }
I20240502 21:28:53.068992 3082 raft_consensus.cc:374] T 1aa57620996e42b6adf941e9d606f7e2 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:53.069123 3082 raft_consensus.cc:729] T 1aa57620996e42b6adf941e9d606f7e2 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ab8df3c796f4b6cb3e29d5045ad3ea0, State: Initialized, Role: FOLLOWER
I20240502 21:28:53.069190 3076 tablet_bootstrap.cc:492] T 1aa57620996e42b6adf941e9d606f7e2 P 7d1e5c1b047443cbb28d7b0a9744babf: No bootstrap required, opened a new log
I20240502 21:28:53.069401 3076 ts_tablet_manager.cc:1367] T 1aa57620996e42b6adf941e9d606f7e2 P 7d1e5c1b047443cbb28d7b0a9744babf: Time spent bootstrapping tablet: real 0.128s user 0.027s sys 0.046s
I20240502 21:28:53.069589 3080 tablet_bootstrap.cc:654] T 0658ecbf32754656876452b6f021926b P 153026dd7f8943cc93fedf6f48450fb4: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:53.069468 3082 consensus_queue.cc:260] T 1aa57620996e42b6adf941e9d606f7e2 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } }
I20240502 21:28:53.069877 3075 ts_tablet_manager.cc:1367] T b300ad3948074c728a25f053512bc37c P a62a935ff43d494ca35d0c417943133d: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.000s
I20240502 21:28:53.070318 3076 raft_consensus.cc:348] T 1aa57620996e42b6adf941e9d606f7e2 P 7d1e5c1b047443cbb28d7b0a9744babf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } }
I20240502 21:28:53.070645 3076 raft_consensus.cc:374] T 1aa57620996e42b6adf941e9d606f7e2 P 7d1e5c1b047443cbb28d7b0a9744babf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:53.070775 3076 raft_consensus.cc:729] T 1aa57620996e42b6adf941e9d606f7e2 P 7d1e5c1b047443cbb28d7b0a9744babf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d1e5c1b047443cbb28d7b0a9744babf, State: Initialized, Role: FOLLOWER
I20240502 21:28:53.071040 3076 consensus_queue.cc:260] T 1aa57620996e42b6adf941e9d606f7e2 P 7d1e5c1b047443cbb28d7b0a9744babf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } }
I20240502 21:28:53.071666 3076 ts_tablet_manager.cc:1398] T 1aa57620996e42b6adf941e9d606f7e2 P 7d1e5c1b047443cbb28d7b0a9744babf: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 21:28:53.072113 3076 tablet_bootstrap.cc:492] T b300ad3948074c728a25f053512bc37c P 7d1e5c1b047443cbb28d7b0a9744babf: Bootstrap starting.
I20240502 21:28:53.072772 3079 tablet_bootstrap.cc:492] T b300ad3948074c728a25f053512bc37c P e632c553ff4341328acf1cbaca678392: No bootstrap required, opened a new log
I20240502 21:28:53.073006 3079 ts_tablet_manager.cc:1367] T b300ad3948074c728a25f053512bc37c P e632c553ff4341328acf1cbaca678392: Time spent bootstrapping tablet: real 0.037s user 0.004s sys 0.001s
I20240502 21:28:53.073629 3076 tablet_bootstrap.cc:654] T b300ad3948074c728a25f053512bc37c P 7d1e5c1b047443cbb28d7b0a9744babf: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:53.074452 3079 raft_consensus.cc:348] T b300ad3948074c728a25f053512bc37c P e632c553ff4341328acf1cbaca678392 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } }
I20240502 21:28:53.074795 3079 raft_consensus.cc:374] T b300ad3948074c728a25f053512bc37c P e632c553ff4341328acf1cbaca678392 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:53.074908 3079 raft_consensus.cc:729] T b300ad3948074c728a25f053512bc37c P e632c553ff4341328acf1cbaca678392 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e632c553ff4341328acf1cbaca678392, State: Initialized, Role: FOLLOWER
I20240502 21:28:53.075224 3079 consensus_queue.cc:260] T b300ad3948074c728a25f053512bc37c P e632c553ff4341328acf1cbaca678392 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } }
I20240502 21:28:53.076200 3079 ts_tablet_manager.cc:1398] T b300ad3948074c728a25f053512bc37c P e632c553ff4341328acf1cbaca678392: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 21:28:53.076241 3074 tablet_bootstrap.cc:492] T 1aa57620996e42b6adf941e9d606f7e2 P 16b3a337bc5e462f83ba895ad8660deb: No bootstrap required, opened a new log
I20240502 21:28:53.076488 3074 ts_tablet_manager.cc:1367] T 1aa57620996e42b6adf941e9d606f7e2 P 16b3a337bc5e462f83ba895ad8660deb: Time spent bootstrapping tablet: real 0.142s user 0.062s sys 0.045s
I20240502 21:28:53.076613 3079 tablet_bootstrap.cc:492] T 0658ecbf32754656876452b6f021926b P e632c553ff4341328acf1cbaca678392: Bootstrap starting.
I20240502 21:28:53.077329 3074 raft_consensus.cc:348] T 1aa57620996e42b6adf941e9d606f7e2 P 16b3a337bc5e462f83ba895ad8660deb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } }
I20240502 21:28:53.077667 3074 raft_consensus.cc:374] T 1aa57620996e42b6adf941e9d606f7e2 P 16b3a337bc5e462f83ba895ad8660deb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:53.077803 3074 raft_consensus.cc:729] T 1aa57620996e42b6adf941e9d606f7e2 P 16b3a337bc5e462f83ba895ad8660deb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16b3a337bc5e462f83ba895ad8660deb, State: Initialized, Role: FOLLOWER
I20240502 21:28:53.078079 3074 consensus_queue.cc:260] T 1aa57620996e42b6adf941e9d606f7e2 P 16b3a337bc5e462f83ba895ad8660deb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } }
I20240502 21:28:53.078409 3075 raft_consensus.cc:348] T b300ad3948074c728a25f053512bc37c P a62a935ff43d494ca35d0c417943133d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } }
I20240502 21:28:53.078779 3075 raft_consensus.cc:374] T b300ad3948074c728a25f053512bc37c P a62a935ff43d494ca35d0c417943133d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:53.079206 3075 raft_consensus.cc:729] T b300ad3948074c728a25f053512bc37c P a62a935ff43d494ca35d0c417943133d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a62a935ff43d494ca35d0c417943133d, State: Initialized, Role: FOLLOWER
I20240502 21:28:53.079818 3079 tablet_bootstrap.cc:654] T 0658ecbf32754656876452b6f021926b P e632c553ff4341328acf1cbaca678392: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:53.080058 3075 consensus_queue.cc:260] T b300ad3948074c728a25f053512bc37c P a62a935ff43d494ca35d0c417943133d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } }
I20240502 21:28:53.081332 3075 ts_tablet_manager.cc:1398] T b300ad3948074c728a25f053512bc37c P a62a935ff43d494ca35d0c417943133d: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20240502 21:28:53.083189 3075 tablet_bootstrap.cc:492] T 0658ecbf32754656876452b6f021926b P a62a935ff43d494ca35d0c417943133d: Bootstrap starting.
I20240502 21:28:53.085175 3079 tablet_bootstrap.cc:492] T 0658ecbf32754656876452b6f021926b P e632c553ff4341328acf1cbaca678392: No bootstrap required, opened a new log
I20240502 21:28:53.085341 3079 ts_tablet_manager.cc:1367] T 0658ecbf32754656876452b6f021926b P e632c553ff4341328acf1cbaca678392: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.002s
I20240502 21:28:53.085444 3082 ts_tablet_manager.cc:1398] T 1aa57620996e42b6adf941e9d606f7e2 P 6ab8df3c796f4b6cb3e29d5045ad3ea0: Time spent starting tablet: real 0.018s user 0.002s sys 0.000s
I20240502 21:28:53.086124 3079 raft_consensus.cc:348] T 0658ecbf32754656876452b6f021926b P e632c553ff4341328acf1cbaca678392 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:28:53.086405 3079 raft_consensus.cc:374] T 0658ecbf32754656876452b6f021926b P e632c553ff4341328acf1cbaca678392 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:53.086505 3079 raft_consensus.cc:729] T 0658ecbf32754656876452b6f021926b P e632c553ff4341328acf1cbaca678392 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e632c553ff4341328acf1cbaca678392, State: Initialized, Role: FOLLOWER
I20240502 21:28:53.086601 3080 tablet_bootstrap.cc:492] T 0658ecbf32754656876452b6f021926b P 153026dd7f8943cc93fedf6f48450fb4: No bootstrap required, opened a new log
I20240502 21:28:53.086796 3080 ts_tablet_manager.cc:1367] T 0658ecbf32754656876452b6f021926b P 153026dd7f8943cc93fedf6f48450fb4: Time spent bootstrapping tablet: real 0.022s user 0.000s sys 0.004s
I20240502 21:28:53.086773 3079 consensus_queue.cc:260] T 0658ecbf32754656876452b6f021926b P e632c553ff4341328acf1cbaca678392 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:28:53.087733 3080 raft_consensus.cc:348] T 0658ecbf32754656876452b6f021926b P 153026dd7f8943cc93fedf6f48450fb4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:28:53.088037 3080 raft_consensus.cc:374] T 0658ecbf32754656876452b6f021926b P 153026dd7f8943cc93fedf6f48450fb4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:53.088128 3080 raft_consensus.cc:729] T 0658ecbf32754656876452b6f021926b P 153026dd7f8943cc93fedf6f48450fb4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 153026dd7f8943cc93fedf6f48450fb4, State: Initialized, Role: FOLLOWER
I20240502 21:28:53.088284 3079 ts_tablet_manager.cc:1398] T 0658ecbf32754656876452b6f021926b P e632c553ff4341328acf1cbaca678392: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20240502 21:28:53.088450 3080 consensus_queue.cc:260] T 0658ecbf32754656876452b6f021926b P 153026dd7f8943cc93fedf6f48450fb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:28:53.089987 3080 ts_tablet_manager.cc:1398] T 0658ecbf32754656876452b6f021926b P 153026dd7f8943cc93fedf6f48450fb4: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240502 21:28:53.091023 3076 tablet_bootstrap.cc:492] T b300ad3948074c728a25f053512bc37c P 7d1e5c1b047443cbb28d7b0a9744babf: No bootstrap required, opened a new log
I20240502 21:28:53.091248 3075 tablet_bootstrap.cc:654] T 0658ecbf32754656876452b6f021926b P a62a935ff43d494ca35d0c417943133d: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:53.091244 3076 ts_tablet_manager.cc:1367] T b300ad3948074c728a25f053512bc37c P 7d1e5c1b047443cbb28d7b0a9744babf: Time spent bootstrapping tablet: real 0.019s user 0.003s sys 0.003s
I20240502 21:28:53.093102 3076 raft_consensus.cc:348] T b300ad3948074c728a25f053512bc37c P 7d1e5c1b047443cbb28d7b0a9744babf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } }
I20240502 21:28:53.095746 3076 raft_consensus.cc:374] T b300ad3948074c728a25f053512bc37c P 7d1e5c1b047443cbb28d7b0a9744babf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:53.095890 3076 raft_consensus.cc:729] T b300ad3948074c728a25f053512bc37c P 7d1e5c1b047443cbb28d7b0a9744babf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d1e5c1b047443cbb28d7b0a9744babf, State: Initialized, Role: FOLLOWER
I20240502 21:28:53.096256 3076 consensus_queue.cc:260] T b300ad3948074c728a25f053512bc37c P 7d1e5c1b047443cbb28d7b0a9744babf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } }
I20240502 21:28:53.097925 3076 ts_tablet_manager.cc:1398] T b300ad3948074c728a25f053512bc37c P 7d1e5c1b047443cbb28d7b0a9744babf: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240502 21:28:53.098361 3076 tablet_bootstrap.cc:492] T 0658ecbf32754656876452b6f021926b P 7d1e5c1b047443cbb28d7b0a9744babf: Bootstrap starting.
I20240502 21:28:53.099821 3082 tablet_bootstrap.cc:492] T 0658ecbf32754656876452b6f021926b P 6ab8df3c796f4b6cb3e29d5045ad3ea0: Bootstrap starting.
I20240502 21:28:53.100768 3076 tablet_bootstrap.cc:654] T 0658ecbf32754656876452b6f021926b P 7d1e5c1b047443cbb28d7b0a9744babf: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:53.103127 3082 tablet_bootstrap.cc:654] T 0658ecbf32754656876452b6f021926b P 6ab8df3c796f4b6cb3e29d5045ad3ea0: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:53.103127 3075 tablet_bootstrap.cc:492] T 0658ecbf32754656876452b6f021926b P a62a935ff43d494ca35d0c417943133d: No bootstrap required, opened a new log
I20240502 21:28:53.103303 3075 ts_tablet_manager.cc:1367] T 0658ecbf32754656876452b6f021926b P a62a935ff43d494ca35d0c417943133d: Time spent bootstrapping tablet: real 0.021s user 0.005s sys 0.003s
I20240502 21:28:53.104224 3075 raft_consensus.cc:348] T 0658ecbf32754656876452b6f021926b P a62a935ff43d494ca35d0c417943133d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:28:53.104700 3075 raft_consensus.cc:374] T 0658ecbf32754656876452b6f021926b P a62a935ff43d494ca35d0c417943133d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:53.104858 3075 raft_consensus.cc:729] T 0658ecbf32754656876452b6f021926b P a62a935ff43d494ca35d0c417943133d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a62a935ff43d494ca35d0c417943133d, State: Initialized, Role: FOLLOWER
I20240502 21:28:53.105193 3075 consensus_queue.cc:260] T 0658ecbf32754656876452b6f021926b P a62a935ff43d494ca35d0c417943133d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:28:53.105922 3075 ts_tablet_manager.cc:1398] T 0658ecbf32754656876452b6f021926b P a62a935ff43d494ca35d0c417943133d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:28:53.107057 3076 tablet_bootstrap.cc:492] T 0658ecbf32754656876452b6f021926b P 7d1e5c1b047443cbb28d7b0a9744babf: No bootstrap required, opened a new log
I20240502 21:28:53.107205 3081 tablet_bootstrap.cc:492] T 0658ecbf32754656876452b6f021926b P 521bb93035324cecbebed2f6f2a630f8: No bootstrap required, opened a new log
I20240502 21:28:53.107293 3076 ts_tablet_manager.cc:1367] T 0658ecbf32754656876452b6f021926b P 7d1e5c1b047443cbb28d7b0a9744babf: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20240502 21:28:53.107550 3081 ts_tablet_manager.cc:1367] T 0658ecbf32754656876452b6f021926b P 521bb93035324cecbebed2f6f2a630f8: Time spent bootstrapping tablet: real 0.044s user 0.003s sys 0.000s
I20240502 21:28:53.108139 3076 raft_consensus.cc:348] T 0658ecbf32754656876452b6f021926b P 7d1e5c1b047443cbb28d7b0a9744babf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:28:53.108522 3076 raft_consensus.cc:374] T 0658ecbf32754656876452b6f021926b P 7d1e5c1b047443cbb28d7b0a9744babf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:53.108672 3076 raft_consensus.cc:729] T 0658ecbf32754656876452b6f021926b P 7d1e5c1b047443cbb28d7b0a9744babf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d1e5c1b047443cbb28d7b0a9744babf, State: Initialized, Role: FOLLOWER
I20240502 21:28:53.108983 3076 consensus_queue.cc:260] T 0658ecbf32754656876452b6f021926b P 7d1e5c1b047443cbb28d7b0a9744babf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:28:53.109580 3076 ts_tablet_manager.cc:1398] T 0658ecbf32754656876452b6f021926b P 7d1e5c1b047443cbb28d7b0a9744babf: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 21:28:53.110575 3081 raft_consensus.cc:348] T 0658ecbf32754656876452b6f021926b P 521bb93035324cecbebed2f6f2a630f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:28:53.110882 3081 raft_consensus.cc:374] T 0658ecbf32754656876452b6f021926b P 521bb93035324cecbebed2f6f2a630f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:53.110958 3081 raft_consensus.cc:729] T 0658ecbf32754656876452b6f021926b P 521bb93035324cecbebed2f6f2a630f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 521bb93035324cecbebed2f6f2a630f8, State: Initialized, Role: FOLLOWER
I20240502 21:28:53.111244 3081 consensus_queue.cc:260] T 0658ecbf32754656876452b6f021926b P 521bb93035324cecbebed2f6f2a630f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:28:53.111848 3081 ts_tablet_manager.cc:1398] T 0658ecbf32754656876452b6f021926b P 521bb93035324cecbebed2f6f2a630f8: Time spent starting tablet: real 0.004s user 0.000s sys 0.000s
I20240502 21:28:53.116779 3081 tablet_bootstrap.cc:492] T 1aa57620996e42b6adf941e9d606f7e2 P 521bb93035324cecbebed2f6f2a630f8: Bootstrap starting.
I20240502 21:28:53.119464 3081 tablet_bootstrap.cc:654] T 1aa57620996e42b6adf941e9d606f7e2 P 521bb93035324cecbebed2f6f2a630f8: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:53.120805 3074 ts_tablet_manager.cc:1398] T 1aa57620996e42b6adf941e9d606f7e2 P 16b3a337bc5e462f83ba895ad8660deb: Time spent starting tablet: real 0.044s user 0.011s sys 0.032s
I20240502 21:28:53.121512 3081 tablet_bootstrap.cc:492] T 1aa57620996e42b6adf941e9d606f7e2 P 521bb93035324cecbebed2f6f2a630f8: No bootstrap required, opened a new log
I20240502 21:28:53.122076 3082 tablet_bootstrap.cc:492] T 0658ecbf32754656876452b6f021926b P 6ab8df3c796f4b6cb3e29d5045ad3ea0: No bootstrap required, opened a new log
I20240502 21:28:53.122253 3074 tablet_bootstrap.cc:492] T b300ad3948074c728a25f053512bc37c P 16b3a337bc5e462f83ba895ad8660deb: Bootstrap starting.
I20240502 21:28:53.122318 3082 ts_tablet_manager.cc:1367] T 0658ecbf32754656876452b6f021926b P 6ab8df3c796f4b6cb3e29d5045ad3ea0: Time spent bootstrapping tablet: real 0.023s user 0.004s sys 0.000s
I20240502 21:28:53.123181 3082 raft_consensus.cc:348] T 0658ecbf32754656876452b6f021926b P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:28:53.123498 3082 raft_consensus.cc:374] T 0658ecbf32754656876452b6f021926b P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:53.123586 3082 raft_consensus.cc:729] T 0658ecbf32754656876452b6f021926b P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ab8df3c796f4b6cb3e29d5045ad3ea0, State: Initialized, Role: FOLLOWER
I20240502 21:28:53.123855 3074 tablet_bootstrap.cc:654] T b300ad3948074c728a25f053512bc37c P 16b3a337bc5e462f83ba895ad8660deb: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:53.124053 3081 ts_tablet_manager.cc:1367] T 1aa57620996e42b6adf941e9d606f7e2 P 521bb93035324cecbebed2f6f2a630f8: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.002s
I20240502 21:28:53.123852 3082 consensus_queue.cc:260] T 0658ecbf32754656876452b6f021926b P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:28:53.124572 3082 ts_tablet_manager.cc:1398] T 0658ecbf32754656876452b6f021926b P 6ab8df3c796f4b6cb3e29d5045ad3ea0: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:28:53.124951 3082 tablet_bootstrap.cc:492] T b300ad3948074c728a25f053512bc37c P 6ab8df3c796f4b6cb3e29d5045ad3ea0: Bootstrap starting.
I20240502 21:28:53.125406 3081 raft_consensus.cc:348] T 1aa57620996e42b6adf941e9d606f7e2 P 521bb93035324cecbebed2f6f2a630f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } }
I20240502 21:28:53.125737 3081 raft_consensus.cc:374] T 1aa57620996e42b6adf941e9d606f7e2 P 521bb93035324cecbebed2f6f2a630f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:53.125870 3081 raft_consensus.cc:729] T 1aa57620996e42b6adf941e9d606f7e2 P 521bb93035324cecbebed2f6f2a630f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 521bb93035324cecbebed2f6f2a630f8, State: Initialized, Role: FOLLOWER
I20240502 21:28:53.126238 3081 consensus_queue.cc:260] T 1aa57620996e42b6adf941e9d606f7e2 P 521bb93035324cecbebed2f6f2a630f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } }
I20240502 21:28:53.127099 3081 ts_tablet_manager.cc:1398] T 1aa57620996e42b6adf941e9d606f7e2 P 521bb93035324cecbebed2f6f2a630f8: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 21:28:53.127401 3082 tablet_bootstrap.cc:654] T b300ad3948074c728a25f053512bc37c P 6ab8df3c796f4b6cb3e29d5045ad3ea0: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:53.132546 3082 tablet_bootstrap.cc:492] T b300ad3948074c728a25f053512bc37c P 6ab8df3c796f4b6cb3e29d5045ad3ea0: No bootstrap required, opened a new log
I20240502 21:28:53.132694 3082 ts_tablet_manager.cc:1367] T b300ad3948074c728a25f053512bc37c P 6ab8df3c796f4b6cb3e29d5045ad3ea0: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20240502 21:28:53.132963 3074 tablet_bootstrap.cc:492] T b300ad3948074c728a25f053512bc37c P 16b3a337bc5e462f83ba895ad8660deb: No bootstrap required, opened a new log
I20240502 21:28:53.133118 3074 ts_tablet_manager.cc:1367] T b300ad3948074c728a25f053512bc37c P 16b3a337bc5e462f83ba895ad8660deb: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.003s
I20240502 21:28:53.133539 3082 raft_consensus.cc:348] T b300ad3948074c728a25f053512bc37c P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } }
I20240502 21:28:53.133723 3082 raft_consensus.cc:374] T b300ad3948074c728a25f053512bc37c P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:53.133656 3074 raft_consensus.cc:348] T b300ad3948074c728a25f053512bc37c P 16b3a337bc5e462f83ba895ad8660deb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } }
I20240502 21:28:53.133812 3082 raft_consensus.cc:729] T b300ad3948074c728a25f053512bc37c P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ab8df3c796f4b6cb3e29d5045ad3ea0, State: Initialized, Role: FOLLOWER
I20240502 21:28:53.133898 3074 raft_consensus.cc:374] T b300ad3948074c728a25f053512bc37c P 16b3a337bc5e462f83ba895ad8660deb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:53.133981 3074 raft_consensus.cc:729] T b300ad3948074c728a25f053512bc37c P 16b3a337bc5e462f83ba895ad8660deb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16b3a337bc5e462f83ba895ad8660deb, State: Initialized, Role: FOLLOWER
I20240502 21:28:53.134039 3082 consensus_queue.cc:260] T b300ad3948074c728a25f053512bc37c P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } }
I20240502 21:28:53.134212 3074 consensus_queue.cc:260] T b300ad3948074c728a25f053512bc37c P 16b3a337bc5e462f83ba895ad8660deb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } }
I20240502 21:28:53.134593 3082 ts_tablet_manager.cc:1398] T b300ad3948074c728a25f053512bc37c P 6ab8df3c796f4b6cb3e29d5045ad3ea0: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 21:28:53.134747 3074 ts_tablet_manager.cc:1398] T b300ad3948074c728a25f053512bc37c P 16b3a337bc5e462f83ba895ad8660deb: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 21:28:53.135306 3074 tablet_bootstrap.cc:492] T 0658ecbf32754656876452b6f021926b P 16b3a337bc5e462f83ba895ad8660deb: Bootstrap starting.
I20240502 21:28:53.136765 3074 tablet_bootstrap.cc:654] T 0658ecbf32754656876452b6f021926b P 16b3a337bc5e462f83ba895ad8660deb: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:53.139667 3074 tablet_bootstrap.cc:492] T 0658ecbf32754656876452b6f021926b P 16b3a337bc5e462f83ba895ad8660deb: No bootstrap required, opened a new log
I20240502 21:28:53.139788 3074 ts_tablet_manager.cc:1367] T 0658ecbf32754656876452b6f021926b P 16b3a337bc5e462f83ba895ad8660deb: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240502 21:28:53.140313 3074 raft_consensus.cc:348] T 0658ecbf32754656876452b6f021926b P 16b3a337bc5e462f83ba895ad8660deb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:28:53.140522 3074 raft_consensus.cc:374] T 0658ecbf32754656876452b6f021926b P 16b3a337bc5e462f83ba895ad8660deb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:53.140609 3074 raft_consensus.cc:729] T 0658ecbf32754656876452b6f021926b P 16b3a337bc5e462f83ba895ad8660deb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16b3a337bc5e462f83ba895ad8660deb, State: Initialized, Role: FOLLOWER
I20240502 21:28:53.140805 3074 consensus_queue.cc:260] T 0658ecbf32754656876452b6f021926b P 16b3a337bc5e462f83ba895ad8660deb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:28:53.141244 3074 ts_tablet_manager.cc:1398] T 0658ecbf32754656876452b6f021926b P 16b3a337bc5e462f83ba895ad8660deb: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 21:28:53.150405 2963 raft_consensus.cc:482] T 1aa57620996e42b6adf941e9d606f7e2 P 521bb93035324cecbebed2f6f2a630f8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:28:53.150583 2963 raft_consensus.cc:504] T 1aa57620996e42b6adf941e9d606f7e2 P 521bb93035324cecbebed2f6f2a630f8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } }
I20240502 21:28:53.151681 2963 leader_election.cc:288] T 1aa57620996e42b6adf941e9d606f7e2 P 521bb93035324cecbebed2f6f2a630f8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a62a935ff43d494ca35d0c417943133d (127.15.255.130:37991), 7d1e5c1b047443cbb28d7b0a9744babf (127.15.255.129:37897), 16b3a337bc5e462f83ba895ad8660deb (127.15.255.131:36033), e632c553ff4341328acf1cbaca678392 (127.15.255.133:37439), 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132:44947), 6ab8df3c796f4b6cb3e29d5045ad3ea0 (127.15.255.135:34701)
I20240502 21:28:53.151871 1549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1aa57620996e42b6adf941e9d606f7e2" candidate_uuid: "521bb93035324cecbebed2f6f2a630f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a62a935ff43d494ca35d0c417943133d" is_pre_election: true
I20240502 21:28:53.151898 1415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1aa57620996e42b6adf941e9d606f7e2" candidate_uuid: "521bb93035324cecbebed2f6f2a630f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" is_pre_election: true
I20240502 21:28:53.152125 1415 raft_consensus.cc:2454] T 1aa57620996e42b6adf941e9d606f7e2 P 7d1e5c1b047443cbb28d7b0a9744babf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 521bb93035324cecbebed2f6f2a630f8 in term 0.
I20240502 21:28:53.152127 1549 raft_consensus.cc:2454] T 1aa57620996e42b6adf941e9d606f7e2 P a62a935ff43d494ca35d0c417943133d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 521bb93035324cecbebed2f6f2a630f8 in term 0.
I20240502 21:28:53.152395 1953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1aa57620996e42b6adf941e9d606f7e2" candidate_uuid: "521bb93035324cecbebed2f6f2a630f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e632c553ff4341328acf1cbaca678392" is_pre_election: true
I20240502 21:28:53.152330 1818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1aa57620996e42b6adf941e9d606f7e2" candidate_uuid: "521bb93035324cecbebed2f6f2a630f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "153026dd7f8943cc93fedf6f48450fb4" is_pre_election: true
I20240502 21:28:53.152539 1686 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1aa57620996e42b6adf941e9d606f7e2" candidate_uuid: "521bb93035324cecbebed2f6f2a630f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16b3a337bc5e462f83ba895ad8660deb" is_pre_election: true
I20240502 21:28:53.152642 1818 raft_consensus.cc:2454] T 1aa57620996e42b6adf941e9d606f7e2 P 153026dd7f8943cc93fedf6f48450fb4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 521bb93035324cecbebed2f6f2a630f8 in term 0.
I20240502 21:28:53.152642 1953 raft_consensus.cc:2454] T 1aa57620996e42b6adf941e9d606f7e2 P e632c553ff4341328acf1cbaca678392 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 521bb93035324cecbebed2f6f2a630f8 in term 0.
I20240502 21:28:53.152752 1686 raft_consensus.cc:2454] T 1aa57620996e42b6adf941e9d606f7e2 P 16b3a337bc5e462f83ba895ad8660deb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 521bb93035324cecbebed2f6f2a630f8 in term 0.
I20240502 21:28:53.153085 2022 leader_election.cc:302] T 1aa57620996e42b6adf941e9d606f7e2 P 521bb93035324cecbebed2f6f2a630f8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 5 responses out of 7 voters: 5 yes votes; 0 no votes. yes voters: 153026dd7f8943cc93fedf6f48450fb4, 521bb93035324cecbebed2f6f2a630f8, 7d1e5c1b047443cbb28d7b0a9744babf, a62a935ff43d494ca35d0c417943133d, e632c553ff4341328acf1cbaca678392; no voters:
I20240502 21:28:53.153246 2227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1aa57620996e42b6adf941e9d606f7e2" candidate_uuid: "521bb93035324cecbebed2f6f2a630f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" is_pre_election: true
I20240502 21:28:53.153375 2963 raft_consensus.cc:2781] T 1aa57620996e42b6adf941e9d606f7e2 P 521bb93035324cecbebed2f6f2a630f8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:28:53.153468 2963 raft_consensus.cc:482] T 1aa57620996e42b6adf941e9d606f7e2 P 521bb93035324cecbebed2f6f2a630f8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:28:53.153493 2227 raft_consensus.cc:2454] T 1aa57620996e42b6adf941e9d606f7e2 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 521bb93035324cecbebed2f6f2a630f8 in term 0.
I20240502 21:28:53.153573 2963 raft_consensus.cc:3037] T 1aa57620996e42b6adf941e9d606f7e2 P 521bb93035324cecbebed2f6f2a630f8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:53.155544 2963 raft_consensus.cc:504] T 1aa57620996e42b6adf941e9d606f7e2 P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } }
I20240502 21:28:53.156544 1549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1aa57620996e42b6adf941e9d606f7e2" candidate_uuid: "521bb93035324cecbebed2f6f2a630f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a62a935ff43d494ca35d0c417943133d"
I20240502 21:28:53.156668 1415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1aa57620996e42b6adf941e9d606f7e2" candidate_uuid: "521bb93035324cecbebed2f6f2a630f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d1e5c1b047443cbb28d7b0a9744babf"
I20240502 21:28:53.156761 1549 raft_consensus.cc:3037] T 1aa57620996e42b6adf941e9d606f7e2 P a62a935ff43d494ca35d0c417943133d [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:53.156860 1415 raft_consensus.cc:3037] T 1aa57620996e42b6adf941e9d606f7e2 P 7d1e5c1b047443cbb28d7b0a9744babf [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:53.156885 2963 leader_election.cc:288] T 1aa57620996e42b6adf941e9d606f7e2 P 521bb93035324cecbebed2f6f2a630f8 [CANDIDATE]: Term 1 election: Requested vote from peers a62a935ff43d494ca35d0c417943133d (127.15.255.130:37991), 7d1e5c1b047443cbb28d7b0a9744babf (127.15.255.129:37897), 16b3a337bc5e462f83ba895ad8660deb (127.15.255.131:36033), e632c553ff4341328acf1cbaca678392 (127.15.255.133:37439), 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132:44947), 6ab8df3c796f4b6cb3e29d5045ad3ea0 (127.15.255.135:34701)
I20240502 21:28:53.157016 1953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1aa57620996e42b6adf941e9d606f7e2" candidate_uuid: "521bb93035324cecbebed2f6f2a630f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e632c553ff4341328acf1cbaca678392"
I20240502 21:28:53.157203 1953 raft_consensus.cc:3037] T 1aa57620996e42b6adf941e9d606f7e2 P e632c553ff4341328acf1cbaca678392 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:53.158936 1415 raft_consensus.cc:2454] T 1aa57620996e42b6adf941e9d606f7e2 P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 521bb93035324cecbebed2f6f2a630f8 in term 1.
I20240502 21:28:53.159296 1686 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1aa57620996e42b6adf941e9d606f7e2" candidate_uuid: "521bb93035324cecbebed2f6f2a630f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16b3a337bc5e462f83ba895ad8660deb"
I20240502 21:28:53.159442 1549 raft_consensus.cc:2454] T 1aa57620996e42b6adf941e9d606f7e2 P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 521bb93035324cecbebed2f6f2a630f8 in term 1.
I20240502 21:28:53.159493 1686 raft_consensus.cc:3037] T 1aa57620996e42b6adf941e9d606f7e2 P 16b3a337bc5e462f83ba895ad8660deb [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:53.159823 1953 raft_consensus.cc:2454] T 1aa57620996e42b6adf941e9d606f7e2 P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 521bb93035324cecbebed2f6f2a630f8 in term 1.
I20240502 21:28:53.160123 2227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1aa57620996e42b6adf941e9d606f7e2" candidate_uuid: "521bb93035324cecbebed2f6f2a630f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0"
I20240502 21:28:53.160192 2022 leader_election.cc:302] T 1aa57620996e42b6adf941e9d606f7e2 P 521bb93035324cecbebed2f6f2a630f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 521bb93035324cecbebed2f6f2a630f8, 7d1e5c1b047443cbb28d7b0a9744babf, a62a935ff43d494ca35d0c417943133d, e632c553ff4341328acf1cbaca678392; no voters:
I20240502 21:28:53.160316 2227 raft_consensus.cc:3037] T 1aa57620996e42b6adf941e9d606f7e2 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:53.160490 2963 raft_consensus.cc:2781] T 1aa57620996e42b6adf941e9d606f7e2 P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:28:53.160630 2963 raft_consensus.cc:686] T 1aa57620996e42b6adf941e9d606f7e2 P 521bb93035324cecbebed2f6f2a630f8 [term 1 LEADER]: Becoming Leader. State: Replica: 521bb93035324cecbebed2f6f2a630f8, State: Running, Role: LEADER
I20240502 21:28:53.160598 1818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1aa57620996e42b6adf941e9d606f7e2" candidate_uuid: "521bb93035324cecbebed2f6f2a630f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "153026dd7f8943cc93fedf6f48450fb4"
I20240502 21:28:53.160797 1818 raft_consensus.cc:3037] T 1aa57620996e42b6adf941e9d606f7e2 P 153026dd7f8943cc93fedf6f48450fb4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:53.160977 2963 consensus_queue.cc:237] T 1aa57620996e42b6adf941e9d606f7e2 P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } }
I20240502 21:28:53.161975 1686 raft_consensus.cc:2454] T 1aa57620996e42b6adf941e9d606f7e2 P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 521bb93035324cecbebed2f6f2a630f8 in term 1.
I20240502 21:28:53.163476 1818 raft_consensus.cc:2454] T 1aa57620996e42b6adf941e9d606f7e2 P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 521bb93035324cecbebed2f6f2a630f8 in term 1.
I20240502 21:28:53.163558 2227 raft_consensus.cc:2454] T 1aa57620996e42b6adf941e9d606f7e2 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 521bb93035324cecbebed2f6f2a630f8 in term 1.
I20240502 21:28:53.165432 1264 catalog_manager.cc:5484] T 1aa57620996e42b6adf941e9d606f7e2 P 521bb93035324cecbebed2f6f2a630f8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 521bb93035324cecbebed2f6f2a630f8 (127.15.255.134). New cstate: current_term: 1 leader_uuid: "521bb93035324cecbebed2f6f2a630f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } health_report { overall_health: UNKNOWN } } }
I20240502 21:28:53.174240 2963 raft_consensus.cc:482] T b300ad3948074c728a25f053512bc37c P 521bb93035324cecbebed2f6f2a630f8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:28:53.174386 2963 raft_consensus.cc:504] T b300ad3948074c728a25f053512bc37c P 521bb93035324cecbebed2f6f2a630f8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } }
I20240502 21:28:53.175362 1415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b300ad3948074c728a25f053512bc37c" candidate_uuid: "521bb93035324cecbebed2f6f2a630f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" is_pre_election: true
I20240502 21:28:53.175432 1549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b300ad3948074c728a25f053512bc37c" candidate_uuid: "521bb93035324cecbebed2f6f2a630f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a62a935ff43d494ca35d0c417943133d" is_pre_election: true
I20240502 21:28:53.175591 1415 raft_consensus.cc:2454] T b300ad3948074c728a25f053512bc37c P 7d1e5c1b047443cbb28d7b0a9744babf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 521bb93035324cecbebed2f6f2a630f8 in term 0.
I20240502 21:28:53.175611 1549 raft_consensus.cc:2454] T b300ad3948074c728a25f053512bc37c P a62a935ff43d494ca35d0c417943133d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 521bb93035324cecbebed2f6f2a630f8 in term 0.
I20240502 21:28:53.175572 1686 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b300ad3948074c728a25f053512bc37c" candidate_uuid: "521bb93035324cecbebed2f6f2a630f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16b3a337bc5e462f83ba895ad8660deb" is_pre_election: true
I20240502 21:28:53.175572 1818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b300ad3948074c728a25f053512bc37c" candidate_uuid: "521bb93035324cecbebed2f6f2a630f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "153026dd7f8943cc93fedf6f48450fb4" is_pre_election: true
I20240502 21:28:53.175767 1686 raft_consensus.cc:2454] T b300ad3948074c728a25f053512bc37c P 16b3a337bc5e462f83ba895ad8660deb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 521bb93035324cecbebed2f6f2a630f8 in term 0.
I20240502 21:28:53.175762 1818 raft_consensus.cc:2454] T b300ad3948074c728a25f053512bc37c P 153026dd7f8943cc93fedf6f48450fb4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 521bb93035324cecbebed2f6f2a630f8 in term 0.
I20240502 21:28:53.176105 2024 leader_election.cc:302] T b300ad3948074c728a25f053512bc37c P 521bb93035324cecbebed2f6f2a630f8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 153026dd7f8943cc93fedf6f48450fb4, 521bb93035324cecbebed2f6f2a630f8, 7d1e5c1b047443cbb28d7b0a9744babf, a62a935ff43d494ca35d0c417943133d; no voters:
I20240502 21:28:53.176373 3005 raft_consensus.cc:2781] T b300ad3948074c728a25f053512bc37c P 521bb93035324cecbebed2f6f2a630f8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:28:53.176494 3005 raft_consensus.cc:482] T b300ad3948074c728a25f053512bc37c P 521bb93035324cecbebed2f6f2a630f8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:28:53.176592 2963 leader_election.cc:288] T b300ad3948074c728a25f053512bc37c P 521bb93035324cecbebed2f6f2a630f8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7d1e5c1b047443cbb28d7b0a9744babf (127.15.255.129:37897), a62a935ff43d494ca35d0c417943133d (127.15.255.130:37991), 16b3a337bc5e462f83ba895ad8660deb (127.15.255.131:36033), 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132:44947), e632c553ff4341328acf1cbaca678392 (127.15.255.133:37439), 6ab8df3c796f4b6cb3e29d5045ad3ea0 (127.15.255.135:34701)
I20240502 21:28:53.176620 3005 raft_consensus.cc:3037] T b300ad3948074c728a25f053512bc37c P 521bb93035324cecbebed2f6f2a630f8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:53.176882 1953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b300ad3948074c728a25f053512bc37c" candidate_uuid: "521bb93035324cecbebed2f6f2a630f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e632c553ff4341328acf1cbaca678392" is_pre_election: true
I20240502 21:28:53.177047 1953 raft_consensus.cc:2454] T b300ad3948074c728a25f053512bc37c P e632c553ff4341328acf1cbaca678392 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 521bb93035324cecbebed2f6f2a630f8 in term 0.
I20240502 21:28:53.177424 2227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b300ad3948074c728a25f053512bc37c" candidate_uuid: "521bb93035324cecbebed2f6f2a630f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" is_pre_election: true
I20240502 21:28:53.177601 2227 raft_consensus.cc:2454] T b300ad3948074c728a25f053512bc37c P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 521bb93035324cecbebed2f6f2a630f8 in term 0.
I20240502 21:28:53.178915 3005 raft_consensus.cc:504] T b300ad3948074c728a25f053512bc37c P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } }
I20240502 21:28:53.179893 3005 leader_election.cc:288] T b300ad3948074c728a25f053512bc37c P 521bb93035324cecbebed2f6f2a630f8 [CANDIDATE]: Term 1 election: Requested vote from peers 7d1e5c1b047443cbb28d7b0a9744babf (127.15.255.129:37897), a62a935ff43d494ca35d0c417943133d (127.15.255.130:37991), 16b3a337bc5e462f83ba895ad8660deb (127.15.255.131:36033), 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132:44947), e632c553ff4341328acf1cbaca678392 (127.15.255.133:37439), 6ab8df3c796f4b6cb3e29d5045ad3ea0 (127.15.255.135:34701)
I20240502 21:28:53.179888 1415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b300ad3948074c728a25f053512bc37c" candidate_uuid: "521bb93035324cecbebed2f6f2a630f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d1e5c1b047443cbb28d7b0a9744babf"
I20240502 21:28:53.180079 1415 raft_consensus.cc:3037] T b300ad3948074c728a25f053512bc37c P 7d1e5c1b047443cbb28d7b0a9744babf [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:53.180059 1686 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b300ad3948074c728a25f053512bc37c" candidate_uuid: "521bb93035324cecbebed2f6f2a630f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16b3a337bc5e462f83ba895ad8660deb"
I20240502 21:28:53.180227 1686 raft_consensus.cc:3037] T b300ad3948074c728a25f053512bc37c P 16b3a337bc5e462f83ba895ad8660deb [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:53.180215 1953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b300ad3948074c728a25f053512bc37c" candidate_uuid: "521bb93035324cecbebed2f6f2a630f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e632c553ff4341328acf1cbaca678392"
I20240502 21:28:53.180373 1953 raft_consensus.cc:3037] T b300ad3948074c728a25f053512bc37c P e632c553ff4341328acf1cbaca678392 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:53.180928 2227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b300ad3948074c728a25f053512bc37c" candidate_uuid: "521bb93035324cecbebed2f6f2a630f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0"
I20240502 21:28:53.181102 2227 raft_consensus.cc:3037] T b300ad3948074c728a25f053512bc37c P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:53.182674 1415 raft_consensus.cc:2454] T b300ad3948074c728a25f053512bc37c P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 521bb93035324cecbebed2f6f2a630f8 in term 1.
I20240502 21:28:53.182988 1953 raft_consensus.cc:2454] T b300ad3948074c728a25f053512bc37c P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 521bb93035324cecbebed2f6f2a630f8 in term 1.
I20240502 21:28:53.183099 1686 raft_consensus.cc:2454] T b300ad3948074c728a25f053512bc37c P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 521bb93035324cecbebed2f6f2a630f8 in term 1.
I20240502 21:28:53.183463 2025 leader_election.cc:302] T b300ad3948074c728a25f053512bc37c P 521bb93035324cecbebed2f6f2a630f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 16b3a337bc5e462f83ba895ad8660deb, 521bb93035324cecbebed2f6f2a630f8, 7d1e5c1b047443cbb28d7b0a9744babf, e632c553ff4341328acf1cbaca678392; no voters:
I20240502 21:28:53.183538 1818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b300ad3948074c728a25f053512bc37c" candidate_uuid: "521bb93035324cecbebed2f6f2a630f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "153026dd7f8943cc93fedf6f48450fb4"
I20240502 21:28:53.183604 1549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b300ad3948074c728a25f053512bc37c" candidate_uuid: "521bb93035324cecbebed2f6f2a630f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a62a935ff43d494ca35d0c417943133d"
I20240502 21:28:53.183737 1818 raft_consensus.cc:3037] T b300ad3948074c728a25f053512bc37c P 153026dd7f8943cc93fedf6f48450fb4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:53.183766 2227 raft_consensus.cc:2454] T b300ad3948074c728a25f053512bc37c P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 521bb93035324cecbebed2f6f2a630f8 in term 1.
I20240502 21:28:53.183831 1549 raft_consensus.cc:3037] T b300ad3948074c728a25f053512bc37c P a62a935ff43d494ca35d0c417943133d [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:53.183992 3005 raft_consensus.cc:2781] T b300ad3948074c728a25f053512bc37c P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:28:53.184190 3005 raft_consensus.cc:686] T b300ad3948074c728a25f053512bc37c P 521bb93035324cecbebed2f6f2a630f8 [term 1 LEADER]: Becoming Leader. State: Replica: 521bb93035324cecbebed2f6f2a630f8, State: Running, Role: LEADER
I20240502 21:28:53.184491 3005 consensus_queue.cc:237] T b300ad3948074c728a25f053512bc37c P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } }
I20240502 21:28:53.186447 1818 raft_consensus.cc:2454] T b300ad3948074c728a25f053512bc37c P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 521bb93035324cecbebed2f6f2a630f8 in term 1.
I20240502 21:28:53.186599 1549 raft_consensus.cc:2454] T b300ad3948074c728a25f053512bc37c P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 521bb93035324cecbebed2f6f2a630f8 in term 1.
I20240502 21:28:53.188148 1264 catalog_manager.cc:5484] T b300ad3948074c728a25f053512bc37c P 521bb93035324cecbebed2f6f2a630f8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 521bb93035324cecbebed2f6f2a630f8 (127.15.255.134). New cstate: current_term: 1 leader_uuid: "521bb93035324cecbebed2f6f2a630f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } health_report { overall_health: UNKNOWN } } }
I20240502 21:28:53.240773 3064 raft_consensus.cc:482] T 0658ecbf32754656876452b6f021926b P e632c553ff4341328acf1cbaca678392 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:28:53.240926 3064 raft_consensus.cc:504] T 0658ecbf32754656876452b6f021926b P e632c553ff4341328acf1cbaca678392 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:28:53.241981 3064 leader_election.cc:288] T 0658ecbf32754656876452b6f021926b P e632c553ff4341328acf1cbaca678392 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132:44947), 521bb93035324cecbebed2f6f2a630f8 (127.15.255.134:40821), 6ab8df3c796f4b6cb3e29d5045ad3ea0 (127.15.255.135:34701), a62a935ff43d494ca35d0c417943133d (127.15.255.130:37991), 7d1e5c1b047443cbb28d7b0a9744babf (127.15.255.129:37897), 16b3a337bc5e462f83ba895ad8660deb (127.15.255.131:36033)
I20240502 21:28:53.242003 1818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0658ecbf32754656876452b6f021926b" candidate_uuid: "e632c553ff4341328acf1cbaca678392" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "153026dd7f8943cc93fedf6f48450fb4" is_pre_election: true
I20240502 21:28:53.242100 2088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0658ecbf32754656876452b6f021926b" candidate_uuid: "e632c553ff4341328acf1cbaca678392" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "521bb93035324cecbebed2f6f2a630f8" is_pre_election: true
I20240502 21:28:53.242247 1818 raft_consensus.cc:2454] T 0658ecbf32754656876452b6f021926b P 153026dd7f8943cc93fedf6f48450fb4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e632c553ff4341328acf1cbaca678392 in term 0.
I20240502 21:28:53.242300 2088 raft_consensus.cc:2454] T 0658ecbf32754656876452b6f021926b P 521bb93035324cecbebed2f6f2a630f8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e632c553ff4341328acf1cbaca678392 in term 0.
I20240502 21:28:53.242720 2227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0658ecbf32754656876452b6f021926b" candidate_uuid: "e632c553ff4341328acf1cbaca678392" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" is_pre_election: true
I20240502 21:28:53.242817 1686 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0658ecbf32754656876452b6f021926b" candidate_uuid: "e632c553ff4341328acf1cbaca678392" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16b3a337bc5e462f83ba895ad8660deb" is_pre_election: true
I20240502 21:28:53.242972 2227 raft_consensus.cc:2454] T 0658ecbf32754656876452b6f021926b P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e632c553ff4341328acf1cbaca678392 in term 0.
I20240502 21:28:53.243031 1686 raft_consensus.cc:2454] T 0658ecbf32754656876452b6f021926b P 16b3a337bc5e462f83ba895ad8660deb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e632c553ff4341328acf1cbaca678392 in term 0.
I20240502 21:28:53.243364 1549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0658ecbf32754656876452b6f021926b" candidate_uuid: "e632c553ff4341328acf1cbaca678392" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a62a935ff43d494ca35d0c417943133d" is_pre_election: true
I20240502 21:28:53.243448 1887 leader_election.cc:302] T 0658ecbf32754656876452b6f021926b P e632c553ff4341328acf1cbaca678392 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 153026dd7f8943cc93fedf6f48450fb4, 16b3a337bc5e462f83ba895ad8660deb, 521bb93035324cecbebed2f6f2a630f8, e632c553ff4341328acf1cbaca678392; no voters:
I20240502 21:28:53.243480 1415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0658ecbf32754656876452b6f021926b" candidate_uuid: "e632c553ff4341328acf1cbaca678392" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" is_pre_election: true
I20240502 21:28:53.243603 1549 raft_consensus.cc:2454] T 0658ecbf32754656876452b6f021926b P a62a935ff43d494ca35d0c417943133d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e632c553ff4341328acf1cbaca678392 in term 0.
I20240502 21:28:53.243677 1415 raft_consensus.cc:2454] T 0658ecbf32754656876452b6f021926b P 7d1e5c1b047443cbb28d7b0a9744babf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e632c553ff4341328acf1cbaca678392 in term 0.
I20240502 21:28:53.243700 3064 raft_consensus.cc:2781] T 0658ecbf32754656876452b6f021926b P e632c553ff4341328acf1cbaca678392 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:28:53.243803 3064 raft_consensus.cc:482] T 0658ecbf32754656876452b6f021926b P e632c553ff4341328acf1cbaca678392 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:28:53.243891 3064 raft_consensus.cc:3037] T 0658ecbf32754656876452b6f021926b P e632c553ff4341328acf1cbaca678392 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:53.246130 3064 raft_consensus.cc:504] T 0658ecbf32754656876452b6f021926b P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:28:53.247138 2088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0658ecbf32754656876452b6f021926b" candidate_uuid: "e632c553ff4341328acf1cbaca678392" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "521bb93035324cecbebed2f6f2a630f8"
I20240502 21:28:53.247152 1818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0658ecbf32754656876452b6f021926b" candidate_uuid: "e632c553ff4341328acf1cbaca678392" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "153026dd7f8943cc93fedf6f48450fb4"
I20240502 21:28:53.247361 2088 raft_consensus.cc:3037] T 0658ecbf32754656876452b6f021926b P 521bb93035324cecbebed2f6f2a630f8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:53.247375 1818 raft_consensus.cc:3037] T 0658ecbf32754656876452b6f021926b P 153026dd7f8943cc93fedf6f48450fb4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:53.247850 1415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0658ecbf32754656876452b6f021926b" candidate_uuid: "e632c553ff4341328acf1cbaca678392" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d1e5c1b047443cbb28d7b0a9744babf"
I20240502 21:28:53.248086 1415 raft_consensus.cc:3037] T 0658ecbf32754656876452b6f021926b P 7d1e5c1b047443cbb28d7b0a9744babf [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:53.248656 3064 leader_election.cc:288] T 0658ecbf32754656876452b6f021926b P e632c553ff4341328acf1cbaca678392 [CANDIDATE]: Term 1 election: Requested vote from peers 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132:44947), 521bb93035324cecbebed2f6f2a630f8 (127.15.255.134:40821), 6ab8df3c796f4b6cb3e29d5045ad3ea0 (127.15.255.135:34701), a62a935ff43d494ca35d0c417943133d (127.15.255.130:37991), 7d1e5c1b047443cbb28d7b0a9744babf (127.15.255.129:37897), 16b3a337bc5e462f83ba895ad8660deb (127.15.255.131:36033)
I20240502 21:28:53.249619 1818 raft_consensus.cc:2454] T 0658ecbf32754656876452b6f021926b P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e632c553ff4341328acf1cbaca678392 in term 1.
I20240502 21:28:53.249913 2088 raft_consensus.cc:2454] T 0658ecbf32754656876452b6f021926b P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e632c553ff4341328acf1cbaca678392 in term 1.
I20240502 21:28:53.250582 2227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0658ecbf32754656876452b6f021926b" candidate_uuid: "e632c553ff4341328acf1cbaca678392" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0"
I20240502 21:28:53.250686 1415 raft_consensus.cc:2454] T 0658ecbf32754656876452b6f021926b P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e632c553ff4341328acf1cbaca678392 in term 1.
I20240502 21:28:53.250788 2227 raft_consensus.cc:3037] T 0658ecbf32754656876452b6f021926b P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:53.251116 1885 leader_election.cc:302] T 0658ecbf32754656876452b6f021926b P e632c553ff4341328acf1cbaca678392 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 153026dd7f8943cc93fedf6f48450fb4, 521bb93035324cecbebed2f6f2a630f8, 7d1e5c1b047443cbb28d7b0a9744babf, e632c553ff4341328acf1cbaca678392; no voters:
I20240502 21:28:53.251400 3064 raft_consensus.cc:2781] T 0658ecbf32754656876452b6f021926b P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:28:53.251533 3064 raft_consensus.cc:686] T 0658ecbf32754656876452b6f021926b P e632c553ff4341328acf1cbaca678392 [term 1 LEADER]: Becoming Leader. State: Replica: e632c553ff4341328acf1cbaca678392, State: Running, Role: LEADER
I20240502 21:28:53.251907 3064 consensus_queue.cc:237] T 0658ecbf32754656876452b6f021926b P e632c553ff4341328acf1cbaca678392 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:28:53.252602 1686 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0658ecbf32754656876452b6f021926b" candidate_uuid: "e632c553ff4341328acf1cbaca678392" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16b3a337bc5e462f83ba895ad8660deb"
I20240502 21:28:53.252820 1686 raft_consensus.cc:3037] T 0658ecbf32754656876452b6f021926b P 16b3a337bc5e462f83ba895ad8660deb [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:53.253675 2227 raft_consensus.cc:2454] T 0658ecbf32754656876452b6f021926b P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e632c553ff4341328acf1cbaca678392 in term 1.
I20240502 21:28:53.255185 1686 raft_consensus.cc:2454] T 0658ecbf32754656876452b6f021926b P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e632c553ff4341328acf1cbaca678392 in term 1.
I20240502 21:28:53.255494 1549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0658ecbf32754656876452b6f021926b" candidate_uuid: "e632c553ff4341328acf1cbaca678392" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a62a935ff43d494ca35d0c417943133d"
I20240502 21:28:53.255697 1549 raft_consensus.cc:3037] T 0658ecbf32754656876452b6f021926b P a62a935ff43d494ca35d0c417943133d [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:53.255754 1264 catalog_manager.cc:5484] T 0658ecbf32754656876452b6f021926b P e632c553ff4341328acf1cbaca678392 reported cstate change: term changed from 0 to 1, leader changed from <none> to e632c553ff4341328acf1cbaca678392 (127.15.255.133). New cstate: current_term: 1 leader_uuid: "e632c553ff4341328acf1cbaca678392" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } health_report { overall_health: UNKNOWN } } }
I20240502 21:28:53.257581 1549 raft_consensus.cc:2454] T 0658ecbf32754656876452b6f021926b P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e632c553ff4341328acf1cbaca678392 in term 1.
I20240502 21:28:53.561524 2963 consensus_queue.cc:1035] T 1aa57620996e42b6adf941e9d606f7e2 P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:28:53.579210 2963 consensus_queue.cc:1035] T b300ad3948074c728a25f053512bc37c P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:28:53.608491 2963 consensus_queue.cc:1035] T b300ad3948074c728a25f053512bc37c P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:28:53.624274 2963 consensus_queue.cc:1035] T 1aa57620996e42b6adf941e9d606f7e2 P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:28:53.629889 2963 consensus_queue.cc:1035] T 1aa57620996e42b6adf941e9d606f7e2 P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:28:53.634871 2963 consensus_queue.cc:1035] T 1aa57620996e42b6adf941e9d606f7e2 P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:28:53.636659 2963 consensus_queue.cc:1035] T 1aa57620996e42b6adf941e9d606f7e2 P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:28:53.649921 3064 consensus_queue.cc:1035] T 0658ecbf32754656876452b6f021926b P e632c553ff4341328acf1cbaca678392 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:28:53.653326 3064 consensus_queue.cc:1035] T 0658ecbf32754656876452b6f021926b P e632c553ff4341328acf1cbaca678392 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:28:53.670018 2963 consensus_queue.cc:1035] T 1aa57620996e42b6adf941e9d606f7e2 P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:28:53.673693 2963 consensus_queue.cc:1035] T b300ad3948074c728a25f053512bc37c P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 21:28:53.693431 3107 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:28:53.693805 3107 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240502 21:28:53.701224 2963 consensus_queue.cc:1035] T b300ad3948074c728a25f053512bc37c P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 21:28:53.704931 3107 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 21:28:53.717442 3116 consensus_queue.cc:1035] T b300ad3948074c728a25f053512bc37c P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:28:53.759047 3005 consensus_queue.cc:1035] T b300ad3948074c728a25f053512bc37c P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:28:53.761312 3064 consensus_queue.cc:1035] T 0658ecbf32754656876452b6f021926b P e632c553ff4341328acf1cbaca678392 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:28:53.764567 3064 consensus_queue.cc:1035] T 0658ecbf32754656876452b6f021926b P e632c553ff4341328acf1cbaca678392 [LEADER]: Connected to new peer: Peer: permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:28:53.771080 3064 consensus_queue.cc:1035] T 0658ecbf32754656876452b6f021926b P e632c553ff4341328acf1cbaca678392 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_2'
I20240502 21:28:53.790071 3155 consensus_queue.cc:1035] T 0658ecbf32754656876452b6f021926b P e632c553ff4341328acf1cbaca678392 [LEADER]: Connected to new peer: Peer: permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 21:28:54.041477 2025 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.131:36033, user_credentials={real_user=slave}} blocked reactor thread for 74760.8us
W20240502 21:28:54.058856 2024 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.132:44947, user_credentials={real_user=slave}} blocked reactor thread for 113244us
W20240502 21:28:54.082001 2023 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.130:37991, user_credentials={real_user=slave}} blocked reactor thread for 52365.6us
INSERT report
rows total: 2000
time total: 411.184 ms
time per row: 0.205592 ms
I20240502 21:28:54.318042 1266 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44934:
name: "rebalance_test_table_3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 7
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
W20240502 21:28:54.318835 1266 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table rebalance_test_table_3 in case of a server failure: 8 tablet servers would be needed, 7 are available. Consider bringing up more tablet servers.
I20240502 21:28:54.328241 2069 tablet_service.cc:1467] Processing CreateTablet for tablet f8bee9bdcbff404280f000bdaeadf465 (DEFAULT_TABLE table=rebalance_test_table_3 [id=a205513324dd4db5a5ee4ed1928aefa6]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:54.328378 1933 tablet_service.cc:1467] Processing CreateTablet for tablet f8bee9bdcbff404280f000bdaeadf465 (DEFAULT_TABLE table=rebalance_test_table_3 [id=a205513324dd4db5a5ee4ed1928aefa6]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:54.328846 1933 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8bee9bdcbff404280f000bdaeadf465. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:54.329277 2069 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8bee9bdcbff404280f000bdaeadf465. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:54.329277 1798 tablet_service.cc:1467] Processing CreateTablet for tablet f8bee9bdcbff404280f000bdaeadf465 (DEFAULT_TABLE table=rebalance_test_table_3 [id=a205513324dd4db5a5ee4ed1928aefa6]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:54.329706 1798 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8bee9bdcbff404280f000bdaeadf465. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:54.329798 1393 tablet_service.cc:1467] Processing CreateTablet for tablet f8bee9bdcbff404280f000bdaeadf465 (DEFAULT_TABLE table=rebalance_test_table_3 [id=a205513324dd4db5a5ee4ed1928aefa6]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:54.330221 1393 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8bee9bdcbff404280f000bdaeadf465. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:54.332769 1526 tablet_service.cc:1467] Processing CreateTablet for tablet f8bee9bdcbff404280f000bdaeadf465 (DEFAULT_TABLE table=rebalance_test_table_3 [id=a205513324dd4db5a5ee4ed1928aefa6]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:54.333189 1526 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8bee9bdcbff404280f000bdaeadf465. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:54.333691 1662 tablet_service.cc:1467] Processing CreateTablet for tablet f8bee9bdcbff404280f000bdaeadf465 (DEFAULT_TABLE table=rebalance_test_table_3 [id=a205513324dd4db5a5ee4ed1928aefa6]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:54.334153 1662 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8bee9bdcbff404280f000bdaeadf465. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:54.334790 2067 tablet_service.cc:1467] Processing CreateTablet for tablet bcddb146822e4573837a5697d74134f7 (DEFAULT_TABLE table=rebalance_test_table_3 [id=a205513324dd4db5a5ee4ed1928aefa6]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:54.335157 2067 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bcddb146822e4573837a5697d74134f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:54.335181 1932 tablet_service.cc:1467] Processing CreateTablet for tablet bcddb146822e4573837a5697d74134f7 (DEFAULT_TABLE table=rebalance_test_table_3 [id=a205513324dd4db5a5ee4ed1928aefa6]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:54.335551 1932 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bcddb146822e4573837a5697d74134f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:54.336861 2207 tablet_service.cc:1467] Processing CreateTablet for tablet f8bee9bdcbff404280f000bdaeadf465 (DEFAULT_TABLE table=rebalance_test_table_3 [id=a205513324dd4db5a5ee4ed1928aefa6]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:54.337285 2207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8bee9bdcbff404280f000bdaeadf465. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:54.345171 1800 tablet_service.cc:1467] Processing CreateTablet for tablet bcddb146822e4573837a5697d74134f7 (DEFAULT_TABLE table=rebalance_test_table_3 [id=a205513324dd4db5a5ee4ed1928aefa6]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:54.345541 1800 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bcddb146822e4573837a5697d74134f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:54.351415 3220 tablet_bootstrap.cc:492] T f8bee9bdcbff404280f000bdaeadf465 P e632c553ff4341328acf1cbaca678392: Bootstrap starting.
I20240502 21:28:54.351364 2206 tablet_service.cc:1467] Processing CreateTablet for tablet bcddb146822e4573837a5697d74134f7 (DEFAULT_TABLE table=rebalance_test_table_3 [id=a205513324dd4db5a5ee4ed1928aefa6]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:54.350190 3215 tablet_bootstrap.cc:492] T bcddb146822e4573837a5697d74134f7 P 521bb93035324cecbebed2f6f2a630f8: Bootstrap starting.
I20240502 21:28:54.351724 2206 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bcddb146822e4573837a5697d74134f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:54.351799 1394 tablet_service.cc:1467] Processing CreateTablet for tablet bcddb146822e4573837a5697d74134f7 (DEFAULT_TABLE table=rebalance_test_table_3 [id=a205513324dd4db5a5ee4ed1928aefa6]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:54.352195 1394 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bcddb146822e4573837a5697d74134f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:54.353168 1528 tablet_service.cc:1467] Processing CreateTablet for tablet bcddb146822e4573837a5697d74134f7 (DEFAULT_TABLE table=rebalance_test_table_3 [id=a205513324dd4db5a5ee4ed1928aefa6]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:54.353529 1528 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bcddb146822e4573837a5697d74134f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:54.354612 3220 tablet_bootstrap.cc:654] T f8bee9bdcbff404280f000bdaeadf465 P e632c553ff4341328acf1cbaca678392: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:54.357887 3215 tablet_bootstrap.cc:654] T bcddb146822e4573837a5697d74134f7 P 521bb93035324cecbebed2f6f2a630f8: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:54.361330 1664 tablet_service.cc:1467] Processing CreateTablet for tablet bcddb146822e4573837a5697d74134f7 (DEFAULT_TABLE table=rebalance_test_table_3 [id=a205513324dd4db5a5ee4ed1928aefa6]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:54.361639 1799 tablet_service.cc:1467] Processing CreateTablet for tablet 815178fbd07b4631b507927683a475c9 (DEFAULT_TABLE table=rebalance_test_table_3 [id=a205513324dd4db5a5ee4ed1928aefa6]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:54.361725 1664 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bcddb146822e4573837a5697d74134f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:54.362006 1799 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 815178fbd07b4631b507927683a475c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:54.362357 2067 tablet_service.cc:1467] Processing CreateTablet for tablet 815178fbd07b4631b507927683a475c9 (DEFAULT_TABLE table=rebalance_test_table_3 [id=a205513324dd4db5a5ee4ed1928aefa6]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:54.362780 2067 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 815178fbd07b4631b507927683a475c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:54.366829 1663 tablet_service.cc:1467] Processing CreateTablet for tablet 815178fbd07b4631b507927683a475c9 (DEFAULT_TABLE table=rebalance_test_table_3 [id=a205513324dd4db5a5ee4ed1928aefa6]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:54.367198 1663 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 815178fbd07b4631b507927683a475c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:54.368275 1394 tablet_service.cc:1467] Processing CreateTablet for tablet 815178fbd07b4631b507927683a475c9 (DEFAULT_TABLE table=rebalance_test_table_3 [id=a205513324dd4db5a5ee4ed1928aefa6]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:54.368346 2206 tablet_service.cc:1467] Processing CreateTablet for tablet 815178fbd07b4631b507927683a475c9 (DEFAULT_TABLE table=rebalance_test_table_3 [id=a205513324dd4db5a5ee4ed1928aefa6]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:54.368676 1394 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 815178fbd07b4631b507927683a475c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:54.368878 2206 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 815178fbd07b4631b507927683a475c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:54.372833 1528 tablet_service.cc:1467] Processing CreateTablet for tablet 815178fbd07b4631b507927683a475c9 (DEFAULT_TABLE table=rebalance_test_table_3 [id=a205513324dd4db5a5ee4ed1928aefa6]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:54.373234 1528 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 815178fbd07b4631b507927683a475c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:54.376255 1933 tablet_service.cc:1467] Processing CreateTablet for tablet 815178fbd07b4631b507927683a475c9 (DEFAULT_TABLE table=rebalance_test_table_3 [id=a205513324dd4db5a5ee4ed1928aefa6]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:54.376675 1933 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 815178fbd07b4631b507927683a475c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:54.383795 3215 tablet_bootstrap.cc:492] T bcddb146822e4573837a5697d74134f7 P 521bb93035324cecbebed2f6f2a630f8: No bootstrap required, opened a new log
I20240502 21:28:54.383984 3215 ts_tablet_manager.cc:1367] T bcddb146822e4573837a5697d74134f7 P 521bb93035324cecbebed2f6f2a630f8: Time spent bootstrapping tablet: real 0.034s user 0.003s sys 0.007s
I20240502 21:28:54.384400 3219 tablet_bootstrap.cc:492] T f8bee9bdcbff404280f000bdaeadf465 P 6ab8df3c796f4b6cb3e29d5045ad3ea0: Bootstrap starting.
I20240502 21:28:54.384936 3215 raft_consensus.cc:348] T bcddb146822e4573837a5697d74134f7 P 521bb93035324cecbebed2f6f2a630f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:28:54.385351 3215 raft_consensus.cc:374] T bcddb146822e4573837a5697d74134f7 P 521bb93035324cecbebed2f6f2a630f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:54.385488 3215 raft_consensus.cc:729] T bcddb146822e4573837a5697d74134f7 P 521bb93035324cecbebed2f6f2a630f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 521bb93035324cecbebed2f6f2a630f8, State: Initialized, Role: FOLLOWER
I20240502 21:28:54.385777 3215 consensus_queue.cc:260] T bcddb146822e4573837a5697d74134f7 P 521bb93035324cecbebed2f6f2a630f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:28:54.386044 3219 tablet_bootstrap.cc:654] T f8bee9bdcbff404280f000bdaeadf465 P 6ab8df3c796f4b6cb3e29d5045ad3ea0: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:54.387213 3215 ts_tablet_manager.cc:1398] T bcddb146822e4573837a5697d74134f7 P 521bb93035324cecbebed2f6f2a630f8: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240502 21:28:54.387569 3215 tablet_bootstrap.cc:492] T f8bee9bdcbff404280f000bdaeadf465 P 521bb93035324cecbebed2f6f2a630f8: Bootstrap starting.
I20240502 21:28:54.389519 3218 tablet_bootstrap.cc:492] T f8bee9bdcbff404280f000bdaeadf465 P a62a935ff43d494ca35d0c417943133d: Bootstrap starting.
I20240502 21:28:54.390405 3215 tablet_bootstrap.cc:654] T f8bee9bdcbff404280f000bdaeadf465 P 521bb93035324cecbebed2f6f2a630f8: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:54.390619 3217 tablet_bootstrap.cc:492] T f8bee9bdcbff404280f000bdaeadf465 P 16b3a337bc5e462f83ba895ad8660deb: Bootstrap starting.
I20240502 21:28:54.390838 3223 tablet_bootstrap.cc:492] T f8bee9bdcbff404280f000bdaeadf465 P 153026dd7f8943cc93fedf6f48450fb4: Bootstrap starting.
I20240502 21:28:54.391481 3216 tablet_bootstrap.cc:492] T f8bee9bdcbff404280f000bdaeadf465 P 7d1e5c1b047443cbb28d7b0a9744babf: Bootstrap starting.
I20240502 21:28:54.392707 3223 tablet_bootstrap.cc:654] T f8bee9bdcbff404280f000bdaeadf465 P 153026dd7f8943cc93fedf6f48450fb4: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:54.393649 3218 tablet_bootstrap.cc:654] T f8bee9bdcbff404280f000bdaeadf465 P a62a935ff43d494ca35d0c417943133d: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:54.394346 3217 tablet_bootstrap.cc:654] T f8bee9bdcbff404280f000bdaeadf465 P 16b3a337bc5e462f83ba895ad8660deb: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:54.394680 3219 tablet_bootstrap.cc:492] T f8bee9bdcbff404280f000bdaeadf465 P 6ab8df3c796f4b6cb3e29d5045ad3ea0: No bootstrap required, opened a new log
I20240502 21:28:54.394863 3219 ts_tablet_manager.cc:1367] T f8bee9bdcbff404280f000bdaeadf465 P 6ab8df3c796f4b6cb3e29d5045ad3ea0: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.005s
I20240502 21:28:54.395654 3219 raft_consensus.cc:348] T f8bee9bdcbff404280f000bdaeadf465 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:28:54.396762 3220 tablet_bootstrap.cc:492] T f8bee9bdcbff404280f000bdaeadf465 P e632c553ff4341328acf1cbaca678392: No bootstrap required, opened a new log
I20240502 21:28:54.397018 3220 ts_tablet_manager.cc:1367] T f8bee9bdcbff404280f000bdaeadf465 P e632c553ff4341328acf1cbaca678392: Time spent bootstrapping tablet: real 0.046s user 0.001s sys 0.012s
I20240502 21:28:54.397830 3220 raft_consensus.cc:348] T f8bee9bdcbff404280f000bdaeadf465 P e632c553ff4341328acf1cbaca678392 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:28:54.398152 3220 raft_consensus.cc:374] T f8bee9bdcbff404280f000bdaeadf465 P e632c553ff4341328acf1cbaca678392 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:54.398270 3220 raft_consensus.cc:729] T f8bee9bdcbff404280f000bdaeadf465 P e632c553ff4341328acf1cbaca678392 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e632c553ff4341328acf1cbaca678392, State: Initialized, Role: FOLLOWER
I20240502 21:28:54.398399 3216 tablet_bootstrap.cc:654] T f8bee9bdcbff404280f000bdaeadf465 P 7d1e5c1b047443cbb28d7b0a9744babf: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:54.398597 3220 consensus_queue.cc:260] T f8bee9bdcbff404280f000bdaeadf465 P e632c553ff4341328acf1cbaca678392 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:28:54.399436 3220 ts_tablet_manager.cc:1398] T f8bee9bdcbff404280f000bdaeadf465 P e632c553ff4341328acf1cbaca678392: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:28:54.399825 3220 tablet_bootstrap.cc:492] T bcddb146822e4573837a5697d74134f7 P e632c553ff4341328acf1cbaca678392: Bootstrap starting.
I20240502 21:28:54.396091 3219 raft_consensus.cc:374] T f8bee9bdcbff404280f000bdaeadf465 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:54.400020 3219 raft_consensus.cc:729] T f8bee9bdcbff404280f000bdaeadf465 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ab8df3c796f4b6cb3e29d5045ad3ea0, State: Initialized, Role: FOLLOWER
I20240502 21:28:54.401602 3220 tablet_bootstrap.cc:654] T bcddb146822e4573837a5697d74134f7 P e632c553ff4341328acf1cbaca678392: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:54.402069 3215 tablet_bootstrap.cc:492] T f8bee9bdcbff404280f000bdaeadf465 P 521bb93035324cecbebed2f6f2a630f8: No bootstrap required, opened a new log
I20240502 21:28:54.401870 3219 consensus_queue.cc:260] T f8bee9bdcbff404280f000bdaeadf465 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:28:54.402309 3215 ts_tablet_manager.cc:1367] T f8bee9bdcbff404280f000bdaeadf465 P 521bb93035324cecbebed2f6f2a630f8: Time spent bootstrapping tablet: real 0.015s user 0.000s sys 0.006s
I20240502 21:28:54.403771 3215 raft_consensus.cc:348] T f8bee9bdcbff404280f000bdaeadf465 P 521bb93035324cecbebed2f6f2a630f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:28:54.404103 3215 raft_consensus.cc:374] T f8bee9bdcbff404280f000bdaeadf465 P 521bb93035324cecbebed2f6f2a630f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:54.404196 3215 raft_consensus.cc:729] T f8bee9bdcbff404280f000bdaeadf465 P 521bb93035324cecbebed2f6f2a630f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 521bb93035324cecbebed2f6f2a630f8, State: Initialized, Role: FOLLOWER
I20240502 21:28:54.404529 3215 consensus_queue.cc:260] T f8bee9bdcbff404280f000bdaeadf465 P 521bb93035324cecbebed2f6f2a630f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:28:54.405159 3223 tablet_bootstrap.cc:492] T f8bee9bdcbff404280f000bdaeadf465 P 153026dd7f8943cc93fedf6f48450fb4: No bootstrap required, opened a new log
I20240502 21:28:54.405303 3223 ts_tablet_manager.cc:1367] T f8bee9bdcbff404280f000bdaeadf465 P 153026dd7f8943cc93fedf6f48450fb4: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.002s
I20240502 21:28:54.405817 3215 ts_tablet_manager.cc:1398] T f8bee9bdcbff404280f000bdaeadf465 P 521bb93035324cecbebed2f6f2a630f8: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 21:28:54.406134 3215 tablet_bootstrap.cc:492] T 815178fbd07b4631b507927683a475c9 P 521bb93035324cecbebed2f6f2a630f8: Bootstrap starting.
I20240502 21:28:54.406035 3223 raft_consensus.cc:348] T f8bee9bdcbff404280f000bdaeadf465 P 153026dd7f8943cc93fedf6f48450fb4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:28:54.406347 3223 raft_consensus.cc:374] T f8bee9bdcbff404280f000bdaeadf465 P 153026dd7f8943cc93fedf6f48450fb4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:54.406453 3223 raft_consensus.cc:729] T f8bee9bdcbff404280f000bdaeadf465 P 153026dd7f8943cc93fedf6f48450fb4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 153026dd7f8943cc93fedf6f48450fb4, State: Initialized, Role: FOLLOWER
I20240502 21:28:54.406682 3223 consensus_queue.cc:260] T f8bee9bdcbff404280f000bdaeadf465 P 153026dd7f8943cc93fedf6f48450fb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:28:54.407407 3218 tablet_bootstrap.cc:492] T f8bee9bdcbff404280f000bdaeadf465 P a62a935ff43d494ca35d0c417943133d: No bootstrap required, opened a new log
I20240502 21:28:54.407569 3218 ts_tablet_manager.cc:1367] T f8bee9bdcbff404280f000bdaeadf465 P a62a935ff43d494ca35d0c417943133d: Time spent bootstrapping tablet: real 0.018s user 0.005s sys 0.000s
I20240502 21:28:54.407827 3215 tablet_bootstrap.cc:654] T 815178fbd07b4631b507927683a475c9 P 521bb93035324cecbebed2f6f2a630f8: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:54.408370 3218 raft_consensus.cc:348] T f8bee9bdcbff404280f000bdaeadf465 P a62a935ff43d494ca35d0c417943133d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:28:54.408670 3218 raft_consensus.cc:374] T f8bee9bdcbff404280f000bdaeadf465 P a62a935ff43d494ca35d0c417943133d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:54.408771 3218 raft_consensus.cc:729] T f8bee9bdcbff404280f000bdaeadf465 P a62a935ff43d494ca35d0c417943133d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a62a935ff43d494ca35d0c417943133d, State: Initialized, Role: FOLLOWER
I20240502 21:28:54.409041 3218 consensus_queue.cc:260] T f8bee9bdcbff404280f000bdaeadf465 P a62a935ff43d494ca35d0c417943133d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:28:54.409822 3217 tablet_bootstrap.cc:492] T f8bee9bdcbff404280f000bdaeadf465 P 16b3a337bc5e462f83ba895ad8660deb: No bootstrap required, opened a new log
I20240502 21:28:54.410025 3217 ts_tablet_manager.cc:1367] T f8bee9bdcbff404280f000bdaeadf465 P 16b3a337bc5e462f83ba895ad8660deb: Time spent bootstrapping tablet: real 0.020s user 0.003s sys 0.003s
I20240502 21:28:54.411041 3218 ts_tablet_manager.cc:1398] T f8bee9bdcbff404280f000bdaeadf465 P a62a935ff43d494ca35d0c417943133d: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 21:28:54.411031 3217 raft_consensus.cc:348] T f8bee9bdcbff404280f000bdaeadf465 P 16b3a337bc5e462f83ba895ad8660deb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:28:54.411442 3217 raft_consensus.cc:374] T f8bee9bdcbff404280f000bdaeadf465 P 16b3a337bc5e462f83ba895ad8660deb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:54.411567 3217 raft_consensus.cc:729] T f8bee9bdcbff404280f000bdaeadf465 P 16b3a337bc5e462f83ba895ad8660deb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16b3a337bc5e462f83ba895ad8660deb, State: Initialized, Role: FOLLOWER
I20240502 21:28:54.411775 3223 ts_tablet_manager.cc:1398] T f8bee9bdcbff404280f000bdaeadf465 P 153026dd7f8943cc93fedf6f48450fb4: Time spent starting tablet: real 0.006s user 0.002s sys 0.001s
I20240502 21:28:54.411902 3217 consensus_queue.cc:260] T f8bee9bdcbff404280f000bdaeadf465 P 16b3a337bc5e462f83ba895ad8660deb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:28:54.412088 3223 tablet_bootstrap.cc:492] T bcddb146822e4573837a5697d74134f7 P 153026dd7f8943cc93fedf6f48450fb4: Bootstrap starting.
I20240502 21:28:54.413552 3223 tablet_bootstrap.cc:654] T bcddb146822e4573837a5697d74134f7 P 153026dd7f8943cc93fedf6f48450fb4: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:54.414539 3218 tablet_bootstrap.cc:492] T bcddb146822e4573837a5697d74134f7 P a62a935ff43d494ca35d0c417943133d: Bootstrap starting.
I20240502 21:28:54.416267 3218 tablet_bootstrap.cc:654] T bcddb146822e4573837a5697d74134f7 P a62a935ff43d494ca35d0c417943133d: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:54.418231 3215 tablet_bootstrap.cc:492] T 815178fbd07b4631b507927683a475c9 P 521bb93035324cecbebed2f6f2a630f8: No bootstrap required, opened a new log
I20240502 21:28:54.418385 3216 tablet_bootstrap.cc:492] T f8bee9bdcbff404280f000bdaeadf465 P 7d1e5c1b047443cbb28d7b0a9744babf: No bootstrap required, opened a new log
I20240502 21:28:54.418553 3216 ts_tablet_manager.cc:1367] T f8bee9bdcbff404280f000bdaeadf465 P 7d1e5c1b047443cbb28d7b0a9744babf: Time spent bootstrapping tablet: real 0.030s user 0.000s sys 0.003s
I20240502 21:28:54.419046 3219 ts_tablet_manager.cc:1398] T f8bee9bdcbff404280f000bdaeadf465 P 6ab8df3c796f4b6cb3e29d5045ad3ea0: Time spent starting tablet: real 0.024s user 0.000s sys 0.009s
I20240502 21:28:54.419276 3215 ts_tablet_manager.cc:1367] T 815178fbd07b4631b507927683a475c9 P 521bb93035324cecbebed2f6f2a630f8: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.003s
I20240502 21:28:54.419401 3216 raft_consensus.cc:348] T f8bee9bdcbff404280f000bdaeadf465 P 7d1e5c1b047443cbb28d7b0a9744babf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:28:54.419687 3216 raft_consensus.cc:374] T f8bee9bdcbff404280f000bdaeadf465 P 7d1e5c1b047443cbb28d7b0a9744babf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:54.419770 3216 raft_consensus.cc:729] T f8bee9bdcbff404280f000bdaeadf465 P 7d1e5c1b047443cbb28d7b0a9744babf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d1e5c1b047443cbb28d7b0a9744babf, State: Initialized, Role: FOLLOWER
I20240502 21:28:54.419921 3215 raft_consensus.cc:348] T 815178fbd07b4631b507927683a475c9 P 521bb93035324cecbebed2f6f2a630f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:28:54.420125 3215 raft_consensus.cc:374] T 815178fbd07b4631b507927683a475c9 P 521bb93035324cecbebed2f6f2a630f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:54.420015 3216 consensus_queue.cc:260] T f8bee9bdcbff404280f000bdaeadf465 P 7d1e5c1b047443cbb28d7b0a9744babf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:28:54.420219 3215 raft_consensus.cc:729] T 815178fbd07b4631b507927683a475c9 P 521bb93035324cecbebed2f6f2a630f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 521bb93035324cecbebed2f6f2a630f8, State: Initialized, Role: FOLLOWER
I20240502 21:28:54.420469 3215 consensus_queue.cc:260] T 815178fbd07b4631b507927683a475c9 P 521bb93035324cecbebed2f6f2a630f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:28:54.420706 3216 ts_tablet_manager.cc:1398] T f8bee9bdcbff404280f000bdaeadf465 P 7d1e5c1b047443cbb28d7b0a9744babf: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20240502 21:28:54.421120 3216 tablet_bootstrap.cc:492] T bcddb146822e4573837a5697d74134f7 P 7d1e5c1b047443cbb28d7b0a9744babf: Bootstrap starting.
I20240502 21:28:54.421842 3219 tablet_bootstrap.cc:492] T bcddb146822e4573837a5697d74134f7 P 6ab8df3c796f4b6cb3e29d5045ad3ea0: Bootstrap starting.
I20240502 21:28:54.423452 3219 tablet_bootstrap.cc:654] T bcddb146822e4573837a5697d74134f7 P 6ab8df3c796f4b6cb3e29d5045ad3ea0: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:54.425071 3220 tablet_bootstrap.cc:492] T bcddb146822e4573837a5697d74134f7 P e632c553ff4341328acf1cbaca678392: No bootstrap required, opened a new log
I20240502 21:28:54.425319 3220 ts_tablet_manager.cc:1367] T bcddb146822e4573837a5697d74134f7 P e632c553ff4341328acf1cbaca678392: Time spent bootstrapping tablet: real 0.026s user 0.004s sys 0.007s
I20240502 21:28:54.426905 3217 ts_tablet_manager.cc:1398] T f8bee9bdcbff404280f000bdaeadf465 P 16b3a337bc5e462f83ba895ad8660deb: Time spent starting tablet: real 0.017s user 0.001s sys 0.003s
I20240502 21:28:54.427206 3216 tablet_bootstrap.cc:654] T bcddb146822e4573837a5697d74134f7 P 7d1e5c1b047443cbb28d7b0a9744babf: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:54.427349 3220 raft_consensus.cc:348] T bcddb146822e4573837a5697d74134f7 P e632c553ff4341328acf1cbaca678392 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:28:54.427704 3220 raft_consensus.cc:374] T bcddb146822e4573837a5697d74134f7 P e632c553ff4341328acf1cbaca678392 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:54.427870 3220 raft_consensus.cc:729] T bcddb146822e4573837a5697d74134f7 P e632c553ff4341328acf1cbaca678392 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e632c553ff4341328acf1cbaca678392, State: Initialized, Role: FOLLOWER
I20240502 21:28:54.427925 3217 tablet_bootstrap.cc:492] T bcddb146822e4573837a5697d74134f7 P 16b3a337bc5e462f83ba895ad8660deb: Bootstrap starting.
I20240502 21:28:54.428200 3220 consensus_queue.cc:260] T bcddb146822e4573837a5697d74134f7 P e632c553ff4341328acf1cbaca678392 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:28:54.428905 3220 ts_tablet_manager.cc:1398] T bcddb146822e4573837a5697d74134f7 P e632c553ff4341328acf1cbaca678392: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 21:28:54.429352 3220 tablet_bootstrap.cc:492] T 815178fbd07b4631b507927683a475c9 P e632c553ff4341328acf1cbaca678392: Bootstrap starting.
I20240502 21:28:54.430558 3217 tablet_bootstrap.cc:654] T bcddb146822e4573837a5697d74134f7 P 16b3a337bc5e462f83ba895ad8660deb: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:54.430845 3220 tablet_bootstrap.cc:654] T 815178fbd07b4631b507927683a475c9 P e632c553ff4341328acf1cbaca678392: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:54.434703 3215 ts_tablet_manager.cc:1398] T 815178fbd07b4631b507927683a475c9 P 521bb93035324cecbebed2f6f2a630f8: Time spent starting tablet: real 0.015s user 0.002s sys 0.000s
I20240502 21:28:54.441743 3220 tablet_bootstrap.cc:492] T 815178fbd07b4631b507927683a475c9 P e632c553ff4341328acf1cbaca678392: No bootstrap required, opened a new log
I20240502 21:28:54.441910 3220 ts_tablet_manager.cc:1367] T 815178fbd07b4631b507927683a475c9 P e632c553ff4341328acf1cbaca678392: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.001s
I20240502 21:28:54.442657 3220 raft_consensus.cc:348] T 815178fbd07b4631b507927683a475c9 P e632c553ff4341328acf1cbaca678392 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:28:54.442883 3220 raft_consensus.cc:374] T 815178fbd07b4631b507927683a475c9 P e632c553ff4341328acf1cbaca678392 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:54.442981 3220 raft_consensus.cc:729] T 815178fbd07b4631b507927683a475c9 P e632c553ff4341328acf1cbaca678392 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e632c553ff4341328acf1cbaca678392, State: Initialized, Role: FOLLOWER
I20240502 21:28:54.443207 3220 consensus_queue.cc:260] T 815178fbd07b4631b507927683a475c9 P e632c553ff4341328acf1cbaca678392 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:28:54.443703 3220 ts_tablet_manager.cc:1398] T 815178fbd07b4631b507927683a475c9 P e632c553ff4341328acf1cbaca678392: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 21:28:54.452836 3218 tablet_bootstrap.cc:492] T bcddb146822e4573837a5697d74134f7 P a62a935ff43d494ca35d0c417943133d: No bootstrap required, opened a new log
I20240502 21:28:54.453006 3218 ts_tablet_manager.cc:1367] T bcddb146822e4573837a5697d74134f7 P a62a935ff43d494ca35d0c417943133d: Time spent bootstrapping tablet: real 0.039s user 0.002s sys 0.007s
I20240502 21:28:54.453603 3218 raft_consensus.cc:348] T bcddb146822e4573837a5697d74134f7 P a62a935ff43d494ca35d0c417943133d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:28:54.453816 3218 raft_consensus.cc:374] T bcddb146822e4573837a5697d74134f7 P a62a935ff43d494ca35d0c417943133d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:54.453888 3218 raft_consensus.cc:729] T bcddb146822e4573837a5697d74134f7 P a62a935ff43d494ca35d0c417943133d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a62a935ff43d494ca35d0c417943133d, State: Initialized, Role: FOLLOWER
I20240502 21:28:54.454120 3218 consensus_queue.cc:260] T bcddb146822e4573837a5697d74134f7 P a62a935ff43d494ca35d0c417943133d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:28:54.454649 3218 ts_tablet_manager.cc:1398] T bcddb146822e4573837a5697d74134f7 P a62a935ff43d494ca35d0c417943133d: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 21:28:54.455097 3218 tablet_bootstrap.cc:492] T 815178fbd07b4631b507927683a475c9 P a62a935ff43d494ca35d0c417943133d: Bootstrap starting.
I20240502 21:28:54.456307 3218 tablet_bootstrap.cc:654] T 815178fbd07b4631b507927683a475c9 P a62a935ff43d494ca35d0c417943133d: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:54.457826 3149 raft_consensus.cc:482] T bcddb146822e4573837a5697d74134f7 P e632c553ff4341328acf1cbaca678392 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:28:54.457976 3149 raft_consensus.cc:504] T bcddb146822e4573837a5697d74134f7 P e632c553ff4341328acf1cbaca678392 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:28:54.459184 3149 leader_election.cc:288] T bcddb146822e4573837a5697d74134f7 P e632c553ff4341328acf1cbaca678392 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 521bb93035324cecbebed2f6f2a630f8 (127.15.255.134:40821), 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132:44947), 6ab8df3c796f4b6cb3e29d5045ad3ea0 (127.15.255.135:34701), 7d1e5c1b047443cbb28d7b0a9744babf (127.15.255.129:37897), a62a935ff43d494ca35d0c417943133d (127.15.255.130:37991), 16b3a337bc5e462f83ba895ad8660deb (127.15.255.131:36033)
I20240502 21:28:54.459363 1818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcddb146822e4573837a5697d74134f7" candidate_uuid: "e632c553ff4341328acf1cbaca678392" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "153026dd7f8943cc93fedf6f48450fb4" is_pre_election: true
I20240502 21:28:54.459710 1686 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcddb146822e4573837a5697d74134f7" candidate_uuid: "e632c553ff4341328acf1cbaca678392" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16b3a337bc5e462f83ba895ad8660deb" is_pre_election: true
I20240502 21:28:54.459906 2089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcddb146822e4573837a5697d74134f7" candidate_uuid: "e632c553ff4341328acf1cbaca678392" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "521bb93035324cecbebed2f6f2a630f8" is_pre_election: true
I20240502 21:28:54.460114 2089 raft_consensus.cc:2454] T bcddb146822e4573837a5697d74134f7 P 521bb93035324cecbebed2f6f2a630f8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e632c553ff4341328acf1cbaca678392 in term 0.
W20240502 21:28:54.460368 1886 leader_election.cc:341] T bcddb146822e4573837a5697d74134f7 P e632c553ff4341328acf1cbaca678392 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132:44947): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:28:54.460911 3217 tablet_bootstrap.cc:492] T bcddb146822e4573837a5697d74134f7 P 16b3a337bc5e462f83ba895ad8660deb: No bootstrap required, opened a new log
W20240502 21:28:54.460882 1887 leader_election.cc:341] T bcddb146822e4573837a5697d74134f7 P e632c553ff4341328acf1cbaca678392 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 16b3a337bc5e462f83ba895ad8660deb (127.15.255.131:36033): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:28:54.460945 3216 tablet_bootstrap.cc:492] T bcddb146822e4573837a5697d74134f7 P 7d1e5c1b047443cbb28d7b0a9744babf: No bootstrap required, opened a new log
I20240502 21:28:54.461095 3217 ts_tablet_manager.cc:1367] T bcddb146822e4573837a5697d74134f7 P 16b3a337bc5e462f83ba895ad8660deb: Time spent bootstrapping tablet: real 0.033s user 0.003s sys 0.003s
I20240502 21:28:54.461140 3216 ts_tablet_manager.cc:1367] T bcddb146822e4573837a5697d74134f7 P 7d1e5c1b047443cbb28d7b0a9744babf: Time spent bootstrapping tablet: real 0.040s user 0.003s sys 0.000s
I20240502 21:28:54.461314 1415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcddb146822e4573837a5697d74134f7" candidate_uuid: "e632c553ff4341328acf1cbaca678392" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" is_pre_election: true
I20240502 21:28:54.461549 2227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcddb146822e4573837a5697d74134f7" candidate_uuid: "e632c553ff4341328acf1cbaca678392" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" is_pre_election: true
I20240502 21:28:54.461861 3217 raft_consensus.cc:348] T bcddb146822e4573837a5697d74134f7 P 16b3a337bc5e462f83ba895ad8660deb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:28:54.462038 1550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcddb146822e4573837a5697d74134f7" candidate_uuid: "e632c553ff4341328acf1cbaca678392" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a62a935ff43d494ca35d0c417943133d" is_pre_election: true
I20240502 21:28:54.461905 3216 raft_consensus.cc:348] T bcddb146822e4573837a5697d74134f7 P 7d1e5c1b047443cbb28d7b0a9744babf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:28:54.462145 3217 raft_consensus.cc:374] T bcddb146822e4573837a5697d74134f7 P 16b3a337bc5e462f83ba895ad8660deb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:54.462199 3216 raft_consensus.cc:374] T bcddb146822e4573837a5697d74134f7 P 7d1e5c1b047443cbb28d7b0a9744babf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:54.462224 3217 raft_consensus.cc:729] T bcddb146822e4573837a5697d74134f7 P 16b3a337bc5e462f83ba895ad8660deb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16b3a337bc5e462f83ba895ad8660deb, State: Initialized, Role: FOLLOWER
I20240502 21:28:54.462217 1550 raft_consensus.cc:2454] T bcddb146822e4573837a5697d74134f7 P a62a935ff43d494ca35d0c417943133d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e632c553ff4341328acf1cbaca678392 in term 0.
I20240502 21:28:54.462273 3216 raft_consensus.cc:729] T bcddb146822e4573837a5697d74134f7 P 7d1e5c1b047443cbb28d7b0a9744babf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d1e5c1b047443cbb28d7b0a9744babf, State: Initialized, Role: FOLLOWER
I20240502 21:28:54.462479 3217 consensus_queue.cc:260] T bcddb146822e4573837a5697d74134f7 P 16b3a337bc5e462f83ba895ad8660deb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:28:54.462550 3216 consensus_queue.cc:260] T bcddb146822e4573837a5697d74134f7 P 7d1e5c1b047443cbb28d7b0a9744babf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
W20240502 21:28:54.462842 1885 leader_election.cc:341] T bcddb146822e4573837a5697d74134f7 P e632c553ff4341328acf1cbaca678392 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6ab8df3c796f4b6cb3e29d5045ad3ea0 (127.15.255.135:34701): Illegal state: must be running to vote when last-logged opid is not known
W20240502 21:28:54.463104 1885 leader_election.cc:341] T bcddb146822e4573837a5697d74134f7 P e632c553ff4341328acf1cbaca678392 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7d1e5c1b047443cbb28d7b0a9744babf (127.15.255.129:37897): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:28:54.463194 1885 leader_election.cc:302] T bcddb146822e4573837a5697d74134f7 P e632c553ff4341328acf1cbaca678392 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 7 responses out of 7 voters: 3 yes votes; 4 no votes. yes voters: 521bb93035324cecbebed2f6f2a630f8, a62a935ff43d494ca35d0c417943133d, e632c553ff4341328acf1cbaca678392; no voters: 153026dd7f8943cc93fedf6f48450fb4, 16b3a337bc5e462f83ba895ad8660deb, 6ab8df3c796f4b6cb3e29d5045ad3ea0, 7d1e5c1b047443cbb28d7b0a9744babf
I20240502 21:28:54.463716 3149 raft_consensus.cc:2726] T bcddb146822e4573837a5697d74134f7 P e632c553ff4341328acf1cbaca678392 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 21:28:54.463852 3217 ts_tablet_manager.cc:1398] T bcddb146822e4573837a5697d74134f7 P 16b3a337bc5e462f83ba895ad8660deb: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240502 21:28:54.464150 3217 tablet_bootstrap.cc:492] T 815178fbd07b4631b507927683a475c9 P 16b3a337bc5e462f83ba895ad8660deb: Bootstrap starting.
I20240502 21:28:54.464829 3218 tablet_bootstrap.cc:492] T 815178fbd07b4631b507927683a475c9 P a62a935ff43d494ca35d0c417943133d: No bootstrap required, opened a new log
I20240502 21:28:54.465190 3216 ts_tablet_manager.cc:1398] T bcddb146822e4573837a5697d74134f7 P 7d1e5c1b047443cbb28d7b0a9744babf: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 21:28:54.465929 3223 tablet_bootstrap.cc:492] T bcddb146822e4573837a5697d74134f7 P 153026dd7f8943cc93fedf6f48450fb4: No bootstrap required, opened a new log
I20240502 21:28:54.466073 3219 tablet_bootstrap.cc:492] T bcddb146822e4573837a5697d74134f7 P 6ab8df3c796f4b6cb3e29d5045ad3ea0: No bootstrap required, opened a new log
I20240502 21:28:54.466218 3218 ts_tablet_manager.cc:1367] T 815178fbd07b4631b507927683a475c9 P a62a935ff43d494ca35d0c417943133d: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.000s
I20240502 21:28:54.466267 3219 ts_tablet_manager.cc:1367] T bcddb146822e4573837a5697d74134f7 P 6ab8df3c796f4b6cb3e29d5045ad3ea0: Time spent bootstrapping tablet: real 0.044s user 0.000s sys 0.008s
I20240502 21:28:54.467069 3223 ts_tablet_manager.cc:1367] T bcddb146822e4573837a5697d74134f7 P 153026dd7f8943cc93fedf6f48450fb4: Time spent bootstrapping tablet: real 0.055s user 0.005s sys 0.008s
I20240502 21:28:54.466975 3218 raft_consensus.cc:348] T 815178fbd07b4631b507927683a475c9 P a62a935ff43d494ca35d0c417943133d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:28:54.467021 3219 raft_consensus.cc:348] T bcddb146822e4573837a5697d74134f7 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:28:54.467235 3218 raft_consensus.cc:374] T 815178fbd07b4631b507927683a475c9 P a62a935ff43d494ca35d0c417943133d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:54.467342 3218 raft_consensus.cc:729] T 815178fbd07b4631b507927683a475c9 P a62a935ff43d494ca35d0c417943133d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a62a935ff43d494ca35d0c417943133d, State: Initialized, Role: FOLLOWER
I20240502 21:28:54.467478 3216 tablet_bootstrap.cc:492] T 815178fbd07b4631b507927683a475c9 P 7d1e5c1b047443cbb28d7b0a9744babf: Bootstrap starting.
I20240502 21:28:54.467615 3218 consensus_queue.cc:260] T 815178fbd07b4631b507927683a475c9 P a62a935ff43d494ca35d0c417943133d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:28:54.468457 3217 tablet_bootstrap.cc:654] T 815178fbd07b4631b507927683a475c9 P 16b3a337bc5e462f83ba895ad8660deb: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:54.469115 3218 ts_tablet_manager.cc:1398] T 815178fbd07b4631b507927683a475c9 P a62a935ff43d494ca35d0c417943133d: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 21:28:54.469153 3216 tablet_bootstrap.cc:654] T 815178fbd07b4631b507927683a475c9 P 7d1e5c1b047443cbb28d7b0a9744babf: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:54.469637 3219 raft_consensus.cc:374] T bcddb146822e4573837a5697d74134f7 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:54.469779 3219 raft_consensus.cc:729] T bcddb146822e4573837a5697d74134f7 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ab8df3c796f4b6cb3e29d5045ad3ea0, State: Initialized, Role: FOLLOWER
I20240502 21:28:54.470105 3219 consensus_queue.cc:260] T bcddb146822e4573837a5697d74134f7 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:28:54.470775 3219 ts_tablet_manager.cc:1398] T bcddb146822e4573837a5697d74134f7 P 6ab8df3c796f4b6cb3e29d5045ad3ea0: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240502 21:28:54.469115 3223 raft_consensus.cc:348] T bcddb146822e4573837a5697d74134f7 P 153026dd7f8943cc93fedf6f48450fb4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:28:54.471292 3223 raft_consensus.cc:374] T bcddb146822e4573837a5697d74134f7 P 153026dd7f8943cc93fedf6f48450fb4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:54.471429 3223 raft_consensus.cc:729] T bcddb146822e4573837a5697d74134f7 P 153026dd7f8943cc93fedf6f48450fb4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 153026dd7f8943cc93fedf6f48450fb4, State: Initialized, Role: FOLLOWER
I20240502 21:28:54.471635 3219 tablet_bootstrap.cc:492] T 815178fbd07b4631b507927683a475c9 P 6ab8df3c796f4b6cb3e29d5045ad3ea0: Bootstrap starting.
I20240502 21:28:54.471751 3223 consensus_queue.cc:260] T bcddb146822e4573837a5697d74134f7 P 153026dd7f8943cc93fedf6f48450fb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:28:54.472585 3223 ts_tablet_manager.cc:1398] T bcddb146822e4573837a5697d74134f7 P 153026dd7f8943cc93fedf6f48450fb4: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 21:28:54.472926 3223 tablet_bootstrap.cc:492] T 815178fbd07b4631b507927683a475c9 P 153026dd7f8943cc93fedf6f48450fb4: Bootstrap starting.
I20240502 21:28:54.474061 3219 tablet_bootstrap.cc:654] T 815178fbd07b4631b507927683a475c9 P 6ab8df3c796f4b6cb3e29d5045ad3ea0: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:54.475085 3223 tablet_bootstrap.cc:654] T 815178fbd07b4631b507927683a475c9 P 153026dd7f8943cc93fedf6f48450fb4: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:54.477840 3216 tablet_bootstrap.cc:492] T 815178fbd07b4631b507927683a475c9 P 7d1e5c1b047443cbb28d7b0a9744babf: No bootstrap required, opened a new log
I20240502 21:28:54.477996 3216 ts_tablet_manager.cc:1367] T 815178fbd07b4631b507927683a475c9 P 7d1e5c1b047443cbb28d7b0a9744babf: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.003s
I20240502 21:28:54.478660 3122 raft_consensus.cc:482] T f8bee9bdcbff404280f000bdaeadf465 P 7d1e5c1b047443cbb28d7b0a9744babf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:28:54.478767 3216 raft_consensus.cc:348] T 815178fbd07b4631b507927683a475c9 P 7d1e5c1b047443cbb28d7b0a9744babf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:28:54.479048 3216 raft_consensus.cc:374] T 815178fbd07b4631b507927683a475c9 P 7d1e5c1b047443cbb28d7b0a9744babf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:54.478842 3122 raft_consensus.cc:504] T f8bee9bdcbff404280f000bdaeadf465 P 7d1e5c1b047443cbb28d7b0a9744babf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:28:54.479154 3216 raft_consensus.cc:729] T 815178fbd07b4631b507927683a475c9 P 7d1e5c1b047443cbb28d7b0a9744babf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d1e5c1b047443cbb28d7b0a9744babf, State: Initialized, Role: FOLLOWER
I20240502 21:28:54.479452 3216 consensus_queue.cc:260] T 815178fbd07b4631b507927683a475c9 P 7d1e5c1b047443cbb28d7b0a9744babf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:28:54.479676 3217 tablet_bootstrap.cc:492] T 815178fbd07b4631b507927683a475c9 P 16b3a337bc5e462f83ba895ad8660deb: No bootstrap required, opened a new log
I20240502 21:28:54.479806 3217 ts_tablet_manager.cc:1367] T 815178fbd07b4631b507927683a475c9 P 16b3a337bc5e462f83ba895ad8660deb: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.000s
I20240502 21:28:54.480255 3122 leader_election.cc:288] T f8bee9bdcbff404280f000bdaeadf465 P 7d1e5c1b047443cbb28d7b0a9744babf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132:44947), 521bb93035324cecbebed2f6f2a630f8 (127.15.255.134:40821), e632c553ff4341328acf1cbaca678392 (127.15.255.133:37439), 6ab8df3c796f4b6cb3e29d5045ad3ea0 (127.15.255.135:34701), a62a935ff43d494ca35d0c417943133d (127.15.255.130:37991), 16b3a337bc5e462f83ba895ad8660deb (127.15.255.131:36033)
I20240502 21:28:54.480569 3217 raft_consensus.cc:348] T 815178fbd07b4631b507927683a475c9 P 16b3a337bc5e462f83ba895ad8660deb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:28:54.480832 3217 raft_consensus.cc:374] T 815178fbd07b4631b507927683a475c9 P 16b3a337bc5e462f83ba895ad8660deb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:54.480912 3217 raft_consensus.cc:729] T 815178fbd07b4631b507927683a475c9 P 16b3a337bc5e462f83ba895ad8660deb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16b3a337bc5e462f83ba895ad8660deb, State: Initialized, Role: FOLLOWER
I20240502 21:28:54.481181 3217 consensus_queue.cc:260] T 815178fbd07b4631b507927683a475c9 P 16b3a337bc5e462f83ba895ad8660deb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:28:54.481993 3216 ts_tablet_manager.cc:1398] T 815178fbd07b4631b507927683a475c9 P 7d1e5c1b047443cbb28d7b0a9744babf: Time spent starting tablet: real 0.004s user 0.003s sys 0.001s
I20240502 21:28:54.482043 1954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8bee9bdcbff404280f000bdaeadf465" candidate_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e632c553ff4341328acf1cbaca678392" is_pre_election: true
I20240502 21:28:54.482270 1954 raft_consensus.cc:2454] T f8bee9bdcbff404280f000bdaeadf465 P e632c553ff4341328acf1cbaca678392 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d1e5c1b047443cbb28d7b0a9744babf in term 0.
I20240502 21:28:54.483144 2089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8bee9bdcbff404280f000bdaeadf465" candidate_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "521bb93035324cecbebed2f6f2a630f8" is_pre_election: true
I20240502 21:28:54.483309 3217 ts_tablet_manager.cc:1398] T 815178fbd07b4631b507927683a475c9 P 16b3a337bc5e462f83ba895ad8660deb: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 21:28:54.483372 2089 raft_consensus.cc:2454] T f8bee9bdcbff404280f000bdaeadf465 P 521bb93035324cecbebed2f6f2a630f8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d1e5c1b047443cbb28d7b0a9744babf in term 0.
I20240502 21:28:54.484813 1818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8bee9bdcbff404280f000bdaeadf465" candidate_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "153026dd7f8943cc93fedf6f48450fb4" is_pre_election: true
I20240502 21:28:54.484978 2227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8bee9bdcbff404280f000bdaeadf465" candidate_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" is_pre_election: true
I20240502 21:28:54.485031 1818 raft_consensus.cc:2454] T f8bee9bdcbff404280f000bdaeadf465 P 153026dd7f8943cc93fedf6f48450fb4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d1e5c1b047443cbb28d7b0a9744babf in term 0.
I20240502 21:28:54.485185 2227 raft_consensus.cc:2454] T f8bee9bdcbff404280f000bdaeadf465 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d1e5c1b047443cbb28d7b0a9744babf in term 0.
I20240502 21:28:54.485939 3223 tablet_bootstrap.cc:492] T 815178fbd07b4631b507927683a475c9 P 153026dd7f8943cc93fedf6f48450fb4: No bootstrap required, opened a new log
I20240502 21:28:54.485918 1550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8bee9bdcbff404280f000bdaeadf465" candidate_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a62a935ff43d494ca35d0c417943133d" is_pre_election: true
I20240502 21:28:54.486088 3223 ts_tablet_manager.cc:1367] T 815178fbd07b4631b507927683a475c9 P 153026dd7f8943cc93fedf6f48450fb4: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.003s
I20240502 21:28:54.486137 1550 raft_consensus.cc:2454] T f8bee9bdcbff404280f000bdaeadf465 P a62a935ff43d494ca35d0c417943133d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d1e5c1b047443cbb28d7b0a9744babf in term 0.
I20240502 21:28:54.486578 1348 leader_election.cc:302] T f8bee9bdcbff404280f000bdaeadf465 P 7d1e5c1b047443cbb28d7b0a9744babf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 6ab8df3c796f4b6cb3e29d5045ad3ea0, 7d1e5c1b047443cbb28d7b0a9744babf, a62a935ff43d494ca35d0c417943133d, e632c553ff4341328acf1cbaca678392; no voters:
I20240502 21:28:54.486742 3223 raft_consensus.cc:348] T 815178fbd07b4631b507927683a475c9 P 153026dd7f8943cc93fedf6f48450fb4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:28:54.486918 3247 raft_consensus.cc:2781] T f8bee9bdcbff404280f000bdaeadf465 P 7d1e5c1b047443cbb28d7b0a9744babf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:28:54.487004 3223 raft_consensus.cc:374] T 815178fbd07b4631b507927683a475c9 P 153026dd7f8943cc93fedf6f48450fb4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:54.487095 3223 raft_consensus.cc:729] T 815178fbd07b4631b507927683a475c9 P 153026dd7f8943cc93fedf6f48450fb4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 153026dd7f8943cc93fedf6f48450fb4, State: Initialized, Role: FOLLOWER
I20240502 21:28:54.487129 3247 raft_consensus.cc:482] T f8bee9bdcbff404280f000bdaeadf465 P 7d1e5c1b047443cbb28d7b0a9744babf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:28:54.487340 3247 raft_consensus.cc:3037] T f8bee9bdcbff404280f000bdaeadf465 P 7d1e5c1b047443cbb28d7b0a9744babf [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:54.487337 3223 consensus_queue.cc:260] T 815178fbd07b4631b507927683a475c9 P 153026dd7f8943cc93fedf6f48450fb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:28:54.487879 3223 ts_tablet_manager.cc:1398] T 815178fbd07b4631b507927683a475c9 P 153026dd7f8943cc93fedf6f48450fb4: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 21:28:54.488943 1686 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8bee9bdcbff404280f000bdaeadf465" candidate_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16b3a337bc5e462f83ba895ad8660deb" is_pre_election: true
I20240502 21:28:54.489157 1686 raft_consensus.cc:2454] T f8bee9bdcbff404280f000bdaeadf465 P 16b3a337bc5e462f83ba895ad8660deb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d1e5c1b047443cbb28d7b0a9744babf in term 0.
I20240502 21:28:54.491343 3247 raft_consensus.cc:504] T f8bee9bdcbff404280f000bdaeadf465 P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:28:54.493150 2089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8bee9bdcbff404280f000bdaeadf465" candidate_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "521bb93035324cecbebed2f6f2a630f8"
I20240502 21:28:54.493350 2089 raft_consensus.cc:3037] T f8bee9bdcbff404280f000bdaeadf465 P 521bb93035324cecbebed2f6f2a630f8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:54.493710 1818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8bee9bdcbff404280f000bdaeadf465" candidate_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "153026dd7f8943cc93fedf6f48450fb4"
I20240502 21:28:54.493867 1818 raft_consensus.cc:3037] T f8bee9bdcbff404280f000bdaeadf465 P 153026dd7f8943cc93fedf6f48450fb4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:54.495968 2089 raft_consensus.cc:2454] T f8bee9bdcbff404280f000bdaeadf465 P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d1e5c1b047443cbb28d7b0a9744babf in term 1.
I20240502 21:28:54.496580 1818 raft_consensus.cc:2454] T f8bee9bdcbff404280f000bdaeadf465 P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d1e5c1b047443cbb28d7b0a9744babf in term 1.
I20240502 21:28:54.497330 3219 tablet_bootstrap.cc:492] T 815178fbd07b4631b507927683a475c9 P 6ab8df3c796f4b6cb3e29d5045ad3ea0: No bootstrap required, opened a new log
I20240502 21:28:54.497493 3219 ts_tablet_manager.cc:1367] T 815178fbd07b4631b507927683a475c9 P 6ab8df3c796f4b6cb3e29d5045ad3ea0: Time spent bootstrapping tablet: real 0.026s user 0.003s sys 0.000s
I20240502 21:28:54.498241 3219 raft_consensus.cc:348] T 815178fbd07b4631b507927683a475c9 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:28:54.498507 3219 raft_consensus.cc:374] T 815178fbd07b4631b507927683a475c9 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:54.498605 3219 raft_consensus.cc:729] T 815178fbd07b4631b507927683a475c9 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ab8df3c796f4b6cb3e29d5045ad3ea0, State: Initialized, Role: FOLLOWER
I20240502 21:28:54.498859 3219 consensus_queue.cc:260] T 815178fbd07b4631b507927683a475c9 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:28:54.499439 3247 leader_election.cc:288] T f8bee9bdcbff404280f000bdaeadf465 P 7d1e5c1b047443cbb28d7b0a9744babf [CANDIDATE]: Term 1 election: Requested vote from peers 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132:44947), 521bb93035324cecbebed2f6f2a630f8 (127.15.255.134:40821), e632c553ff4341328acf1cbaca678392 (127.15.255.133:37439), 6ab8df3c796f4b6cb3e29d5045ad3ea0 (127.15.255.135:34701), a62a935ff43d494ca35d0c417943133d (127.15.255.130:37991), 16b3a337bc5e462f83ba895ad8660deb (127.15.255.131:36033)
I20240502 21:28:54.499392 1954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8bee9bdcbff404280f000bdaeadf465" candidate_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e632c553ff4341328acf1cbaca678392"
I20240502 21:28:54.499588 1954 raft_consensus.cc:3037] T f8bee9bdcbff404280f000bdaeadf465 P e632c553ff4341328acf1cbaca678392 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:54.499686 3219 ts_tablet_manager.cc:1398] T 815178fbd07b4631b507927683a475c9 P 6ab8df3c796f4b6cb3e29d5045ad3ea0: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240502 21:28:54.499725 1686 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8bee9bdcbff404280f000bdaeadf465" candidate_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16b3a337bc5e462f83ba895ad8660deb"
I20240502 21:28:54.499912 1686 raft_consensus.cc:3037] T f8bee9bdcbff404280f000bdaeadf465 P 16b3a337bc5e462f83ba895ad8660deb [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:54.502398 1954 raft_consensus.cc:2454] T f8bee9bdcbff404280f000bdaeadf465 P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d1e5c1b047443cbb28d7b0a9744babf in term 1.
I20240502 21:28:54.502609 1686 raft_consensus.cc:2454] T f8bee9bdcbff404280f000bdaeadf465 P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d1e5c1b047443cbb28d7b0a9744babf in term 1.
I20240502 21:28:54.502944 1347 leader_election.cc:302] T f8bee9bdcbff404280f000bdaeadf465 P 7d1e5c1b047443cbb28d7b0a9744babf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 153026dd7f8943cc93fedf6f48450fb4, 521bb93035324cecbebed2f6f2a630f8, 7d1e5c1b047443cbb28d7b0a9744babf, e632c553ff4341328acf1cbaca678392; no voters:
I20240502 21:28:54.503551 2227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8bee9bdcbff404280f000bdaeadf465" candidate_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0"
I20240502 21:28:54.503573 1550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8bee9bdcbff404280f000bdaeadf465" candidate_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a62a935ff43d494ca35d0c417943133d"
I20240502 21:28:54.503751 1550 raft_consensus.cc:3037] T f8bee9bdcbff404280f000bdaeadf465 P a62a935ff43d494ca35d0c417943133d [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:54.503759 2227 raft_consensus.cc:3037] T f8bee9bdcbff404280f000bdaeadf465 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:54.505338 3247 raft_consensus.cc:2781] T f8bee9bdcbff404280f000bdaeadf465 P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:28:54.505504 3247 raft_consensus.cc:686] T f8bee9bdcbff404280f000bdaeadf465 P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 LEADER]: Becoming Leader. State: Replica: 7d1e5c1b047443cbb28d7b0a9744babf, State: Running, Role: LEADER
I20240502 21:28:54.505822 3247 consensus_queue.cc:237] T f8bee9bdcbff404280f000bdaeadf465 P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:28:54.506371 2227 raft_consensus.cc:2454] T f8bee9bdcbff404280f000bdaeadf465 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d1e5c1b047443cbb28d7b0a9744babf in term 1.
I20240502 21:28:54.506807 1550 raft_consensus.cc:2454] T f8bee9bdcbff404280f000bdaeadf465 P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d1e5c1b047443cbb28d7b0a9744babf in term 1.
I20240502 21:28:54.509196 1266 catalog_manager.cc:5484] T f8bee9bdcbff404280f000bdaeadf465 P 7d1e5c1b047443cbb28d7b0a9744babf reported cstate change: term changed from 0 to 1, leader changed from <none> to 7d1e5c1b047443cbb28d7b0a9744babf (127.15.255.129). New cstate: current_term: 1 leader_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } health_report { overall_health: UNKNOWN } } }
I20240502 21:28:54.523981 3247 raft_consensus.cc:482] T 815178fbd07b4631b507927683a475c9 P 7d1e5c1b047443cbb28d7b0a9744babf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:28:54.524144 3247 raft_consensus.cc:504] T 815178fbd07b4631b507927683a475c9 P 7d1e5c1b047443cbb28d7b0a9744babf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:28:54.525277 3247 leader_election.cc:288] T 815178fbd07b4631b507927683a475c9 P 7d1e5c1b047443cbb28d7b0a9744babf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132:44947), 521bb93035324cecbebed2f6f2a630f8 (127.15.255.134:40821), e632c553ff4341328acf1cbaca678392 (127.15.255.133:37439), 6ab8df3c796f4b6cb3e29d5045ad3ea0 (127.15.255.135:34701), a62a935ff43d494ca35d0c417943133d (127.15.255.130:37991), 16b3a337bc5e462f83ba895ad8660deb (127.15.255.131:36033)
I20240502 21:28:54.525362 2089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "815178fbd07b4631b507927683a475c9" candidate_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "521bb93035324cecbebed2f6f2a630f8" is_pre_election: true
I20240502 21:28:54.525595 2089 raft_consensus.cc:2454] T 815178fbd07b4631b507927683a475c9 P 521bb93035324cecbebed2f6f2a630f8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d1e5c1b047443cbb28d7b0a9744babf in term 0.
I20240502 21:28:54.525624 1954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "815178fbd07b4631b507927683a475c9" candidate_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e632c553ff4341328acf1cbaca678392" is_pre_election: true
I20240502 21:28:54.525758 1550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "815178fbd07b4631b507927683a475c9" candidate_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a62a935ff43d494ca35d0c417943133d" is_pre_election: true
I20240502 21:28:54.525774 2227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "815178fbd07b4631b507927683a475c9" candidate_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" is_pre_election: true
I20240502 21:28:54.525830 1954 raft_consensus.cc:2454] T 815178fbd07b4631b507927683a475c9 P e632c553ff4341328acf1cbaca678392 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d1e5c1b047443cbb28d7b0a9744babf in term 0.
I20240502 21:28:54.525993 1550 raft_consensus.cc:2454] T 815178fbd07b4631b507927683a475c9 P a62a935ff43d494ca35d0c417943133d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d1e5c1b047443cbb28d7b0a9744babf in term 0.
I20240502 21:28:54.526444 1818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "815178fbd07b4631b507927683a475c9" candidate_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "153026dd7f8943cc93fedf6f48450fb4" is_pre_election: true
I20240502 21:28:54.526484 1347 leader_election.cc:302] T 815178fbd07b4631b507927683a475c9 P 7d1e5c1b047443cbb28d7b0a9744babf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 521bb93035324cecbebed2f6f2a630f8, 7d1e5c1b047443cbb28d7b0a9744babf, a62a935ff43d494ca35d0c417943133d, e632c553ff4341328acf1cbaca678392; no voters:
I20240502 21:28:54.526677 1818 raft_consensus.cc:2454] T 815178fbd07b4631b507927683a475c9 P 153026dd7f8943cc93fedf6f48450fb4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d1e5c1b047443cbb28d7b0a9744babf in term 0.
I20240502 21:28:54.526670 1686 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "815178fbd07b4631b507927683a475c9" candidate_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16b3a337bc5e462f83ba895ad8660deb" is_pre_election: true
I20240502 21:28:54.526912 1686 raft_consensus.cc:2454] T 815178fbd07b4631b507927683a475c9 P 16b3a337bc5e462f83ba895ad8660deb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d1e5c1b047443cbb28d7b0a9744babf in term 0.
I20240502 21:28:54.526947 3247 raft_consensus.cc:2781] T 815178fbd07b4631b507927683a475c9 P 7d1e5c1b047443cbb28d7b0a9744babf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:28:54.527036 3247 raft_consensus.cc:482] T 815178fbd07b4631b507927683a475c9 P 7d1e5c1b047443cbb28d7b0a9744babf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:28:54.527127 3247 raft_consensus.cc:3037] T 815178fbd07b4631b507927683a475c9 P 7d1e5c1b047443cbb28d7b0a9744babf [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:54.525975 2227 raft_consensus.cc:2454] T 815178fbd07b4631b507927683a475c9 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d1e5c1b047443cbb28d7b0a9744babf in term 0.
I20240502 21:28:54.529237 3247 raft_consensus.cc:504] T 815178fbd07b4631b507927683a475c9 P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:28:54.530164 1818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "815178fbd07b4631b507927683a475c9" candidate_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "153026dd7f8943cc93fedf6f48450fb4"
I20240502 21:28:54.530254 2089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "815178fbd07b4631b507927683a475c9" candidate_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "521bb93035324cecbebed2f6f2a630f8"
I20240502 21:28:54.530370 1818 raft_consensus.cc:3037] T 815178fbd07b4631b507927683a475c9 P 153026dd7f8943cc93fedf6f48450fb4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:54.530378 1954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "815178fbd07b4631b507927683a475c9" candidate_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e632c553ff4341328acf1cbaca678392"
I20240502 21:28:54.530472 2089 raft_consensus.cc:3037] T 815178fbd07b4631b507927683a475c9 P 521bb93035324cecbebed2f6f2a630f8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:54.530570 1954 raft_consensus.cc:3037] T 815178fbd07b4631b507927683a475c9 P e632c553ff4341328acf1cbaca678392 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:54.530689 2227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "815178fbd07b4631b507927683a475c9" candidate_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0"
I20240502 21:28:54.530932 2227 raft_consensus.cc:3037] T 815178fbd07b4631b507927683a475c9 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:54.532882 1818 raft_consensus.cc:2454] T 815178fbd07b4631b507927683a475c9 P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d1e5c1b047443cbb28d7b0a9744babf in term 1.
I20240502 21:28:54.533548 1550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "815178fbd07b4631b507927683a475c9" candidate_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a62a935ff43d494ca35d0c417943133d"
I20240502 21:28:54.533757 1550 raft_consensus.cc:3037] T 815178fbd07b4631b507927683a475c9 P a62a935ff43d494ca35d0c417943133d [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:54.533819 1954 raft_consensus.cc:2454] T 815178fbd07b4631b507927683a475c9 P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d1e5c1b047443cbb28d7b0a9744babf in term 1.
I20240502 21:28:54.533819 2089 raft_consensus.cc:2454] T 815178fbd07b4631b507927683a475c9 P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d1e5c1b047443cbb28d7b0a9744babf in term 1.
I20240502 21:28:54.534297 1347 leader_election.cc:302] T 815178fbd07b4631b507927683a475c9 P 7d1e5c1b047443cbb28d7b0a9744babf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 153026dd7f8943cc93fedf6f48450fb4, 521bb93035324cecbebed2f6f2a630f8, 7d1e5c1b047443cbb28d7b0a9744babf, e632c553ff4341328acf1cbaca678392; no voters:
I20240502 21:28:54.534557 3122 raft_consensus.cc:2781] T 815178fbd07b4631b507927683a475c9 P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:28:54.534987 3122 raft_consensus.cc:686] T 815178fbd07b4631b507927683a475c9 P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 LEADER]: Becoming Leader. State: Replica: 7d1e5c1b047443cbb28d7b0a9744babf, State: Running, Role: LEADER
I20240502 21:28:54.535351 3122 consensus_queue.cc:237] T 815178fbd07b4631b507927683a475c9 P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:28:54.535745 2227 raft_consensus.cc:2454] T 815178fbd07b4631b507927683a475c9 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d1e5c1b047443cbb28d7b0a9744babf in term 1.
I20240502 21:28:54.536319 1550 raft_consensus.cc:2454] T 815178fbd07b4631b507927683a475c9 P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d1e5c1b047443cbb28d7b0a9744babf in term 1.
I20240502 21:28:54.536840 3247 leader_election.cc:288] T 815178fbd07b4631b507927683a475c9 P 7d1e5c1b047443cbb28d7b0a9744babf [CANDIDATE]: Term 1 election: Requested vote from peers 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132:44947), 521bb93035324cecbebed2f6f2a630f8 (127.15.255.134:40821), e632c553ff4341328acf1cbaca678392 (127.15.255.133:37439), 6ab8df3c796f4b6cb3e29d5045ad3ea0 (127.15.255.135:34701), a62a935ff43d494ca35d0c417943133d (127.15.255.130:37991), 16b3a337bc5e462f83ba895ad8660deb (127.15.255.131:36033)
I20240502 21:28:54.536967 1686 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "815178fbd07b4631b507927683a475c9" candidate_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16b3a337bc5e462f83ba895ad8660deb"
I20240502 21:28:54.537217 1686 raft_consensus.cc:3037] T 815178fbd07b4631b507927683a475c9 P 16b3a337bc5e462f83ba895ad8660deb [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:54.539202 1686 raft_consensus.cc:2454] T 815178fbd07b4631b507927683a475c9 P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d1e5c1b047443cbb28d7b0a9744babf in term 1.
I20240502 21:28:54.538991 1266 catalog_manager.cc:5484] T 815178fbd07b4631b507927683a475c9 P 7d1e5c1b047443cbb28d7b0a9744babf reported cstate change: term changed from 0 to 1, leader changed from <none> to 7d1e5c1b047443cbb28d7b0a9744babf (127.15.255.129). New cstate: current_term: 1 leader_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } health_report { overall_health: UNKNOWN } } }
I20240502 21:28:54.580410 3232 raft_consensus.cc:482] T bcddb146822e4573837a5697d74134f7 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:28:54.580621 3232 raft_consensus.cc:504] T bcddb146822e4573837a5697d74134f7 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:28:54.581786 3232 leader_election.cc:288] T bcddb146822e4573837a5697d74134f7 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 521bb93035324cecbebed2f6f2a630f8 (127.15.255.134:40821), e632c553ff4341328acf1cbaca678392 (127.15.255.133:37439), 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132:44947), 7d1e5c1b047443cbb28d7b0a9744babf (127.15.255.129:37897), a62a935ff43d494ca35d0c417943133d (127.15.255.130:37991), 16b3a337bc5e462f83ba895ad8660deb (127.15.255.131:36033)
I20240502 21:28:54.582209 1818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcddb146822e4573837a5697d74134f7" candidate_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "153026dd7f8943cc93fedf6f48450fb4" is_pre_election: true
I20240502 21:28:54.582228 2089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcddb146822e4573837a5697d74134f7" candidate_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "521bb93035324cecbebed2f6f2a630f8" is_pre_election: true
I20240502 21:28:54.582464 2089 raft_consensus.cc:2454] T bcddb146822e4573837a5697d74134f7 P 521bb93035324cecbebed2f6f2a630f8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ab8df3c796f4b6cb3e29d5045ad3ea0 in term 0.
I20240502 21:28:54.582469 1818 raft_consensus.cc:2454] T bcddb146822e4573837a5697d74134f7 P 153026dd7f8943cc93fedf6f48450fb4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ab8df3c796f4b6cb3e29d5045ad3ea0 in term 0.
I20240502 21:28:54.582583 1550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcddb146822e4573837a5697d74134f7" candidate_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a62a935ff43d494ca35d0c417943133d" is_pre_election: true
I20240502 21:28:54.582808 1550 raft_consensus.cc:2454] T bcddb146822e4573837a5697d74134f7 P a62a935ff43d494ca35d0c417943133d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ab8df3c796f4b6cb3e29d5045ad3ea0 in term 0.
I20240502 21:28:54.583247 2159 leader_election.cc:302] T bcddb146822e4573837a5697d74134f7 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 153026dd7f8943cc93fedf6f48450fb4, 521bb93035324cecbebed2f6f2a630f8, 6ab8df3c796f4b6cb3e29d5045ad3ea0, a62a935ff43d494ca35d0c417943133d; no voters:
I20240502 21:28:54.583299 1954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcddb146822e4573837a5697d74134f7" candidate_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e632c553ff4341328acf1cbaca678392" is_pre_election: true
I20240502 21:28:54.583508 1954 raft_consensus.cc:2454] T bcddb146822e4573837a5697d74134f7 P e632c553ff4341328acf1cbaca678392 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ab8df3c796f4b6cb3e29d5045ad3ea0 in term 0.
I20240502 21:28:54.583505 3232 raft_consensus.cc:2781] T bcddb146822e4573837a5697d74134f7 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:28:54.583621 3232 raft_consensus.cc:482] T bcddb146822e4573837a5697d74134f7 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:28:54.583714 3232 raft_consensus.cc:3037] T bcddb146822e4573837a5697d74134f7 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:54.583628 1415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcddb146822e4573837a5697d74134f7" candidate_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" is_pre_election: true
I20240502 21:28:54.583873 1415 raft_consensus.cc:2454] T bcddb146822e4573837a5697d74134f7 P 7d1e5c1b047443cbb28d7b0a9744babf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ab8df3c796f4b6cb3e29d5045ad3ea0 in term 0.
I20240502 21:28:54.584298 1686 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcddb146822e4573837a5697d74134f7" candidate_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16b3a337bc5e462f83ba895ad8660deb" is_pre_election: true
I20240502 21:28:54.584553 1686 raft_consensus.cc:2454] T bcddb146822e4573837a5697d74134f7 P 16b3a337bc5e462f83ba895ad8660deb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ab8df3c796f4b6cb3e29d5045ad3ea0 in term 0.
I20240502 21:28:54.585963 3232 raft_consensus.cc:504] T bcddb146822e4573837a5697d74134f7 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:28:54.587002 2089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcddb146822e4573837a5697d74134f7" candidate_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "521bb93035324cecbebed2f6f2a630f8"
I20240502 21:28:54.587198 2089 raft_consensus.cc:3037] T bcddb146822e4573837a5697d74134f7 P 521bb93035324cecbebed2f6f2a630f8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:54.587142 1954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcddb146822e4573837a5697d74134f7" candidate_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e632c553ff4341328acf1cbaca678392"
I20240502 21:28:54.587174 1818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcddb146822e4573837a5697d74134f7" candidate_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "153026dd7f8943cc93fedf6f48450fb4"
I20240502 21:28:54.587325 1954 raft_consensus.cc:3037] T bcddb146822e4573837a5697d74134f7 P e632c553ff4341328acf1cbaca678392 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:54.587353 1818 raft_consensus.cc:3037] T bcddb146822e4573837a5697d74134f7 P 153026dd7f8943cc93fedf6f48450fb4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:54.589893 2089 raft_consensus.cc:2454] T bcddb146822e4573837a5697d74134f7 P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ab8df3c796f4b6cb3e29d5045ad3ea0 in term 1.
I20240502 21:28:54.590178 1954 raft_consensus.cc:2454] T bcddb146822e4573837a5697d74134f7 P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ab8df3c796f4b6cb3e29d5045ad3ea0 in term 1.
I20240502 21:28:54.592026 1818 raft_consensus.cc:2454] T bcddb146822e4573837a5697d74134f7 P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ab8df3c796f4b6cb3e29d5045ad3ea0 in term 1.
I20240502 21:28:54.592453 2160 leader_election.cc:302] T bcddb146822e4573837a5697d74134f7 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 153026dd7f8943cc93fedf6f48450fb4, 521bb93035324cecbebed2f6f2a630f8, 6ab8df3c796f4b6cb3e29d5045ad3ea0, e632c553ff4341328acf1cbaca678392; no voters:
I20240502 21:28:54.592878 1415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcddb146822e4573837a5697d74134f7" candidate_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d1e5c1b047443cbb28d7b0a9744babf"
I20240502 21:28:54.593129 1415 raft_consensus.cc:3037] T bcddb146822e4573837a5697d74134f7 P 7d1e5c1b047443cbb28d7b0a9744babf [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:54.596074 1415 raft_consensus.cc:2454] T bcddb146822e4573837a5697d74134f7 P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ab8df3c796f4b6cb3e29d5045ad3ea0 in term 1.
I20240502 21:28:54.597038 3232 leader_election.cc:288] T bcddb146822e4573837a5697d74134f7 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [CANDIDATE]: Term 1 election: Requested vote from peers 521bb93035324cecbebed2f6f2a630f8 (127.15.255.134:40821), e632c553ff4341328acf1cbaca678392 (127.15.255.133:37439), 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132:44947), 7d1e5c1b047443cbb28d7b0a9744babf (127.15.255.129:37897), a62a935ff43d494ca35d0c417943133d (127.15.255.130:37991), 16b3a337bc5e462f83ba895ad8660deb (127.15.255.131:36033)
I20240502 21:28:54.597306 3232 raft_consensus.cc:2781] T bcddb146822e4573837a5697d74134f7 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:28:54.597339 1550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcddb146822e4573837a5697d74134f7" candidate_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a62a935ff43d494ca35d0c417943133d"
I20240502 21:28:54.597472 3232 raft_consensus.cc:686] T bcddb146822e4573837a5697d74134f7 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 LEADER]: Becoming Leader. State: Replica: 6ab8df3c796f4b6cb3e29d5045ad3ea0, State: Running, Role: LEADER
I20240502 21:28:54.597555 1550 raft_consensus.cc:3037] T bcddb146822e4573837a5697d74134f7 P a62a935ff43d494ca35d0c417943133d [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:54.598192 3232 consensus_queue.cc:237] T bcddb146822e4573837a5697d74134f7 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:28:54.600273 1550 raft_consensus.cc:2454] T bcddb146822e4573837a5697d74134f7 P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ab8df3c796f4b6cb3e29d5045ad3ea0 in term 1.
I20240502 21:28:54.601316 1686 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcddb146822e4573837a5697d74134f7" candidate_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16b3a337bc5e462f83ba895ad8660deb"
I20240502 21:28:54.601562 1686 raft_consensus.cc:3037] T bcddb146822e4573837a5697d74134f7 P 16b3a337bc5e462f83ba895ad8660deb [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:54.606585 1686 raft_consensus.cc:2454] T bcddb146822e4573837a5697d74134f7 P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ab8df3c796f4b6cb3e29d5045ad3ea0 in term 1.
I20240502 21:28:54.607126 1266 catalog_manager.cc:5484] T bcddb146822e4573837a5697d74134f7 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6ab8df3c796f4b6cb3e29d5045ad3ea0 (127.15.255.135). New cstate: current_term: 1 leader_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } health_report { overall_health: UNKNOWN } } }
I20240502 21:28:54.889096 3122 consensus_queue.cc:1035] T f8bee9bdcbff404280f000bdaeadf465 P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:28:54.919410 3122 consensus_queue.cc:1035] T 815178fbd07b4631b507927683a475c9 P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:28:54.934350 3122 consensus_queue.cc:1035] T 815178fbd07b4631b507927683a475c9 P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 21:28:54.969446 3254 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:28:54.969699 3254 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:28:54.976928 3254 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 21:28:54.981874 3232 consensus_queue.cc:1035] T bcddb146822e4573837a5697d74134f7 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:28:54.988138 3122 consensus_queue.cc:1035] T 815178fbd07b4631b507927683a475c9 P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:28:54.992802 3232 consensus_queue.cc:1035] T bcddb146822e4573837a5697d74134f7 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:28:55.000221 3122 consensus_queue.cc:1035] T 815178fbd07b4631b507927683a475c9 P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:28:55.003623 3232 consensus_queue.cc:1035] T bcddb146822e4573837a5697d74134f7 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:28:55.007565 3247 consensus_queue.cc:1035] T f8bee9bdcbff404280f000bdaeadf465 P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:28:55.013288 3122 consensus_queue.cc:1035] T 815178fbd07b4631b507927683a475c9 P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:28:55.016764 3247 consensus_queue.cc:1035] T 815178fbd07b4631b507927683a475c9 P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:28:55.025511 3247 consensus_queue.cc:1035] T f8bee9bdcbff404280f000bdaeadf465 P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:28:55.076789 3251 consensus_queue.cc:1035] T bcddb146822e4573837a5697d74134f7 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:28:55.081727 3122 consensus_queue.cc:1035] T f8bee9bdcbff404280f000bdaeadf465 P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:28:55.089201 3249 consensus_queue.cc:1035] T f8bee9bdcbff404280f000bdaeadf465 P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:28:55.093307 3251 consensus_queue.cc:1035] T bcddb146822e4573837a5697d74134f7 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:28:55.116225 3232 consensus_queue.cc:1035] T bcddb146822e4573837a5697d74134f7 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:28:55.118225 3249 consensus_queue.cc:1035] T f8bee9bdcbff404280f000bdaeadf465 P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_3'
INSERT report
rows total: 2000
time total: 295.542 ms
time per row: 0.147771 ms
-----------------------------------------------------
Suppressions used:
count bytes template
16 4220 libcrypto.so.1.1
-----------------------------------------------------
I20240502 21:28:56.553856 1266 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44934:
name: "rebalance_test_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 7
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
W20240502 21:28:56.554627 1266 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table rebalance_test_table_4 in case of a server failure: 8 tablet servers would be needed, 7 are available. Consider bringing up more tablet servers.
I20240502 21:28:56.564713 2207 tablet_service.cc:1467] Processing CreateTablet for tablet c46e36b0bceb45b4bafbf1ffaf8e039e (DEFAULT_TABLE table=rebalance_test_table_4 [id=d523de924ba14cb493e559296c083a34]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:56.564687 1933 tablet_service.cc:1467] Processing CreateTablet for tablet c46e36b0bceb45b4bafbf1ffaf8e039e (DEFAULT_TABLE table=rebalance_test_table_4 [id=d523de924ba14cb493e559296c083a34]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:56.565152 1933 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c46e36b0bceb45b4bafbf1ffaf8e039e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:56.565152 2207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c46e36b0bceb45b4bafbf1ffaf8e039e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:56.565812 1394 tablet_service.cc:1467] Processing CreateTablet for tablet c46e36b0bceb45b4bafbf1ffaf8e039e (DEFAULT_TABLE table=rebalance_test_table_4 [id=d523de924ba14cb493e559296c083a34]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:56.566282 1394 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c46e36b0bceb45b4bafbf1ffaf8e039e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:56.569404 1798 tablet_service.cc:1467] Processing CreateTablet for tablet c46e36b0bceb45b4bafbf1ffaf8e039e (DEFAULT_TABLE table=rebalance_test_table_4 [id=d523de924ba14cb493e559296c083a34]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:56.569825 1798 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c46e36b0bceb45b4bafbf1ffaf8e039e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:56.569805 1799 tablet_service.cc:1467] Processing CreateTablet for tablet 05e885ae741c43808c6dfbcdc5769942 (DEFAULT_TABLE table=rebalance_test_table_4 [id=d523de924ba14cb493e559296c083a34]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:56.570205 1799 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05e885ae741c43808c6dfbcdc5769942. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:56.571962 1662 tablet_service.cc:1467] Processing CreateTablet for tablet c46e36b0bceb45b4bafbf1ffaf8e039e (DEFAULT_TABLE table=rebalance_test_table_4 [id=d523de924ba14cb493e559296c083a34]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:56.572223 1663 tablet_service.cc:1467] Processing CreateTablet for tablet 05e885ae741c43808c6dfbcdc5769942 (DEFAULT_TABLE table=rebalance_test_table_4 [id=d523de924ba14cb493e559296c083a34]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:56.572767 1800 tablet_service.cc:1467] Processing CreateTablet for tablet aae7b5a01a0543afb1b5a7f44a9a864c (DEFAULT_TABLE table=rebalance_test_table_4 [id=d523de924ba14cb493e559296c083a34]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:56.572764 1664 tablet_service.cc:1467] Processing CreateTablet for tablet aae7b5a01a0543afb1b5a7f44a9a864c (DEFAULT_TABLE table=rebalance_test_table_4 [id=d523de924ba14cb493e559296c083a34]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:56.572415 1662 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c46e36b0bceb45b4bafbf1ffaf8e039e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:56.573159 1800 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aae7b5a01a0543afb1b5a7f44a9a864c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:56.575367 2206 tablet_service.cc:1467] Processing CreateTablet for tablet 05e885ae741c43808c6dfbcdc5769942 (DEFAULT_TABLE table=rebalance_test_table_4 [id=d523de924ba14cb493e559296c083a34]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:56.575768 2206 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05e885ae741c43808c6dfbcdc5769942. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:56.576726 1664 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aae7b5a01a0543afb1b5a7f44a9a864c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:56.576807 1526 tablet_service.cc:1467] Processing CreateTablet for tablet c46e36b0bceb45b4bafbf1ffaf8e039e (DEFAULT_TABLE table=rebalance_test_table_4 [id=d523de924ba14cb493e559296c083a34]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:56.577203 1526 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c46e36b0bceb45b4bafbf1ffaf8e039e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:56.580755 2205 tablet_service.cc:1467] Processing CreateTablet for tablet aae7b5a01a0543afb1b5a7f44a9a864c (DEFAULT_TABLE table=rebalance_test_table_4 [id=d523de924ba14cb493e559296c083a34]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:56.581123 2205 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aae7b5a01a0543afb1b5a7f44a9a864c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:56.581729 2067 tablet_service.cc:1467] Processing CreateTablet for tablet c46e36b0bceb45b4bafbf1ffaf8e039e (DEFAULT_TABLE table=rebalance_test_table_4 [id=d523de924ba14cb493e559296c083a34]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:56.581867 2068 tablet_service.cc:1467] Processing CreateTablet for tablet aae7b5a01a0543afb1b5a7f44a9a864c (DEFAULT_TABLE table=rebalance_test_table_4 [id=d523de924ba14cb493e559296c083a34]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:56.582183 2067 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c46e36b0bceb45b4bafbf1ffaf8e039e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:56.582835 1932 tablet_service.cc:1467] Processing CreateTablet for tablet 05e885ae741c43808c6dfbcdc5769942 (DEFAULT_TABLE table=rebalance_test_table_4 [id=d523de924ba14cb493e559296c083a34]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:56.583201 1932 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05e885ae741c43808c6dfbcdc5769942. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:56.584779 2069 tablet_service.cc:1467] Processing CreateTablet for tablet 05e885ae741c43808c6dfbcdc5769942 (DEFAULT_TABLE table=rebalance_test_table_4 [id=d523de924ba14cb493e559296c083a34]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:56.585139 2069 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05e885ae741c43808c6dfbcdc5769942. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:56.585091 1528 tablet_service.cc:1467] Processing CreateTablet for tablet 05e885ae741c43808c6dfbcdc5769942 (DEFAULT_TABLE table=rebalance_test_table_4 [id=d523de924ba14cb493e559296c083a34]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:56.585454 1528 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05e885ae741c43808c6dfbcdc5769942. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:56.587015 1931 tablet_service.cc:1467] Processing CreateTablet for tablet aae7b5a01a0543afb1b5a7f44a9a864c (DEFAULT_TABLE table=rebalance_test_table_4 [id=d523de924ba14cb493e559296c083a34]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:56.587371 1931 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aae7b5a01a0543afb1b5a7f44a9a864c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:56.588807 1527 tablet_service.cc:1467] Processing CreateTablet for tablet aae7b5a01a0543afb1b5a7f44a9a864c (DEFAULT_TABLE table=rebalance_test_table_4 [id=d523de924ba14cb493e559296c083a34]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:56.589221 1527 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aae7b5a01a0543afb1b5a7f44a9a864c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:56.591791 2068 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aae7b5a01a0543afb1b5a7f44a9a864c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:56.593520 1393 tablet_service.cc:1467] Processing CreateTablet for tablet 05e885ae741c43808c6dfbcdc5769942 (DEFAULT_TABLE table=rebalance_test_table_4 [id=d523de924ba14cb493e559296c083a34]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:56.593920 1393 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05e885ae741c43808c6dfbcdc5769942. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:56.596836 1392 tablet_service.cc:1467] Processing CreateTablet for tablet aae7b5a01a0543afb1b5a7f44a9a864c (DEFAULT_TABLE table=rebalance_test_table_4 [id=d523de924ba14cb493e559296c083a34]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:56.597213 1392 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aae7b5a01a0543afb1b5a7f44a9a864c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:56.598220 3365 tablet_bootstrap.cc:492] T c46e36b0bceb45b4bafbf1ffaf8e039e P 153026dd7f8943cc93fedf6f48450fb4: Bootstrap starting.
I20240502 21:28:56.599786 3365 tablet_bootstrap.cc:654] T c46e36b0bceb45b4bafbf1ffaf8e039e P 153026dd7f8943cc93fedf6f48450fb4: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:56.602392 1663 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05e885ae741c43808c6dfbcdc5769942. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:56.618991 3369 tablet_bootstrap.cc:492] T aae7b5a01a0543afb1b5a7f44a9a864c P e632c553ff4341328acf1cbaca678392: Bootstrap starting.
I20240502 21:28:56.622247 3366 tablet_bootstrap.cc:492] T c46e36b0bceb45b4bafbf1ffaf8e039e P 521bb93035324cecbebed2f6f2a630f8: Bootstrap starting.
I20240502 21:28:56.624063 3371 tablet_bootstrap.cc:492] T c46e36b0bceb45b4bafbf1ffaf8e039e P a62a935ff43d494ca35d0c417943133d: Bootstrap starting.
I20240502 21:28:56.626075 3366 tablet_bootstrap.cc:654] T c46e36b0bceb45b4bafbf1ffaf8e039e P 521bb93035324cecbebed2f6f2a630f8: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:56.620632 3369 tablet_bootstrap.cc:654] T aae7b5a01a0543afb1b5a7f44a9a864c P e632c553ff4341328acf1cbaca678392: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:56.625916 3371 tablet_bootstrap.cc:654] T c46e36b0bceb45b4bafbf1ffaf8e039e P a62a935ff43d494ca35d0c417943133d: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:56.628238 3364 tablet_bootstrap.cc:492] T c46e36b0bceb45b4bafbf1ffaf8e039e P 6ab8df3c796f4b6cb3e29d5045ad3ea0: Bootstrap starting.
I20240502 21:28:56.628595 3372 tablet_bootstrap.cc:492] T 05e885ae741c43808c6dfbcdc5769942 P 7d1e5c1b047443cbb28d7b0a9744babf: Bootstrap starting.
I20240502 21:28:56.630642 3364 tablet_bootstrap.cc:654] T c46e36b0bceb45b4bafbf1ffaf8e039e P 6ab8df3c796f4b6cb3e29d5045ad3ea0: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:56.632532 3371 tablet_bootstrap.cc:492] T c46e36b0bceb45b4bafbf1ffaf8e039e P a62a935ff43d494ca35d0c417943133d: No bootstrap required, opened a new log
I20240502 21:28:56.632674 3371 ts_tablet_manager.cc:1367] T c46e36b0bceb45b4bafbf1ffaf8e039e P a62a935ff43d494ca35d0c417943133d: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.000s
I20240502 21:28:56.633381 3371 raft_consensus.cc:348] T c46e36b0bceb45b4bafbf1ffaf8e039e P a62a935ff43d494ca35d0c417943133d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:28:56.633634 3372 tablet_bootstrap.cc:654] T 05e885ae741c43808c6dfbcdc5769942 P 7d1e5c1b047443cbb28d7b0a9744babf: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:56.633652 3371 raft_consensus.cc:374] T c46e36b0bceb45b4bafbf1ffaf8e039e P a62a935ff43d494ca35d0c417943133d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:56.633733 3371 raft_consensus.cc:729] T c46e36b0bceb45b4bafbf1ffaf8e039e P a62a935ff43d494ca35d0c417943133d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a62a935ff43d494ca35d0c417943133d, State: Initialized, Role: FOLLOWER
I20240502 21:28:56.633952 3371 consensus_queue.cc:260] T c46e36b0bceb45b4bafbf1ffaf8e039e P a62a935ff43d494ca35d0c417943133d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:28:56.634702 3371 ts_tablet_manager.cc:1398] T c46e36b0bceb45b4bafbf1ffaf8e039e P a62a935ff43d494ca35d0c417943133d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:28:56.635020 3371 tablet_bootstrap.cc:492] T 05e885ae741c43808c6dfbcdc5769942 P a62a935ff43d494ca35d0c417943133d: Bootstrap starting.
I20240502 21:28:56.635196 3253 raft_consensus.cc:482] T c46e36b0bceb45b4bafbf1ffaf8e039e P a62a935ff43d494ca35d0c417943133d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:28:56.635361 3253 raft_consensus.cc:504] T c46e36b0bceb45b4bafbf1ffaf8e039e P a62a935ff43d494ca35d0c417943133d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:28:56.635957 3364 tablet_bootstrap.cc:492] T c46e36b0bceb45b4bafbf1ffaf8e039e P 6ab8df3c796f4b6cb3e29d5045ad3ea0: No bootstrap required, opened a new log
I20240502 21:28:56.636195 3364 ts_tablet_manager.cc:1367] T c46e36b0bceb45b4bafbf1ffaf8e039e P 6ab8df3c796f4b6cb3e29d5045ad3ea0: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.000s
I20240502 21:28:56.636552 3371 tablet_bootstrap.cc:654] T 05e885ae741c43808c6dfbcdc5769942 P a62a935ff43d494ca35d0c417943133d: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:56.637017 1818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c46e36b0bceb45b4bafbf1ffaf8e039e" candidate_uuid: "a62a935ff43d494ca35d0c417943133d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "153026dd7f8943cc93fedf6f48450fb4" is_pre_election: true
W20240502 21:28:56.637672 1482 leader_election.cc:341] T c46e36b0bceb45b4bafbf1ffaf8e039e P a62a935ff43d494ca35d0c417943133d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132:44947): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:28:56.637867 3364 raft_consensus.cc:348] T c46e36b0bceb45b4bafbf1ffaf8e039e P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:28:56.638336 1954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c46e36b0bceb45b4bafbf1ffaf8e039e" candidate_uuid: "a62a935ff43d494ca35d0c417943133d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e632c553ff4341328acf1cbaca678392" is_pre_election: true
I20240502 21:28:56.638474 3364 raft_consensus.cc:374] T c46e36b0bceb45b4bafbf1ffaf8e039e P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:56.638667 3364 raft_consensus.cc:729] T c46e36b0bceb45b4bafbf1ffaf8e039e P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ab8df3c796f4b6cb3e29d5045ad3ea0, State: Initialized, Role: FOLLOWER
I20240502 21:28:56.639214 3253 leader_election.cc:288] T c46e36b0bceb45b4bafbf1ffaf8e039e P a62a935ff43d494ca35d0c417943133d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132:44947), e632c553ff4341328acf1cbaca678392 (127.15.255.133:37439), 521bb93035324cecbebed2f6f2a630f8 (127.15.255.134:40821), 6ab8df3c796f4b6cb3e29d5045ad3ea0 (127.15.255.135:34701), 7d1e5c1b047443cbb28d7b0a9744babf (127.15.255.129:37897), 16b3a337bc5e462f83ba895ad8660deb (127.15.255.131:36033)
I20240502 21:28:56.639434 3365 tablet_bootstrap.cc:492] T c46e36b0bceb45b4bafbf1ffaf8e039e P 153026dd7f8943cc93fedf6f48450fb4: No bootstrap required, opened a new log
I20240502 21:28:56.639631 3365 ts_tablet_manager.cc:1367] T c46e36b0bceb45b4bafbf1ffaf8e039e P 153026dd7f8943cc93fedf6f48450fb4: Time spent bootstrapping tablet: real 0.042s user 0.012s sys 0.021s
W20240502 21:28:56.640514 1481 leader_election.cc:341] T c46e36b0bceb45b4bafbf1ffaf8e039e P a62a935ff43d494ca35d0c417943133d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7d1e5c1b047443cbb28d7b0a9744babf (127.15.255.129:37897): Not found: Tablet not found: c46e36b0bceb45b4bafbf1ffaf8e039e
I20240502 21:28:56.641064 1685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c46e36b0bceb45b4bafbf1ffaf8e039e" candidate_uuid: "a62a935ff43d494ca35d0c417943133d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16b3a337bc5e462f83ba895ad8660deb" is_pre_election: true
I20240502 21:28:56.640698 3365 raft_consensus.cc:348] T c46e36b0bceb45b4bafbf1ffaf8e039e P 153026dd7f8943cc93fedf6f48450fb4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
W20240502 21:28:56.641677 1483 leader_election.cc:341] T c46e36b0bceb45b4bafbf1ffaf8e039e P a62a935ff43d494ca35d0c417943133d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 16b3a337bc5e462f83ba895ad8660deb (127.15.255.131:36033): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:28:56.641717 3365 raft_consensus.cc:374] T c46e36b0bceb45b4bafbf1ffaf8e039e P 153026dd7f8943cc93fedf6f48450fb4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:56.641808 3365 raft_consensus.cc:729] T c46e36b0bceb45b4bafbf1ffaf8e039e P 153026dd7f8943cc93fedf6f48450fb4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 153026dd7f8943cc93fedf6f48450fb4, State: Initialized, Role: FOLLOWER
W20240502 21:28:56.642135 1480 leader_election.cc:341] T c46e36b0bceb45b4bafbf1ffaf8e039e P a62a935ff43d494ca35d0c417943133d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e632c553ff4341328acf1cbaca678392 (127.15.255.133:37439): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:28:56.642040 3365 consensus_queue.cc:260] T c46e36b0bceb45b4bafbf1ffaf8e039e P 153026dd7f8943cc93fedf6f48450fb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:28:56.642284 1480 leader_election.cc:302] T c46e36b0bceb45b4bafbf1ffaf8e039e P a62a935ff43d494ca35d0c417943133d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 5 responses out of 7 voters: 1 yes votes; 4 no votes. yes voters: a62a935ff43d494ca35d0c417943133d; no voters: 153026dd7f8943cc93fedf6f48450fb4, 16b3a337bc5e462f83ba895ad8660deb, 7d1e5c1b047443cbb28d7b0a9744babf, e632c553ff4341328acf1cbaca678392
I20240502 21:28:56.642580 3365 ts_tablet_manager.cc:1398] T c46e36b0bceb45b4bafbf1ffaf8e039e P 153026dd7f8943cc93fedf6f48450fb4: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 21:28:56.642905 3365 tablet_bootstrap.cc:492] T 05e885ae741c43808c6dfbcdc5769942 P 153026dd7f8943cc93fedf6f48450fb4: Bootstrap starting.
I20240502 21:28:56.642861 2089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c46e36b0bceb45b4bafbf1ffaf8e039e" candidate_uuid: "a62a935ff43d494ca35d0c417943133d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "521bb93035324cecbebed2f6f2a630f8" is_pre_election: true
I20240502 21:28:56.644318 3364 consensus_queue.cc:260] T c46e36b0bceb45b4bafbf1ffaf8e039e P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:28:56.645303 3365 tablet_bootstrap.cc:654] T 05e885ae741c43808c6dfbcdc5769942 P 153026dd7f8943cc93fedf6f48450fb4: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:56.641235 2227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c46e36b0bceb45b4bafbf1ffaf8e039e" candidate_uuid: "a62a935ff43d494ca35d0c417943133d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" is_pre_election: true
I20240502 21:28:56.645967 2227 raft_consensus.cc:2454] T c46e36b0bceb45b4bafbf1ffaf8e039e P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a62a935ff43d494ca35d0c417943133d in term 0.
I20240502 21:28:56.646451 3364 ts_tablet_manager.cc:1398] T c46e36b0bceb45b4bafbf1ffaf8e039e P 6ab8df3c796f4b6cb3e29d5045ad3ea0: Time spent starting tablet: real 0.010s user 0.000s sys 0.002s
I20240502 21:28:56.646908 3364 tablet_bootstrap.cc:492] T aae7b5a01a0543afb1b5a7f44a9a864c P 6ab8df3c796f4b6cb3e29d5045ad3ea0: Bootstrap starting.
I20240502 21:28:56.648355 3364 tablet_bootstrap.cc:654] T aae7b5a01a0543afb1b5a7f44a9a864c P 6ab8df3c796f4b6cb3e29d5045ad3ea0: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:56.648985 3253 raft_consensus.cc:2726] T c46e36b0bceb45b4bafbf1ffaf8e039e P a62a935ff43d494ca35d0c417943133d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240502 21:28:56.649309 1480 leader_election.cc:341] T c46e36b0bceb45b4bafbf1ffaf8e039e P a62a935ff43d494ca35d0c417943133d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 521bb93035324cecbebed2f6f2a630f8 (127.15.255.134:40821): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:28:56.651103 3365 tablet_bootstrap.cc:492] T 05e885ae741c43808c6dfbcdc5769942 P 153026dd7f8943cc93fedf6f48450fb4: No bootstrap required, opened a new log
I20240502 21:28:56.651268 3365 ts_tablet_manager.cc:1367] T 05e885ae741c43808c6dfbcdc5769942 P 153026dd7f8943cc93fedf6f48450fb4: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240502 21:28:56.651965 3365 raft_consensus.cc:348] T 05e885ae741c43808c6dfbcdc5769942 P 153026dd7f8943cc93fedf6f48450fb4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } }
I20240502 21:28:56.652237 3365 raft_consensus.cc:374] T 05e885ae741c43808c6dfbcdc5769942 P 153026dd7f8943cc93fedf6f48450fb4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:56.652315 3365 raft_consensus.cc:729] T 05e885ae741c43808c6dfbcdc5769942 P 153026dd7f8943cc93fedf6f48450fb4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 153026dd7f8943cc93fedf6f48450fb4, State: Initialized, Role: FOLLOWER
I20240502 21:28:56.652557 3365 consensus_queue.cc:260] T 05e885ae741c43808c6dfbcdc5769942 P 153026dd7f8943cc93fedf6f48450fb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } }
I20240502 21:28:56.653105 3365 ts_tablet_manager.cc:1398] T 05e885ae741c43808c6dfbcdc5769942 P 153026dd7f8943cc93fedf6f48450fb4: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 21:28:56.653409 3365 tablet_bootstrap.cc:492] T aae7b5a01a0543afb1b5a7f44a9a864c P 153026dd7f8943cc93fedf6f48450fb4: Bootstrap starting.
I20240502 21:28:56.654807 3365 tablet_bootstrap.cc:654] T aae7b5a01a0543afb1b5a7f44a9a864c P 153026dd7f8943cc93fedf6f48450fb4: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:56.659791 3372 tablet_bootstrap.cc:492] T 05e885ae741c43808c6dfbcdc5769942 P 7d1e5c1b047443cbb28d7b0a9744babf: No bootstrap required, opened a new log
I20240502 21:28:56.660014 3366 tablet_bootstrap.cc:492] T c46e36b0bceb45b4bafbf1ffaf8e039e P 521bb93035324cecbebed2f6f2a630f8: No bootstrap required, opened a new log
I20240502 21:28:56.660240 3372 ts_tablet_manager.cc:1367] T 05e885ae741c43808c6dfbcdc5769942 P 7d1e5c1b047443cbb28d7b0a9744babf: Time spent bootstrapping tablet: real 0.032s user 0.004s sys 0.000s
I20240502 21:28:56.660624 3367 tablet_bootstrap.cc:492] T aae7b5a01a0543afb1b5a7f44a9a864c P 16b3a337bc5e462f83ba895ad8660deb: Bootstrap starting.
I20240502 21:28:56.661293 3366 ts_tablet_manager.cc:1367] T c46e36b0bceb45b4bafbf1ffaf8e039e P 521bb93035324cecbebed2f6f2a630f8: Time spent bootstrapping tablet: real 0.039s user 0.003s sys 0.003s
I20240502 21:28:56.662009 3372 raft_consensus.cc:348] T 05e885ae741c43808c6dfbcdc5769942 P 7d1e5c1b047443cbb28d7b0a9744babf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } }
I20240502 21:28:56.662703 3364 tablet_bootstrap.cc:492] T aae7b5a01a0543afb1b5a7f44a9a864c P 6ab8df3c796f4b6cb3e29d5045ad3ea0: No bootstrap required, opened a new log
I20240502 21:28:56.662907 3364 ts_tablet_manager.cc:1367] T aae7b5a01a0543afb1b5a7f44a9a864c P 6ab8df3c796f4b6cb3e29d5045ad3ea0: Time spent bootstrapping tablet: real 0.016s user 0.006s sys 0.008s
I20240502 21:28:56.663116 3372 raft_consensus.cc:374] T 05e885ae741c43808c6dfbcdc5769942 P 7d1e5c1b047443cbb28d7b0a9744babf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:56.663224 3372 raft_consensus.cc:729] T 05e885ae741c43808c6dfbcdc5769942 P 7d1e5c1b047443cbb28d7b0a9744babf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d1e5c1b047443cbb28d7b0a9744babf, State: Initialized, Role: FOLLOWER
I20240502 21:28:56.662932 3366 raft_consensus.cc:348] T c46e36b0bceb45b4bafbf1ffaf8e039e P 521bb93035324cecbebed2f6f2a630f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:28:56.663578 3366 raft_consensus.cc:374] T c46e36b0bceb45b4bafbf1ffaf8e039e P 521bb93035324cecbebed2f6f2a630f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:56.663908 3364 raft_consensus.cc:348] T aae7b5a01a0543afb1b5a7f44a9a864c P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } }
I20240502 21:28:56.663195 3367 tablet_bootstrap.cc:654] T aae7b5a01a0543afb1b5a7f44a9a864c P 16b3a337bc5e462f83ba895ad8660deb: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:56.664191 3364 raft_consensus.cc:374] T aae7b5a01a0543afb1b5a7f44a9a864c P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:56.664382 3364 raft_consensus.cc:729] T aae7b5a01a0543afb1b5a7f44a9a864c P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ab8df3c796f4b6cb3e29d5045ad3ea0, State: Initialized, Role: FOLLOWER
I20240502 21:28:56.664788 3371 tablet_bootstrap.cc:492] T 05e885ae741c43808c6dfbcdc5769942 P a62a935ff43d494ca35d0c417943133d: No bootstrap required, opened a new log
I20240502 21:28:56.664952 3371 ts_tablet_manager.cc:1367] T 05e885ae741c43808c6dfbcdc5769942 P a62a935ff43d494ca35d0c417943133d: Time spent bootstrapping tablet: real 0.030s user 0.008s sys 0.011s
I20240502 21:28:56.664718 3364 consensus_queue.cc:260] T aae7b5a01a0543afb1b5a7f44a9a864c P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } }
I20240502 21:28:56.665596 3365 tablet_bootstrap.cc:492] T aae7b5a01a0543afb1b5a7f44a9a864c P 153026dd7f8943cc93fedf6f48450fb4: No bootstrap required, opened a new log
I20240502 21:28:56.665746 3365 ts_tablet_manager.cc:1367] T aae7b5a01a0543afb1b5a7f44a9a864c P 153026dd7f8943cc93fedf6f48450fb4: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.005s
I20240502 21:28:56.665787 3364 ts_tablet_manager.cc:1398] T aae7b5a01a0543afb1b5a7f44a9a864c P 6ab8df3c796f4b6cb3e29d5045ad3ea0: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 21:28:56.665690 3371 raft_consensus.cc:348] T 05e885ae741c43808c6dfbcdc5769942 P a62a935ff43d494ca35d0c417943133d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } }
I20240502 21:28:56.665942 3371 raft_consensus.cc:374] T 05e885ae741c43808c6dfbcdc5769942 P a62a935ff43d494ca35d0c417943133d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:56.666029 3371 raft_consensus.cc:729] T 05e885ae741c43808c6dfbcdc5769942 P a62a935ff43d494ca35d0c417943133d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a62a935ff43d494ca35d0c417943133d, State: Initialized, Role: FOLLOWER
I20240502 21:28:56.666378 3364 tablet_bootstrap.cc:492] T 05e885ae741c43808c6dfbcdc5769942 P 6ab8df3c796f4b6cb3e29d5045ad3ea0: Bootstrap starting.
I20240502 21:28:56.666302 3371 consensus_queue.cc:260] T 05e885ae741c43808c6dfbcdc5769942 P a62a935ff43d494ca35d0c417943133d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } }
I20240502 21:28:56.666435 3365 raft_consensus.cc:348] T aae7b5a01a0543afb1b5a7f44a9a864c P 153026dd7f8943cc93fedf6f48450fb4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } }
I20240502 21:28:56.666676 3365 raft_consensus.cc:374] T aae7b5a01a0543afb1b5a7f44a9a864c P 153026dd7f8943cc93fedf6f48450fb4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:56.666769 3365 raft_consensus.cc:729] T aae7b5a01a0543afb1b5a7f44a9a864c P 153026dd7f8943cc93fedf6f48450fb4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 153026dd7f8943cc93fedf6f48450fb4, State: Initialized, Role: FOLLOWER
I20240502 21:28:56.667043 3371 ts_tablet_manager.cc:1398] T 05e885ae741c43808c6dfbcdc5769942 P a62a935ff43d494ca35d0c417943133d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:28:56.666980 3365 consensus_queue.cc:260] T aae7b5a01a0543afb1b5a7f44a9a864c P 153026dd7f8943cc93fedf6f48450fb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } }
I20240502 21:28:56.667357 3371 tablet_bootstrap.cc:492] T aae7b5a01a0543afb1b5a7f44a9a864c P a62a935ff43d494ca35d0c417943133d: Bootstrap starting.
I20240502 21:28:56.668742 3364 tablet_bootstrap.cc:654] T 05e885ae741c43808c6dfbcdc5769942 P 6ab8df3c796f4b6cb3e29d5045ad3ea0: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:56.669335 3365 ts_tablet_manager.cc:1398] T aae7b5a01a0543afb1b5a7f44a9a864c P 153026dd7f8943cc93fedf6f48450fb4: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20240502 21:28:56.669903 3371 tablet_bootstrap.cc:654] T aae7b5a01a0543afb1b5a7f44a9a864c P a62a935ff43d494ca35d0c417943133d: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:56.673523 3369 tablet_bootstrap.cc:492] T aae7b5a01a0543afb1b5a7f44a9a864c P e632c553ff4341328acf1cbaca678392: No bootstrap required, opened a new log
I20240502 21:28:56.675249 3369 ts_tablet_manager.cc:1367] T aae7b5a01a0543afb1b5a7f44a9a864c P e632c553ff4341328acf1cbaca678392: Time spent bootstrapping tablet: real 0.058s user 0.004s sys 0.002s
I20240502 21:28:56.676052 3369 raft_consensus.cc:348] T aae7b5a01a0543afb1b5a7f44a9a864c P e632c553ff4341328acf1cbaca678392 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } }
I20240502 21:28:56.676329 3369 raft_consensus.cc:374] T aae7b5a01a0543afb1b5a7f44a9a864c P e632c553ff4341328acf1cbaca678392 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:56.676442 3369 raft_consensus.cc:729] T aae7b5a01a0543afb1b5a7f44a9a864c P e632c553ff4341328acf1cbaca678392 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e632c553ff4341328acf1cbaca678392, State: Initialized, Role: FOLLOWER
I20240502 21:28:56.676681 3369 consensus_queue.cc:260] T aae7b5a01a0543afb1b5a7f44a9a864c P e632c553ff4341328acf1cbaca678392 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } }
I20240502 21:28:56.677553 3367 tablet_bootstrap.cc:492] T aae7b5a01a0543afb1b5a7f44a9a864c P 16b3a337bc5e462f83ba895ad8660deb: No bootstrap required, opened a new log
I20240502 21:28:56.678145 3367 ts_tablet_manager.cc:1367] T aae7b5a01a0543afb1b5a7f44a9a864c P 16b3a337bc5e462f83ba895ad8660deb: Time spent bootstrapping tablet: real 0.021s user 0.003s sys 0.006s
I20240502 21:28:56.678953 3367 raft_consensus.cc:348] T aae7b5a01a0543afb1b5a7f44a9a864c P 16b3a337bc5e462f83ba895ad8660deb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } }
I20240502 21:28:56.679308 3367 raft_consensus.cc:374] T aae7b5a01a0543afb1b5a7f44a9a864c P 16b3a337bc5e462f83ba895ad8660deb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:56.679373 3364 tablet_bootstrap.cc:492] T 05e885ae741c43808c6dfbcdc5769942 P 6ab8df3c796f4b6cb3e29d5045ad3ea0: No bootstrap required, opened a new log
I20240502 21:28:56.679487 3367 raft_consensus.cc:729] T aae7b5a01a0543afb1b5a7f44a9a864c P 16b3a337bc5e462f83ba895ad8660deb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16b3a337bc5e462f83ba895ad8660deb, State: Initialized, Role: FOLLOWER
I20240502 21:28:56.679520 3364 ts_tablet_manager.cc:1367] T 05e885ae741c43808c6dfbcdc5769942 P 6ab8df3c796f4b6cb3e29d5045ad3ea0: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.000s
I20240502 21:28:56.679733 3372 consensus_queue.cc:260] T 05e885ae741c43808c6dfbcdc5769942 P 7d1e5c1b047443cbb28d7b0a9744babf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } }
I20240502 21:28:56.679795 3367 consensus_queue.cc:260] T aae7b5a01a0543afb1b5a7f44a9a864c P 16b3a337bc5e462f83ba895ad8660deb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } }
I20240502 21:28:56.680316 3364 raft_consensus.cc:348] T 05e885ae741c43808c6dfbcdc5769942 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } }
I20240502 21:28:56.680581 3364 raft_consensus.cc:374] T 05e885ae741c43808c6dfbcdc5769942 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:56.680680 3364 raft_consensus.cc:729] T 05e885ae741c43808c6dfbcdc5769942 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ab8df3c796f4b6cb3e29d5045ad3ea0, State: Initialized, Role: FOLLOWER
I20240502 21:28:56.663690 3366 raft_consensus.cc:729] T c46e36b0bceb45b4bafbf1ffaf8e039e P 521bb93035324cecbebed2f6f2a630f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 521bb93035324cecbebed2f6f2a630f8, State: Initialized, Role: FOLLOWER
I20240502 21:28:56.682374 3364 consensus_queue.cc:260] T 05e885ae741c43808c6dfbcdc5769942 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } }
I20240502 21:28:56.683070 3364 ts_tablet_manager.cc:1398] T 05e885ae741c43808c6dfbcdc5769942 P 6ab8df3c796f4b6cb3e29d5045ad3ea0: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 21:28:56.683892 3367 ts_tablet_manager.cc:1398] T aae7b5a01a0543afb1b5a7f44a9a864c P 16b3a337bc5e462f83ba895ad8660deb: Time spent starting tablet: real 0.006s user 0.004s sys 0.001s
I20240502 21:28:56.684228 3371 tablet_bootstrap.cc:492] T aae7b5a01a0543afb1b5a7f44a9a864c P a62a935ff43d494ca35d0c417943133d: No bootstrap required, opened a new log
I20240502 21:28:56.684371 3371 ts_tablet_manager.cc:1367] T aae7b5a01a0543afb1b5a7f44a9a864c P a62a935ff43d494ca35d0c417943133d: Time spent bootstrapping tablet: real 0.017s user 0.008s sys 0.007s
I20240502 21:28:56.684895 3367 tablet_bootstrap.cc:492] T c46e36b0bceb45b4bafbf1ffaf8e039e P 16b3a337bc5e462f83ba895ad8660deb: Bootstrap starting.
I20240502 21:28:56.685106 3371 raft_consensus.cc:348] T aae7b5a01a0543afb1b5a7f44a9a864c P a62a935ff43d494ca35d0c417943133d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } }
I20240502 21:28:56.685345 3371 raft_consensus.cc:374] T aae7b5a01a0543afb1b5a7f44a9a864c P a62a935ff43d494ca35d0c417943133d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:56.685441 3371 raft_consensus.cc:729] T aae7b5a01a0543afb1b5a7f44a9a864c P a62a935ff43d494ca35d0c417943133d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a62a935ff43d494ca35d0c417943133d, State: Initialized, Role: FOLLOWER
I20240502 21:28:56.685668 3371 consensus_queue.cc:260] T aae7b5a01a0543afb1b5a7f44a9a864c P a62a935ff43d494ca35d0c417943133d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } }
I20240502 21:28:56.686117 3366 consensus_queue.cc:260] T c46e36b0bceb45b4bafbf1ffaf8e039e P 521bb93035324cecbebed2f6f2a630f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:28:56.686422 3367 tablet_bootstrap.cc:654] T c46e36b0bceb45b4bafbf1ffaf8e039e P 16b3a337bc5e462f83ba895ad8660deb: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:56.687949 3293 raft_consensus.cc:482] T 05e885ae741c43808c6dfbcdc5769942 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:28:56.687990 3371 ts_tablet_manager.cc:1398] T aae7b5a01a0543afb1b5a7f44a9a864c P a62a935ff43d494ca35d0c417943133d: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240502 21:28:56.688087 3293 raft_consensus.cc:504] T 05e885ae741c43808c6dfbcdc5769942 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } }
I20240502 21:28:56.689345 3293 leader_election.cc:288] T 05e885ae741c43808c6dfbcdc5769942 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7d1e5c1b047443cbb28d7b0a9744babf (127.15.255.129:37897), a62a935ff43d494ca35d0c417943133d (127.15.255.130:37991), 16b3a337bc5e462f83ba895ad8660deb (127.15.255.131:36033), 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132:44947), e632c553ff4341328acf1cbaca678392 (127.15.255.133:37439), 521bb93035324cecbebed2f6f2a630f8 (127.15.255.134:40821)
I20240502 21:28:56.690075 1818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05e885ae741c43808c6dfbcdc5769942" candidate_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "153026dd7f8943cc93fedf6f48450fb4" is_pre_election: true
I20240502 21:28:56.690277 1818 raft_consensus.cc:2454] T 05e885ae741c43808c6dfbcdc5769942 P 153026dd7f8943cc93fedf6f48450fb4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ab8df3c796f4b6cb3e29d5045ad3ea0 in term 0.
I20240502 21:28:56.690421 1954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05e885ae741c43808c6dfbcdc5769942" candidate_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e632c553ff4341328acf1cbaca678392" is_pre_election: true
I20240502 21:28:56.691035 2089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05e885ae741c43808c6dfbcdc5769942" candidate_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "521bb93035324cecbebed2f6f2a630f8" is_pre_election: true
W20240502 21:28:56.691687 2158 leader_election.cc:341] T 05e885ae741c43808c6dfbcdc5769942 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 521bb93035324cecbebed2f6f2a630f8 (127.15.255.134:40821): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:28:56.691807 3367 tablet_bootstrap.cc:492] T c46e36b0bceb45b4bafbf1ffaf8e039e P 16b3a337bc5e462f83ba895ad8660deb: No bootstrap required, opened a new log
W20240502 21:28:56.691898 2158 leader_election.cc:341] T 05e885ae741c43808c6dfbcdc5769942 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e632c553ff4341328acf1cbaca678392 (127.15.255.133:37439): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:28:56.691951 3367 ts_tablet_manager.cc:1367] T c46e36b0bceb45b4bafbf1ffaf8e039e P 16b3a337bc5e462f83ba895ad8660deb: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.004s
I20240502 21:28:56.692190 1549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05e885ae741c43808c6dfbcdc5769942" candidate_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a62a935ff43d494ca35d0c417943133d" is_pre_election: true
I20240502 21:28:56.692391 1549 raft_consensus.cc:2454] T 05e885ae741c43808c6dfbcdc5769942 P a62a935ff43d494ca35d0c417943133d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ab8df3c796f4b6cb3e29d5045ad3ea0 in term 0.
I20240502 21:28:56.692821 3366 ts_tablet_manager.cc:1398] T c46e36b0bceb45b4bafbf1ffaf8e039e P 521bb93035324cecbebed2f6f2a630f8: Time spent starting tablet: real 0.030s user 0.001s sys 0.001s
I20240502 21:28:56.692783 3367 raft_consensus.cc:348] T c46e36b0bceb45b4bafbf1ffaf8e039e P 16b3a337bc5e462f83ba895ad8660deb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:28:56.688396 3372 ts_tablet_manager.cc:1398] T 05e885ae741c43808c6dfbcdc5769942 P 7d1e5c1b047443cbb28d7b0a9744babf: Time spent starting tablet: real 0.028s user 0.000s sys 0.002s
I20240502 21:28:56.693024 3367 raft_consensus.cc:374] T c46e36b0bceb45b4bafbf1ffaf8e039e P 16b3a337bc5e462f83ba895ad8660deb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:56.689350 1415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05e885ae741c43808c6dfbcdc5769942" candidate_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" is_pre_election: true
I20240502 21:28:56.693099 3367 raft_consensus.cc:729] T c46e36b0bceb45b4bafbf1ffaf8e039e P 16b3a337bc5e462f83ba895ad8660deb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16b3a337bc5e462f83ba895ad8660deb, State: Initialized, Role: FOLLOWER
I20240502 21:28:56.693172 1415 raft_consensus.cc:2454] T 05e885ae741c43808c6dfbcdc5769942 P 7d1e5c1b047443cbb28d7b0a9744babf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ab8df3c796f4b6cb3e29d5045ad3ea0 in term 0.
I20240502 21:28:56.693287 1685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05e885ae741c43808c6dfbcdc5769942" candidate_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16b3a337bc5e462f83ba895ad8660deb" is_pre_election: true
I20240502 21:28:56.693499 3372 tablet_bootstrap.cc:492] T aae7b5a01a0543afb1b5a7f44a9a864c P 7d1e5c1b047443cbb28d7b0a9744babf: Bootstrap starting.
I20240502 21:28:56.693333 3367 consensus_queue.cc:260] T c46e36b0bceb45b4bafbf1ffaf8e039e P 16b3a337bc5e462f83ba895ad8660deb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:28:56.694010 3367 ts_tablet_manager.cc:1398] T c46e36b0bceb45b4bafbf1ffaf8e039e P 16b3a337bc5e462f83ba895ad8660deb: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:28:56.694018 2159 leader_election.cc:302] T 05e885ae741c43808c6dfbcdc5769942 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 6 responses out of 7 voters: 4 yes votes; 2 no votes. yes voters: 153026dd7f8943cc93fedf6f48450fb4, 6ab8df3c796f4b6cb3e29d5045ad3ea0, 7d1e5c1b047443cbb28d7b0a9744babf, a62a935ff43d494ca35d0c417943133d; no voters: 521bb93035324cecbebed2f6f2a630f8, e632c553ff4341328acf1cbaca678392
I20240502 21:28:56.694295 3367 tablet_bootstrap.cc:492] T 05e885ae741c43808c6dfbcdc5769942 P 16b3a337bc5e462f83ba895ad8660deb: Bootstrap starting.
W20240502 21:28:56.694499 2161 leader_election.cc:341] T 05e885ae741c43808c6dfbcdc5769942 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 16b3a337bc5e462f83ba895ad8660deb (127.15.255.131:36033): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:28:56.694890 3293 raft_consensus.cc:2781] T 05e885ae741c43808c6dfbcdc5769942 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:28:56.695005 3293 raft_consensus.cc:482] T 05e885ae741c43808c6dfbcdc5769942 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:28:56.695096 3293 raft_consensus.cc:3037] T 05e885ae741c43808c6dfbcdc5769942 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:56.695530 3367 tablet_bootstrap.cc:654] T 05e885ae741c43808c6dfbcdc5769942 P 16b3a337bc5e462f83ba895ad8660deb: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:56.688782 3369 ts_tablet_manager.cc:1398] T aae7b5a01a0543afb1b5a7f44a9a864c P e632c553ff4341328acf1cbaca678392: Time spent starting tablet: real 0.013s user 0.000s sys 0.002s
I20240502 21:28:56.698249 3366 tablet_bootstrap.cc:492] T aae7b5a01a0543afb1b5a7f44a9a864c P 521bb93035324cecbebed2f6f2a630f8: Bootstrap starting.
I20240502 21:28:56.698541 3369 tablet_bootstrap.cc:492] T 05e885ae741c43808c6dfbcdc5769942 P e632c553ff4341328acf1cbaca678392: Bootstrap starting.
I20240502 21:28:56.699573 3367 tablet_bootstrap.cc:492] T 05e885ae741c43808c6dfbcdc5769942 P 16b3a337bc5e462f83ba895ad8660deb: No bootstrap required, opened a new log
I20240502 21:28:56.699769 3367 ts_tablet_manager.cc:1367] T 05e885ae741c43808c6dfbcdc5769942 P 16b3a337bc5e462f83ba895ad8660deb: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20240502 21:28:56.700057 3116 raft_consensus.cc:482] T c46e36b0bceb45b4bafbf1ffaf8e039e P 521bb93035324cecbebed2f6f2a630f8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:28:56.700240 3116 raft_consensus.cc:504] T c46e36b0bceb45b4bafbf1ffaf8e039e P 521bb93035324cecbebed2f6f2a630f8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:28:56.700513 3366 tablet_bootstrap.cc:654] T aae7b5a01a0543afb1b5a7f44a9a864c P 521bb93035324cecbebed2f6f2a630f8: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:56.700479 3367 raft_consensus.cc:348] T 05e885ae741c43808c6dfbcdc5769942 P 16b3a337bc5e462f83ba895ad8660deb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } }
I20240502 21:28:56.700726 3367 raft_consensus.cc:374] T 05e885ae741c43808c6dfbcdc5769942 P 16b3a337bc5e462f83ba895ad8660deb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:56.700820 3367 raft_consensus.cc:729] T 05e885ae741c43808c6dfbcdc5769942 P 16b3a337bc5e462f83ba895ad8660deb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16b3a337bc5e462f83ba895ad8660deb, State: Initialized, Role: FOLLOWER
I20240502 21:28:56.701083 3367 consensus_queue.cc:260] T 05e885ae741c43808c6dfbcdc5769942 P 16b3a337bc5e462f83ba895ad8660deb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } }
I20240502 21:28:56.701592 3116 leader_election.cc:288] T c46e36b0bceb45b4bafbf1ffaf8e039e P 521bb93035324cecbebed2f6f2a630f8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132:44947), e632c553ff4341328acf1cbaca678392 (127.15.255.133:37439), 6ab8df3c796f4b6cb3e29d5045ad3ea0 (127.15.255.135:34701), a62a935ff43d494ca35d0c417943133d (127.15.255.130:37991), 7d1e5c1b047443cbb28d7b0a9744babf (127.15.255.129:37897), 16b3a337bc5e462f83ba895ad8660deb (127.15.255.131:36033)
I20240502 21:28:56.701736 3367 ts_tablet_manager.cc:1398] T 05e885ae741c43808c6dfbcdc5769942 P 16b3a337bc5e462f83ba895ad8660deb: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 21:28:56.702288 3369 tablet_bootstrap.cc:654] T 05e885ae741c43808c6dfbcdc5769942 P e632c553ff4341328acf1cbaca678392: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:56.702453 3372 tablet_bootstrap.cc:654] T aae7b5a01a0543afb1b5a7f44a9a864c P 7d1e5c1b047443cbb28d7b0a9744babf: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:56.702446 1685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c46e36b0bceb45b4bafbf1ffaf8e039e" candidate_uuid: "521bb93035324cecbebed2f6f2a630f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16b3a337bc5e462f83ba895ad8660deb" is_pre_election: true
I20240502 21:28:56.702631 1685 raft_consensus.cc:2454] T c46e36b0bceb45b4bafbf1ffaf8e039e P 16b3a337bc5e462f83ba895ad8660deb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 521bb93035324cecbebed2f6f2a630f8 in term 0.
I20240502 21:28:56.702855 1818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c46e36b0bceb45b4bafbf1ffaf8e039e" candidate_uuid: "521bb93035324cecbebed2f6f2a630f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "153026dd7f8943cc93fedf6f48450fb4" is_pre_election: true
I20240502 21:28:56.703078 1818 raft_consensus.cc:2454] T c46e36b0bceb45b4bafbf1ffaf8e039e P 153026dd7f8943cc93fedf6f48450fb4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 521bb93035324cecbebed2f6f2a630f8 in term 0.
I20240502 21:28:56.703413 2227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c46e36b0bceb45b4bafbf1ffaf8e039e" candidate_uuid: "521bb93035324cecbebed2f6f2a630f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" is_pre_election: true
I20240502 21:28:56.703647 2227 raft_consensus.cc:2454] T c46e36b0bceb45b4bafbf1ffaf8e039e P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 521bb93035324cecbebed2f6f2a630f8 in term 0.
I20240502 21:28:56.703979 1415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c46e36b0bceb45b4bafbf1ffaf8e039e" candidate_uuid: "521bb93035324cecbebed2f6f2a630f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" is_pre_election: true
I20240502 21:28:56.704797 3372 tablet_bootstrap.cc:492] T aae7b5a01a0543afb1b5a7f44a9a864c P 7d1e5c1b047443cbb28d7b0a9744babf: No bootstrap required, opened a new log
W20240502 21:28:56.704798 2023 leader_election.cc:341] T c46e36b0bceb45b4bafbf1ffaf8e039e P 521bb93035324cecbebed2f6f2a630f8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7d1e5c1b047443cbb28d7b0a9744babf (127.15.255.129:37897): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:28:56.704955 3372 ts_tablet_manager.cc:1367] T aae7b5a01a0543afb1b5a7f44a9a864c P 7d1e5c1b047443cbb28d7b0a9744babf: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.002s
I20240502 21:28:56.705485 3372 raft_consensus.cc:348] T aae7b5a01a0543afb1b5a7f44a9a864c P 7d1e5c1b047443cbb28d7b0a9744babf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } }
I20240502 21:28:56.705600 1954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c46e36b0bceb45b4bafbf1ffaf8e039e" candidate_uuid: "521bb93035324cecbebed2f6f2a630f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e632c553ff4341328acf1cbaca678392" is_pre_election: true
I20240502 21:28:56.705695 3372 raft_consensus.cc:374] T aae7b5a01a0543afb1b5a7f44a9a864c P 7d1e5c1b047443cbb28d7b0a9744babf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:56.705762 3372 raft_consensus.cc:729] T aae7b5a01a0543afb1b5a7f44a9a864c P 7d1e5c1b047443cbb28d7b0a9744babf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d1e5c1b047443cbb28d7b0a9744babf, State: Initialized, Role: FOLLOWER
I20240502 21:28:56.705968 3372 consensus_queue.cc:260] T aae7b5a01a0543afb1b5a7f44a9a864c P 7d1e5c1b047443cbb28d7b0a9744babf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } }
I20240502 21:28:56.706064 2023 leader_election.cc:302] T c46e36b0bceb45b4bafbf1ffaf8e039e P 521bb93035324cecbebed2f6f2a630f8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 5 responses out of 7 voters: 4 yes votes; 1 no votes. yes voters: 153026dd7f8943cc93fedf6f48450fb4, 16b3a337bc5e462f83ba895ad8660deb, 521bb93035324cecbebed2f6f2a630f8, 6ab8df3c796f4b6cb3e29d5045ad3ea0; no voters: 7d1e5c1b047443cbb28d7b0a9744babf
W20240502 21:28:56.706547 2022 leader_election.cc:341] T c46e36b0bceb45b4bafbf1ffaf8e039e P 521bb93035324cecbebed2f6f2a630f8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e632c553ff4341328acf1cbaca678392 (127.15.255.133:37439): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:28:56.706580 3372 ts_tablet_manager.cc:1398] T aae7b5a01a0543afb1b5a7f44a9a864c P 7d1e5c1b047443cbb28d7b0a9744babf: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 21:28:56.707001 3372 tablet_bootstrap.cc:492] T c46e36b0bceb45b4bafbf1ffaf8e039e P 7d1e5c1b047443cbb28d7b0a9744babf: Bootstrap starting.
I20240502 21:28:56.707044 1549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c46e36b0bceb45b4bafbf1ffaf8e039e" candidate_uuid: "521bb93035324cecbebed2f6f2a630f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a62a935ff43d494ca35d0c417943133d" is_pre_election: true
I20240502 21:28:56.707240 1549 raft_consensus.cc:2454] T c46e36b0bceb45b4bafbf1ffaf8e039e P a62a935ff43d494ca35d0c417943133d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 521bb93035324cecbebed2f6f2a630f8 in term 0.
I20240502 21:28:56.708109 3116 raft_consensus.cc:2781] T c46e36b0bceb45b4bafbf1ffaf8e039e P 521bb93035324cecbebed2f6f2a630f8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:28:56.708212 3116 raft_consensus.cc:482] T c46e36b0bceb45b4bafbf1ffaf8e039e P 521bb93035324cecbebed2f6f2a630f8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:28:56.708230 3372 tablet_bootstrap.cc:654] T c46e36b0bceb45b4bafbf1ffaf8e039e P 7d1e5c1b047443cbb28d7b0a9744babf: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:56.708317 3116 raft_consensus.cc:3037] T c46e36b0bceb45b4bafbf1ffaf8e039e P 521bb93035324cecbebed2f6f2a630f8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:56.709322 3369 tablet_bootstrap.cc:492] T 05e885ae741c43808c6dfbcdc5769942 P e632c553ff4341328acf1cbaca678392: No bootstrap required, opened a new log
I20240502 21:28:56.709497 3369 ts_tablet_manager.cc:1367] T 05e885ae741c43808c6dfbcdc5769942 P e632c553ff4341328acf1cbaca678392: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.000s
I20240502 21:28:56.710289 3369 raft_consensus.cc:348] T 05e885ae741c43808c6dfbcdc5769942 P e632c553ff4341328acf1cbaca678392 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } }
I20240502 21:28:56.710603 3369 raft_consensus.cc:374] T 05e885ae741c43808c6dfbcdc5769942 P e632c553ff4341328acf1cbaca678392 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:56.710681 3369 raft_consensus.cc:729] T 05e885ae741c43808c6dfbcdc5769942 P e632c553ff4341328acf1cbaca678392 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e632c553ff4341328acf1cbaca678392, State: Initialized, Role: FOLLOWER
I20240502 21:28:56.710939 3369 consensus_queue.cc:260] T 05e885ae741c43808c6dfbcdc5769942 P e632c553ff4341328acf1cbaca678392 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } }
I20240502 21:28:56.711519 3369 ts_tablet_manager.cc:1398] T 05e885ae741c43808c6dfbcdc5769942 P e632c553ff4341328acf1cbaca678392: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 21:28:56.711865 3369 tablet_bootstrap.cc:492] T c46e36b0bceb45b4bafbf1ffaf8e039e P e632c553ff4341328acf1cbaca678392: Bootstrap starting.
I20240502 21:28:56.713420 3369 tablet_bootstrap.cc:654] T c46e36b0bceb45b4bafbf1ffaf8e039e P e632c553ff4341328acf1cbaca678392: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:56.716698 3293 raft_consensus.cc:504] T 05e885ae741c43808c6dfbcdc5769942 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } }
I20240502 21:28:56.718036 1415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05e885ae741c43808c6dfbcdc5769942" candidate_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d1e5c1b047443cbb28d7b0a9744babf"
I20240502 21:28:56.718263 1415 raft_consensus.cc:3037] T 05e885ae741c43808c6dfbcdc5769942 P 7d1e5c1b047443cbb28d7b0a9744babf [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:56.720290 3116 raft_consensus.cc:504] T c46e36b0bceb45b4bafbf1ffaf8e039e P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:28:56.720949 1549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05e885ae741c43808c6dfbcdc5769942" candidate_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a62a935ff43d494ca35d0c417943133d"
I20240502 21:28:56.721146 1549 raft_consensus.cc:3037] T 05e885ae741c43808c6dfbcdc5769942 P a62a935ff43d494ca35d0c417943133d [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:56.721696 1818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05e885ae741c43808c6dfbcdc5769942" candidate_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "153026dd7f8943cc93fedf6f48450fb4"
I20240502 21:28:56.721887 1818 raft_consensus.cc:3037] T 05e885ae741c43808c6dfbcdc5769942 P 153026dd7f8943cc93fedf6f48450fb4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:56.723522 3369 tablet_bootstrap.cc:492] T c46e36b0bceb45b4bafbf1ffaf8e039e P e632c553ff4341328acf1cbaca678392: No bootstrap required, opened a new log
I20240502 21:28:56.723659 3369 ts_tablet_manager.cc:1367] T c46e36b0bceb45b4bafbf1ffaf8e039e P e632c553ff4341328acf1cbaca678392: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.007s
I20240502 21:28:56.723740 1549 raft_consensus.cc:2454] T 05e885ae741c43808c6dfbcdc5769942 P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ab8df3c796f4b6cb3e29d5045ad3ea0 in term 1.
I20240502 21:28:56.724383 3369 raft_consensus.cc:348] T c46e36b0bceb45b4bafbf1ffaf8e039e P e632c553ff4341328acf1cbaca678392 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:28:56.724651 3369 raft_consensus.cc:374] T c46e36b0bceb45b4bafbf1ffaf8e039e P e632c553ff4341328acf1cbaca678392 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:56.724735 3369 raft_consensus.cc:729] T c46e36b0bceb45b4bafbf1ffaf8e039e P e632c553ff4341328acf1cbaca678392 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e632c553ff4341328acf1cbaca678392, State: Initialized, Role: FOLLOWER
I20240502 21:28:56.724941 1415 raft_consensus.cc:2454] T 05e885ae741c43808c6dfbcdc5769942 P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ab8df3c796f4b6cb3e29d5045ad3ea0 in term 1.
I20240502 21:28:56.724959 3369 consensus_queue.cc:260] T c46e36b0bceb45b4bafbf1ffaf8e039e P e632c553ff4341328acf1cbaca678392 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:28:56.725239 1818 raft_consensus.cc:2454] T 05e885ae741c43808c6dfbcdc5769942 P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ab8df3c796f4b6cb3e29d5045ad3ea0 in term 1.
I20240502 21:28:56.725530 3369 ts_tablet_manager.cc:1398] T c46e36b0bceb45b4bafbf1ffaf8e039e P e632c553ff4341328acf1cbaca678392: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 21:28:56.725607 2160 leader_election.cc:302] T 05e885ae741c43808c6dfbcdc5769942 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 153026dd7f8943cc93fedf6f48450fb4, 6ab8df3c796f4b6cb3e29d5045ad3ea0, 7d1e5c1b047443cbb28d7b0a9744babf, a62a935ff43d494ca35d0c417943133d; no voters:
I20240502 21:28:56.726610 3394 raft_consensus.cc:2781] T 05e885ae741c43808c6dfbcdc5769942 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:28:56.726895 3394 raft_consensus.cc:686] T 05e885ae741c43808c6dfbcdc5769942 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 LEADER]: Becoming Leader. State: Replica: 6ab8df3c796f4b6cb3e29d5045ad3ea0, State: Running, Role: LEADER
I20240502 21:28:56.727326 3372 tablet_bootstrap.cc:492] T c46e36b0bceb45b4bafbf1ffaf8e039e P 7d1e5c1b047443cbb28d7b0a9744babf: No bootstrap required, opened a new log
I20240502 21:28:56.727231 3394 consensus_queue.cc:237] T 05e885ae741c43808c6dfbcdc5769942 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } }
I20240502 21:28:56.727583 3372 ts_tablet_manager.cc:1367] T c46e36b0bceb45b4bafbf1ffaf8e039e P 7d1e5c1b047443cbb28d7b0a9744babf: Time spent bootstrapping tablet: real 0.021s user 0.004s sys 0.002s
I20240502 21:28:56.728402 3372 raft_consensus.cc:348] T c46e36b0bceb45b4bafbf1ffaf8e039e P 7d1e5c1b047443cbb28d7b0a9744babf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:28:56.728838 3372 raft_consensus.cc:374] T c46e36b0bceb45b4bafbf1ffaf8e039e P 7d1e5c1b047443cbb28d7b0a9744babf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:56.728987 3372 raft_consensus.cc:729] T c46e36b0bceb45b4bafbf1ffaf8e039e P 7d1e5c1b047443cbb28d7b0a9744babf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d1e5c1b047443cbb28d7b0a9744babf, State: Initialized, Role: FOLLOWER
I20240502 21:28:56.729338 3372 consensus_queue.cc:260] T c46e36b0bceb45b4bafbf1ffaf8e039e P 7d1e5c1b047443cbb28d7b0a9744babf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:28:56.729683 3122 raft_consensus.cc:482] T aae7b5a01a0543afb1b5a7f44a9a864c P 7d1e5c1b047443cbb28d7b0a9744babf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:28:56.729876 3122 raft_consensus.cc:504] T aae7b5a01a0543afb1b5a7f44a9a864c P 7d1e5c1b047443cbb28d7b0a9744babf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } }
I20240502 21:28:56.731060 3122 leader_election.cc:288] T aae7b5a01a0543afb1b5a7f44a9a864c P 7d1e5c1b047443cbb28d7b0a9744babf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a62a935ff43d494ca35d0c417943133d (127.15.255.130:37991), 16b3a337bc5e462f83ba895ad8660deb (127.15.255.131:36033), 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132:44947), e632c553ff4341328acf1cbaca678392 (127.15.255.133:37439), 521bb93035324cecbebed2f6f2a630f8 (127.15.255.134:40821), 6ab8df3c796f4b6cb3e29d5045ad3ea0 (127.15.255.135:34701)
I20240502 21:28:56.731864 1685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aae7b5a01a0543afb1b5a7f44a9a864c" candidate_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16b3a337bc5e462f83ba895ad8660deb" is_pre_election: true
I20240502 21:28:56.732102 1685 raft_consensus.cc:2454] T aae7b5a01a0543afb1b5a7f44a9a864c P 16b3a337bc5e462f83ba895ad8660deb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d1e5c1b047443cbb28d7b0a9744babf in term 0.
I20240502 21:28:56.732832 3372 ts_tablet_manager.cc:1398] T c46e36b0bceb45b4bafbf1ffaf8e039e P 7d1e5c1b047443cbb28d7b0a9744babf: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20240502 21:28:56.732887 1818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aae7b5a01a0543afb1b5a7f44a9a864c" candidate_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "153026dd7f8943cc93fedf6f48450fb4" is_pre_election: true
I20240502 21:28:56.732858 2089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aae7b5a01a0543afb1b5a7f44a9a864c" candidate_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "521bb93035324cecbebed2f6f2a630f8" is_pre_election: true
I20240502 21:28:56.733093 1818 raft_consensus.cc:2454] T aae7b5a01a0543afb1b5a7f44a9a864c P 153026dd7f8943cc93fedf6f48450fb4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d1e5c1b047443cbb28d7b0a9744babf in term 0.
I20240502 21:28:56.733297 1954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aae7b5a01a0543afb1b5a7f44a9a864c" candidate_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e632c553ff4341328acf1cbaca678392" is_pre_election: true
I20240502 21:28:56.733508 1954 raft_consensus.cc:2454] T aae7b5a01a0543afb1b5a7f44a9a864c P e632c553ff4341328acf1cbaca678392 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d1e5c1b047443cbb28d7b0a9744babf in term 0.
I20240502 21:28:56.733989 1349 leader_election.cc:302] T aae7b5a01a0543afb1b5a7f44a9a864c P 7d1e5c1b047443cbb28d7b0a9744babf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 153026dd7f8943cc93fedf6f48450fb4, 16b3a337bc5e462f83ba895ad8660deb, 7d1e5c1b047443cbb28d7b0a9744babf, e632c553ff4341328acf1cbaca678392; no voters:
W20240502 21:28:56.734886 1347 leader_election.cc:341] T aae7b5a01a0543afb1b5a7f44a9a864c P 7d1e5c1b047443cbb28d7b0a9744babf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 521bb93035324cecbebed2f6f2a630f8 (127.15.255.134:40821): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:28:56.735450 3122 raft_consensus.cc:2781] T aae7b5a01a0543afb1b5a7f44a9a864c P 7d1e5c1b047443cbb28d7b0a9744babf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:28:56.735549 3122 raft_consensus.cc:482] T aae7b5a01a0543afb1b5a7f44a9a864c P 7d1e5c1b047443cbb28d7b0a9744babf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:28:56.735651 3122 raft_consensus.cc:3037] T aae7b5a01a0543afb1b5a7f44a9a864c P 7d1e5c1b047443cbb28d7b0a9744babf [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:56.738443 3122 raft_consensus.cc:504] T aae7b5a01a0543afb1b5a7f44a9a864c P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } }
I20240502 21:28:56.738662 1264 catalog_manager.cc:5484] T 05e885ae741c43808c6dfbcdc5769942 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6ab8df3c796f4b6cb3e29d5045ad3ea0 (127.15.255.135). New cstate: current_term: 1 leader_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } health_report { overall_health: HEALTHY } } }
I20240502 21:28:56.739742 1685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aae7b5a01a0543afb1b5a7f44a9a864c" candidate_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16b3a337bc5e462f83ba895ad8660deb"
I20240502 21:28:56.739912 1685 raft_consensus.cc:3037] T aae7b5a01a0543afb1b5a7f44a9a864c P 16b3a337bc5e462f83ba895ad8660deb [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:56.740628 1550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aae7b5a01a0543afb1b5a7f44a9a864c" candidate_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a62a935ff43d494ca35d0c417943133d"
I20240502 21:28:56.740839 1550 raft_consensus.cc:3037] T aae7b5a01a0543afb1b5a7f44a9a864c P a62a935ff43d494ca35d0c417943133d [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:56.742480 1685 raft_consensus.cc:2454] T aae7b5a01a0543afb1b5a7f44a9a864c P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d1e5c1b047443cbb28d7b0a9744babf in term 1.
I20240502 21:28:56.743621 1550 raft_consensus.cc:2454] T aae7b5a01a0543afb1b5a7f44a9a864c P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d1e5c1b047443cbb28d7b0a9744babf in term 1.
I20240502 21:28:56.744519 1549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aae7b5a01a0543afb1b5a7f44a9a864c" candidate_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a62a935ff43d494ca35d0c417943133d" is_pre_election: true
I20240502 21:28:56.744746 1549 raft_consensus.cc:2362] T aae7b5a01a0543afb1b5a7f44a9a864c P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 7d1e5c1b047443cbb28d7b0a9744babf in term 1. Re-sending same reply.
I20240502 21:28:56.750152 3122 leader_election.cc:288] T aae7b5a01a0543afb1b5a7f44a9a864c P 7d1e5c1b047443cbb28d7b0a9744babf [CANDIDATE]: Term 1 election: Requested vote from peers a62a935ff43d494ca35d0c417943133d (127.15.255.130:37991), 16b3a337bc5e462f83ba895ad8660deb (127.15.255.131:36033), 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132:44947), e632c553ff4341328acf1cbaca678392 (127.15.255.133:37439), 521bb93035324cecbebed2f6f2a630f8 (127.15.255.134:40821), 6ab8df3c796f4b6cb3e29d5045ad3ea0 (127.15.255.135:34701)
I20240502 21:28:56.750669 3293 leader_election.cc:288] T 05e885ae741c43808c6dfbcdc5769942 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [CANDIDATE]: Term 1 election: Requested vote from peers 7d1e5c1b047443cbb28d7b0a9744babf (127.15.255.129:37897), a62a935ff43d494ca35d0c417943133d (127.15.255.130:37991), 16b3a337bc5e462f83ba895ad8660deb (127.15.255.131:36033), 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132:44947), e632c553ff4341328acf1cbaca678392 (127.15.255.133:37439), 521bb93035324cecbebed2f6f2a630f8 (127.15.255.134:40821)
I20240502 21:28:56.750861 1818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aae7b5a01a0543afb1b5a7f44a9a864c" candidate_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "153026dd7f8943cc93fedf6f48450fb4"
I20240502 21:28:56.751083 1818 raft_consensus.cc:3037] T aae7b5a01a0543afb1b5a7f44a9a864c P 153026dd7f8943cc93fedf6f48450fb4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:56.751915 3116 leader_election.cc:288] T c46e36b0bceb45b4bafbf1ffaf8e039e P 521bb93035324cecbebed2f6f2a630f8 [CANDIDATE]: Term 1 election: Requested vote from peers 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132:44947), e632c553ff4341328acf1cbaca678392 (127.15.255.133:37439), 6ab8df3c796f4b6cb3e29d5045ad3ea0 (127.15.255.135:34701), a62a935ff43d494ca35d0c417943133d (127.15.255.130:37991), 7d1e5c1b047443cbb28d7b0a9744babf (127.15.255.129:37897), 16b3a337bc5e462f83ba895ad8660deb (127.15.255.131:36033)
I20240502 21:28:56.752815 1953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05e885ae741c43808c6dfbcdc5769942" candidate_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e632c553ff4341328acf1cbaca678392"
I20240502 21:28:56.753042 1953 raft_consensus.cc:3037] T 05e885ae741c43808c6dfbcdc5769942 P e632c553ff4341328acf1cbaca678392 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:56.753199 1685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05e885ae741c43808c6dfbcdc5769942" candidate_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16b3a337bc5e462f83ba895ad8660deb"
I20240502 21:28:56.753300 1819 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c46e36b0bceb45b4bafbf1ffaf8e039e" candidate_uuid: "521bb93035324cecbebed2f6f2a630f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "153026dd7f8943cc93fedf6f48450fb4"
I20240502 21:28:56.753401 1685 raft_consensus.cc:3037] T 05e885ae741c43808c6dfbcdc5769942 P 16b3a337bc5e462f83ba895ad8660deb [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:56.753494 1819 raft_consensus.cc:3037] T c46e36b0bceb45b4bafbf1ffaf8e039e P 153026dd7f8943cc93fedf6f48450fb4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:56.754066 3366 tablet_bootstrap.cc:492] T aae7b5a01a0543afb1b5a7f44a9a864c P 521bb93035324cecbebed2f6f2a630f8: No bootstrap required, opened a new log
I20240502 21:28:56.754230 3366 ts_tablet_manager.cc:1367] T aae7b5a01a0543afb1b5a7f44a9a864c P 521bb93035324cecbebed2f6f2a630f8: Time spent bootstrapping tablet: real 0.056s user 0.007s sys 0.038s
I20240502 21:28:56.754925 3366 raft_consensus.cc:348] T aae7b5a01a0543afb1b5a7f44a9a864c P 521bb93035324cecbebed2f6f2a630f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } }
I20240502 21:28:56.755192 3366 raft_consensus.cc:374] T aae7b5a01a0543afb1b5a7f44a9a864c P 521bb93035324cecbebed2f6f2a630f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:56.755263 3366 raft_consensus.cc:729] T aae7b5a01a0543afb1b5a7f44a9a864c P 521bb93035324cecbebed2f6f2a630f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 521bb93035324cecbebed2f6f2a630f8, State: Initialized, Role: FOLLOWER
I20240502 21:28:56.755472 3366 consensus_queue.cc:260] T aae7b5a01a0543afb1b5a7f44a9a864c P 521bb93035324cecbebed2f6f2a630f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } }
I20240502 21:28:56.755640 1953 raft_consensus.cc:2454] T 05e885ae741c43808c6dfbcdc5769942 P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ab8df3c796f4b6cb3e29d5045ad3ea0 in term 1.
I20240502 21:28:56.755839 1685 raft_consensus.cc:2454] T 05e885ae741c43808c6dfbcdc5769942 P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ab8df3c796f4b6cb3e29d5045ad3ea0 in term 1.
I20240502 21:28:56.756001 1819 raft_consensus.cc:2454] T c46e36b0bceb45b4bafbf1ffaf8e039e P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 521bb93035324cecbebed2f6f2a630f8 in term 1.
I20240502 21:28:56.756171 3366 ts_tablet_manager.cc:1398] T aae7b5a01a0543afb1b5a7f44a9a864c P 521bb93035324cecbebed2f6f2a630f8: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 21:28:56.756522 3366 tablet_bootstrap.cc:492] T 05e885ae741c43808c6dfbcdc5769942 P 521bb93035324cecbebed2f6f2a630f8: Bootstrap starting.
I20240502 21:28:56.756484 2225 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aae7b5a01a0543afb1b5a7f44a9a864c" candidate_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0"
I20240502 21:28:56.756661 2225 raft_consensus.cc:3037] T aae7b5a01a0543afb1b5a7f44a9a864c P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:56.756603 1955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c46e36b0bceb45b4bafbf1ffaf8e039e" candidate_uuid: "521bb93035324cecbebed2f6f2a630f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e632c553ff4341328acf1cbaca678392"
I20240502 21:28:56.756785 1955 raft_consensus.cc:3037] T c46e36b0bceb45b4bafbf1ffaf8e039e P e632c553ff4341328acf1cbaca678392 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:56.758003 3366 tablet_bootstrap.cc:654] T 05e885ae741c43808c6dfbcdc5769942 P 521bb93035324cecbebed2f6f2a630f8: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:56.758508 1955 raft_consensus.cc:2454] T c46e36b0bceb45b4bafbf1ffaf8e039e P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 521bb93035324cecbebed2f6f2a630f8 in term 1.
I20240502 21:28:56.759008 2227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aae7b5a01a0543afb1b5a7f44a9a864c" candidate_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" is_pre_election: true
I20240502 21:28:56.759315 2225 raft_consensus.cc:2454] T aae7b5a01a0543afb1b5a7f44a9a864c P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d1e5c1b047443cbb28d7b0a9744babf in term 1.
I20240502 21:28:56.760119 1348 leader_election.cc:302] T aae7b5a01a0543afb1b5a7f44a9a864c P 7d1e5c1b047443cbb28d7b0a9744babf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 16b3a337bc5e462f83ba895ad8660deb, 6ab8df3c796f4b6cb3e29d5045ad3ea0, 7d1e5c1b047443cbb28d7b0a9744babf, a62a935ff43d494ca35d0c417943133d; no voters:
I20240502 21:28:56.760408 3122 raft_consensus.cc:2781] T aae7b5a01a0543afb1b5a7f44a9a864c P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:28:56.760658 3122 raft_consensus.cc:686] T aae7b5a01a0543afb1b5a7f44a9a864c P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 LEADER]: Becoming Leader. State: Replica: 7d1e5c1b047443cbb28d7b0a9744babf, State: Running, Role: LEADER
I20240502 21:28:56.760995 3122 consensus_queue.cc:237] T aae7b5a01a0543afb1b5a7f44a9a864c P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } }
I20240502 21:28:56.761637 2088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aae7b5a01a0543afb1b5a7f44a9a864c" candidate_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "521bb93035324cecbebed2f6f2a630f8"
I20240502 21:28:56.761824 2088 raft_consensus.cc:3037] T aae7b5a01a0543afb1b5a7f44a9a864c P 521bb93035324cecbebed2f6f2a630f8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:56.763119 2089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05e885ae741c43808c6dfbcdc5769942" candidate_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "521bb93035324cecbebed2f6f2a630f8"
I20240502 21:28:56.763803 1414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c46e36b0bceb45b4bafbf1ffaf8e039e" candidate_uuid: "521bb93035324cecbebed2f6f2a630f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d1e5c1b047443cbb28d7b0a9744babf"
I20240502 21:28:56.764003 1414 raft_consensus.cc:3037] T c46e36b0bceb45b4bafbf1ffaf8e039e P 7d1e5c1b047443cbb28d7b0a9744babf [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:56.764626 1954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aae7b5a01a0543afb1b5a7f44a9a864c" candidate_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e632c553ff4341328acf1cbaca678392"
I20240502 21:28:56.764809 1954 raft_consensus.cc:3037] T aae7b5a01a0543afb1b5a7f44a9a864c P e632c553ff4341328acf1cbaca678392 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:56.765133 1549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c46e36b0bceb45b4bafbf1ffaf8e039e" candidate_uuid: "521bb93035324cecbebed2f6f2a630f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a62a935ff43d494ca35d0c417943133d"
I20240502 21:28:56.765331 1549 raft_consensus.cc:3037] T c46e36b0bceb45b4bafbf1ffaf8e039e P a62a935ff43d494ca35d0c417943133d [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:56.766619 1414 raft_consensus.cc:2454] T c46e36b0bceb45b4bafbf1ffaf8e039e P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 521bb93035324cecbebed2f6f2a630f8 in term 1.
I20240502 21:28:56.767012 2023 leader_election.cc:302] T c46e36b0bceb45b4bafbf1ffaf8e039e P 521bb93035324cecbebed2f6f2a630f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 153026dd7f8943cc93fedf6f48450fb4, 521bb93035324cecbebed2f6f2a630f8, 7d1e5c1b047443cbb28d7b0a9744babf, e632c553ff4341328acf1cbaca678392; no voters:
I20240502 21:28:56.767323 3116 raft_consensus.cc:2781] T c46e36b0bceb45b4bafbf1ffaf8e039e P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:28:56.767452 1954 raft_consensus.cc:2454] T aae7b5a01a0543afb1b5a7f44a9a864c P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d1e5c1b047443cbb28d7b0a9744babf in term 1.
I20240502 21:28:56.767489 3116 raft_consensus.cc:686] T c46e36b0bceb45b4bafbf1ffaf8e039e P 521bb93035324cecbebed2f6f2a630f8 [term 1 LEADER]: Becoming Leader. State: Replica: 521bb93035324cecbebed2f6f2a630f8, State: Running, Role: LEADER
W20240502 21:28:56.767740 2158 leader_election.cc:341] T 05e885ae741c43808c6dfbcdc5769942 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 521bb93035324cecbebed2f6f2a630f8 (127.15.255.134:40821): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:28:56.767807 3116 consensus_queue.cc:237] T c46e36b0bceb45b4bafbf1ffaf8e039e P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:28:56.769253 1818 raft_consensus.cc:2454] T aae7b5a01a0543afb1b5a7f44a9a864c P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d1e5c1b047443cbb28d7b0a9744babf in term 1.
I20240502 21:28:56.769865 2088 raft_consensus.cc:2454] T aae7b5a01a0543afb1b5a7f44a9a864c P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d1e5c1b047443cbb28d7b0a9744babf in term 1.
I20240502 21:28:56.770251 2225 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c46e36b0bceb45b4bafbf1ffaf8e039e" candidate_uuid: "521bb93035324cecbebed2f6f2a630f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0"
I20240502 21:28:56.770458 2225 raft_consensus.cc:3037] T c46e36b0bceb45b4bafbf1ffaf8e039e P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:56.773221 2225 raft_consensus.cc:2454] T c46e36b0bceb45b4bafbf1ffaf8e039e P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 521bb93035324cecbebed2f6f2a630f8 in term 1.
I20240502 21:28:56.773610 1685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c46e36b0bceb45b4bafbf1ffaf8e039e" candidate_uuid: "521bb93035324cecbebed2f6f2a630f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16b3a337bc5e462f83ba895ad8660deb"
I20240502 21:28:56.773816 1685 raft_consensus.cc:3037] T c46e36b0bceb45b4bafbf1ffaf8e039e P 16b3a337bc5e462f83ba895ad8660deb [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:56.775722 1549 raft_consensus.cc:2454] T c46e36b0bceb45b4bafbf1ffaf8e039e P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 521bb93035324cecbebed2f6f2a630f8 in term 1.
I20240502 21:28:56.776648 1685 raft_consensus.cc:2454] T c46e36b0bceb45b4bafbf1ffaf8e039e P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 521bb93035324cecbebed2f6f2a630f8 in term 1.
I20240502 21:28:56.777997 3366 tablet_bootstrap.cc:492] T 05e885ae741c43808c6dfbcdc5769942 P 521bb93035324cecbebed2f6f2a630f8: No bootstrap required, opened a new log
I20240502 21:28:56.778169 3366 ts_tablet_manager.cc:1367] T 05e885ae741c43808c6dfbcdc5769942 P 521bb93035324cecbebed2f6f2a630f8: Time spent bootstrapping tablet: real 0.022s user 0.012s sys 0.007s
I20240502 21:28:56.778896 3366 raft_consensus.cc:348] T 05e885ae741c43808c6dfbcdc5769942 P 521bb93035324cecbebed2f6f2a630f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } }
I20240502 21:28:56.779100 3366 raft_consensus.cc:374] T 05e885ae741c43808c6dfbcdc5769942 P 521bb93035324cecbebed2f6f2a630f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:56.779189 3366 raft_consensus.cc:729] T 05e885ae741c43808c6dfbcdc5769942 P 521bb93035324cecbebed2f6f2a630f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 521bb93035324cecbebed2f6f2a630f8, State: Initialized, Role: FOLLOWER
I20240502 21:28:56.779111 1266 catalog_manager.cc:5484] T c46e36b0bceb45b4bafbf1ffaf8e039e P 521bb93035324cecbebed2f6f2a630f8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 521bb93035324cecbebed2f6f2a630f8 (127.15.255.134). New cstate: current_term: 1 leader_uuid: "521bb93035324cecbebed2f6f2a630f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } health_report { overall_health: UNKNOWN } } }
I20240502 21:28:56.779405 3366 consensus_queue.cc:260] T 05e885ae741c43808c6dfbcdc5769942 P 521bb93035324cecbebed2f6f2a630f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } }
I20240502 21:28:56.780041 3366 ts_tablet_manager.cc:1398] T 05e885ae741c43808c6dfbcdc5769942 P 521bb93035324cecbebed2f6f2a630f8: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 21:28:56.780223 1264 catalog_manager.cc:5484] T aae7b5a01a0543afb1b5a7f44a9a864c P 7d1e5c1b047443cbb28d7b0a9744babf reported cstate change: term changed from 0 to 1, leader changed from <none> to 7d1e5c1b047443cbb28d7b0a9744babf (127.15.255.129). New cstate: current_term: 1 leader_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } health_report { overall_health: UNKNOWN } } }
W20240502 21:28:57.135715 3400 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:28:57.136027 3400 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:28:57.144238 3400 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 21:28:57.155236 3394 consensus_queue.cc:1035] T 05e885ae741c43808c6dfbcdc5769942 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:28:57.158358 2088 raft_consensus.cc:3037] T 05e885ae741c43808c6dfbcdc5769942 P 521bb93035324cecbebed2f6f2a630f8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:57.161762 3394 consensus_queue.cc:1035] T 05e885ae741c43808c6dfbcdc5769942 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:28:57.163420 3394 consensus_queue.cc:1035] T 05e885ae741c43808c6dfbcdc5769942 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:28:57.165737 3122 consensus_queue.cc:1035] T aae7b5a01a0543afb1b5a7f44a9a864c P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:28:57.171730 3116 consensus_queue.cc:1035] T c46e36b0bceb45b4bafbf1ffaf8e039e P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:28:57.178508 3356 consensus_queue.cc:1035] T c46e36b0bceb45b4bafbf1ffaf8e039e P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:28:57.181730 3122 consensus_queue.cc:1035] T aae7b5a01a0543afb1b5a7f44a9a864c P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:28:57.199134 3293 consensus_queue.cc:1035] T 05e885ae741c43808c6dfbcdc5769942 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:28:57.199390 3394 consensus_queue.cc:1035] T 05e885ae741c43808c6dfbcdc5769942 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:28:57.199635 3395 consensus_queue.cc:1035] T 05e885ae741c43808c6dfbcdc5769942 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_4'
I20240502 21:28:57.233713 3116 consensus_queue.cc:1035] T c46e36b0bceb45b4bafbf1ffaf8e039e P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:28:57.240693 3122 consensus_queue.cc:1035] T aae7b5a01a0543afb1b5a7f44a9a864c P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:28:57.275355 3276 consensus_queue.cc:1035] T aae7b5a01a0543afb1b5a7f44a9a864c P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:28:57.280591 3356 consensus_queue.cc:1035] T c46e36b0bceb45b4bafbf1ffaf8e039e P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:28:57.281690 3356 consensus_queue.cc:1035] T c46e36b0bceb45b4bafbf1ffaf8e039e P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:28:57.288223 3116 consensus_queue.cc:1035] T c46e36b0bceb45b4bafbf1ffaf8e039e P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:28:57.299458 3276 consensus_queue.cc:1035] T aae7b5a01a0543afb1b5a7f44a9a864c P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:28:57.312096 3276 consensus_queue.cc:1035] T aae7b5a01a0543afb1b5a7f44a9a864c P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 477.96 ms
time per row: 0.23898 ms
I20240502 21:28:57.942188 1263 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44934:
name: "rebalance_test_table_5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 7
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
W20240502 21:28:57.942993 1263 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table rebalance_test_table_5 in case of a server failure: 8 tablet servers would be needed, 7 are available. Consider bringing up more tablet servers.
I20240502 21:28:57.952983 1931 tablet_service.cc:1467] Processing CreateTablet for tablet e45af71e3fc14c579b32ba4dbe2adf78 (DEFAULT_TABLE table=rebalance_test_table_5 [id=9cb803e5bb2545c8b992c90a894085af]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:57.953220 2067 tablet_service.cc:1467] Processing CreateTablet for tablet e45af71e3fc14c579b32ba4dbe2adf78 (DEFAULT_TABLE table=rebalance_test_table_5 [id=9cb803e5bb2545c8b992c90a894085af]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:57.953461 1931 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e45af71e3fc14c579b32ba4dbe2adf78. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:57.953641 2067 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e45af71e3fc14c579b32ba4dbe2adf78. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:57.953548 1800 tablet_service.cc:1467] Processing CreateTablet for tablet e45af71e3fc14c579b32ba4dbe2adf78 (DEFAULT_TABLE table=rebalance_test_table_5 [id=9cb803e5bb2545c8b992c90a894085af]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:57.954013 1800 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e45af71e3fc14c579b32ba4dbe2adf78. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:57.956931 2206 tablet_service.cc:1467] Processing CreateTablet for tablet e45af71e3fc14c579b32ba4dbe2adf78 (DEFAULT_TABLE table=rebalance_test_table_5 [id=9cb803e5bb2545c8b992c90a894085af]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:57.957394 2206 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e45af71e3fc14c579b32ba4dbe2adf78. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:57.959476 2207 tablet_service.cc:1467] Processing CreateTablet for tablet 72355dd734ea4aeba031171e8c39696d (DEFAULT_TABLE table=rebalance_test_table_5 [id=9cb803e5bb2545c8b992c90a894085af]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:57.959827 2207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72355dd734ea4aeba031171e8c39696d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:57.960178 1392 tablet_service.cc:1467] Processing CreateTablet for tablet 72355dd734ea4aeba031171e8c39696d (DEFAULT_TABLE table=rebalance_test_table_5 [id=9cb803e5bb2545c8b992c90a894085af]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:57.960701 1392 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72355dd734ea4aeba031171e8c39696d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:57.960834 1393 tablet_service.cc:1467] Processing CreateTablet for tablet e45af71e3fc14c579b32ba4dbe2adf78 (DEFAULT_TABLE table=rebalance_test_table_5 [id=9cb803e5bb2545c8b992c90a894085af]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:57.961216 1393 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e45af71e3fc14c579b32ba4dbe2adf78. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:57.964799 1394 tablet_service.cc:1467] Processing CreateTablet for tablet 4dad8773f5b54efa9ff527252bac24aa (DEFAULT_TABLE table=rebalance_test_table_5 [id=9cb803e5bb2545c8b992c90a894085af]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:57.965209 1394 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4dad8773f5b54efa9ff527252bac24aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:57.969105 3506 tablet_bootstrap.cc:492] T 72355dd734ea4aeba031171e8c39696d P 6ab8df3c796f4b6cb3e29d5045ad3ea0: Bootstrap starting.
I20240502 21:28:57.969235 1527 tablet_service.cc:1467] Processing CreateTablet for tablet e45af71e3fc14c579b32ba4dbe2adf78 (DEFAULT_TABLE table=rebalance_test_table_5 [id=9cb803e5bb2545c8b992c90a894085af]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:57.969709 1527 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e45af71e3fc14c579b32ba4dbe2adf78. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:57.971450 3506 tablet_bootstrap.cc:654] T 72355dd734ea4aeba031171e8c39696d P 6ab8df3c796f4b6cb3e29d5045ad3ea0: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:57.972849 1526 tablet_service.cc:1467] Processing CreateTablet for tablet 4dad8773f5b54efa9ff527252bac24aa (DEFAULT_TABLE table=rebalance_test_table_5 [id=9cb803e5bb2545c8b992c90a894085af]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:57.973273 1526 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4dad8773f5b54efa9ff527252bac24aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:57.977399 3507 tablet_bootstrap.cc:492] T e45af71e3fc14c579b32ba4dbe2adf78 P 521bb93035324cecbebed2f6f2a630f8: Bootstrap starting.
I20240502 21:28:57.979274 3507 tablet_bootstrap.cc:654] T e45af71e3fc14c579b32ba4dbe2adf78 P 521bb93035324cecbebed2f6f2a630f8: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:57.980314 3506 tablet_bootstrap.cc:492] T 72355dd734ea4aeba031171e8c39696d P 6ab8df3c796f4b6cb3e29d5045ad3ea0: No bootstrap required, opened a new log
I20240502 21:28:57.980473 3506 ts_tablet_manager.cc:1367] T 72355dd734ea4aeba031171e8c39696d P 6ab8df3c796f4b6cb3e29d5045ad3ea0: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.003s
I20240502 21:28:57.980983 2205 tablet_service.cc:1467] Processing CreateTablet for tablet 4dad8773f5b54efa9ff527252bac24aa (DEFAULT_TABLE table=rebalance_test_table_5 [id=9cb803e5bb2545c8b992c90a894085af]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:57.981443 2205 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4dad8773f5b54efa9ff527252bac24aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:57.981514 3506 raft_consensus.cc:348] T 72355dd734ea4aeba031171e8c39696d P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:28:57.981739 3506 raft_consensus.cc:374] T 72355dd734ea4aeba031171e8c39696d P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:57.981842 3506 raft_consensus.cc:729] T 72355dd734ea4aeba031171e8c39696d P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ab8df3c796f4b6cb3e29d5045ad3ea0, State: Initialized, Role: FOLLOWER
I20240502 21:28:57.983937 3506 consensus_queue.cc:260] T 72355dd734ea4aeba031171e8c39696d P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:28:57.985070 3507 tablet_bootstrap.cc:492] T e45af71e3fc14c579b32ba4dbe2adf78 P 521bb93035324cecbebed2f6f2a630f8: No bootstrap required, opened a new log
I20240502 21:28:57.985113 3506 ts_tablet_manager.cc:1398] T 72355dd734ea4aeba031171e8c39696d P 6ab8df3c796f4b6cb3e29d5045ad3ea0: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240502 21:28:57.985292 3507 ts_tablet_manager.cc:1367] T e45af71e3fc14c579b32ba4dbe2adf78 P 521bb93035324cecbebed2f6f2a630f8: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.004s
I20240502 21:28:57.985296 1798 tablet_service.cc:1467] Processing CreateTablet for tablet 72355dd734ea4aeba031171e8c39696d (DEFAULT_TABLE table=rebalance_test_table_5 [id=9cb803e5bb2545c8b992c90a894085af]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:57.985450 1799 tablet_service.cc:1467] Processing CreateTablet for tablet 4dad8773f5b54efa9ff527252bac24aa (DEFAULT_TABLE table=rebalance_test_table_5 [id=9cb803e5bb2545c8b992c90a894085af]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:57.985731 1798 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72355dd734ea4aeba031171e8c39696d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:57.985987 1799 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4dad8773f5b54efa9ff527252bac24aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:57.987007 3507 raft_consensus.cc:348] T e45af71e3fc14c579b32ba4dbe2adf78 P 521bb93035324cecbebed2f6f2a630f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:28:57.987864 3507 raft_consensus.cc:374] T e45af71e3fc14c579b32ba4dbe2adf78 P 521bb93035324cecbebed2f6f2a630f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:57.988127 3507 raft_consensus.cc:729] T e45af71e3fc14c579b32ba4dbe2adf78 P 521bb93035324cecbebed2f6f2a630f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 521bb93035324cecbebed2f6f2a630f8, State: Initialized, Role: FOLLOWER
I20240502 21:28:57.988584 3507 consensus_queue.cc:260] T e45af71e3fc14c579b32ba4dbe2adf78 P 521bb93035324cecbebed2f6f2a630f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:28:57.990027 3507 ts_tablet_manager.cc:1398] T e45af71e3fc14c579b32ba4dbe2adf78 P 521bb93035324cecbebed2f6f2a630f8: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240502 21:28:57.993142 2067 tablet_service.cc:1467] Processing CreateTablet for tablet 4dad8773f5b54efa9ff527252bac24aa (DEFAULT_TABLE table=rebalance_test_table_5 [id=9cb803e5bb2545c8b992c90a894085af]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:57.993531 2067 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4dad8773f5b54efa9ff527252bac24aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:57.993621 1933 tablet_service.cc:1467] Processing CreateTablet for tablet 4dad8773f5b54efa9ff527252bac24aa (DEFAULT_TABLE table=rebalance_test_table_5 [id=9cb803e5bb2545c8b992c90a894085af]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:57.993834 1528 tablet_service.cc:1467] Processing CreateTablet for tablet 72355dd734ea4aeba031171e8c39696d (DEFAULT_TABLE table=rebalance_test_table_5 [id=9cb803e5bb2545c8b992c90a894085af]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:57.993980 1933 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4dad8773f5b54efa9ff527252bac24aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:57.994275 1528 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72355dd734ea4aeba031171e8c39696d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:57.997718 1932 tablet_service.cc:1467] Processing CreateTablet for tablet 72355dd734ea4aeba031171e8c39696d (DEFAULT_TABLE table=rebalance_test_table_5 [id=9cb803e5bb2545c8b992c90a894085af]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:57.998056 1932 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72355dd734ea4aeba031171e8c39696d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:57.998575 1663 tablet_service.cc:1467] Processing CreateTablet for tablet 4dad8773f5b54efa9ff527252bac24aa (DEFAULT_TABLE table=rebalance_test_table_5 [id=9cb803e5bb2545c8b992c90a894085af]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:57.999076 1663 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4dad8773f5b54efa9ff527252bac24aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:58.000784 2068 tablet_service.cc:1467] Processing CreateTablet for tablet 72355dd734ea4aeba031171e8c39696d (DEFAULT_TABLE table=rebalance_test_table_5 [id=9cb803e5bb2545c8b992c90a894085af]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:58.000816 1664 tablet_service.cc:1467] Processing CreateTablet for tablet e45af71e3fc14c579b32ba4dbe2adf78 (DEFAULT_TABLE table=rebalance_test_table_5 [id=9cb803e5bb2545c8b992c90a894085af]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:58.000846 1662 tablet_service.cc:1467] Processing CreateTablet for tablet 72355dd734ea4aeba031171e8c39696d (DEFAULT_TABLE table=rebalance_test_table_5 [id=9cb803e5bb2545c8b992c90a894085af]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:58.001140 2068 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72355dd734ea4aeba031171e8c39696d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:58.001228 1664 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e45af71e3fc14c579b32ba4dbe2adf78. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:58.001781 1662 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72355dd734ea4aeba031171e8c39696d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:58.014870 3507 tablet_bootstrap.cc:492] T 4dad8773f5b54efa9ff527252bac24aa P 521bb93035324cecbebed2f6f2a630f8: Bootstrap starting.
I20240502 21:28:58.016363 3507 tablet_bootstrap.cc:654] T 4dad8773f5b54efa9ff527252bac24aa P 521bb93035324cecbebed2f6f2a630f8: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:58.021049 3508 tablet_bootstrap.cc:492] T 72355dd734ea4aeba031171e8c39696d P 7d1e5c1b047443cbb28d7b0a9744babf: Bootstrap starting.
I20240502 21:28:58.022156 3505 tablet_bootstrap.cc:492] T e45af71e3fc14c579b32ba4dbe2adf78 P e632c553ff4341328acf1cbaca678392: Bootstrap starting.
I20240502 21:28:58.023531 3508 tablet_bootstrap.cc:654] T 72355dd734ea4aeba031171e8c39696d P 7d1e5c1b047443cbb28d7b0a9744babf: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:58.025655 3505 tablet_bootstrap.cc:654] T e45af71e3fc14c579b32ba4dbe2adf78 P e632c553ff4341328acf1cbaca678392: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:58.025733 3506 tablet_bootstrap.cc:492] T e45af71e3fc14c579b32ba4dbe2adf78 P 6ab8df3c796f4b6cb3e29d5045ad3ea0: Bootstrap starting.
I20240502 21:28:58.030592 3506 tablet_bootstrap.cc:654] T e45af71e3fc14c579b32ba4dbe2adf78 P 6ab8df3c796f4b6cb3e29d5045ad3ea0: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:58.035912 3507 tablet_bootstrap.cc:492] T 4dad8773f5b54efa9ff527252bac24aa P 521bb93035324cecbebed2f6f2a630f8: No bootstrap required, opened a new log
I20240502 21:28:58.036098 3507 ts_tablet_manager.cc:1367] T 4dad8773f5b54efa9ff527252bac24aa P 521bb93035324cecbebed2f6f2a630f8: Time spent bootstrapping tablet: real 0.021s user 0.009s sys 0.000s
I20240502 21:28:58.036800 3507 raft_consensus.cc:348] T 4dad8773f5b54efa9ff527252bac24aa P 521bb93035324cecbebed2f6f2a630f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:28:58.037111 3507 raft_consensus.cc:374] T 4dad8773f5b54efa9ff527252bac24aa P 521bb93035324cecbebed2f6f2a630f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:58.037187 3507 raft_consensus.cc:729] T 4dad8773f5b54efa9ff527252bac24aa P 521bb93035324cecbebed2f6f2a630f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 521bb93035324cecbebed2f6f2a630f8, State: Initialized, Role: FOLLOWER
I20240502 21:28:58.037448 3507 consensus_queue.cc:260] T 4dad8773f5b54efa9ff527252bac24aa P 521bb93035324cecbebed2f6f2a630f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:28:58.038103 3507 ts_tablet_manager.cc:1398] T 4dad8773f5b54efa9ff527252bac24aa P 521bb93035324cecbebed2f6f2a630f8: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:28:58.038434 3507 tablet_bootstrap.cc:492] T 72355dd734ea4aeba031171e8c39696d P 521bb93035324cecbebed2f6f2a630f8: Bootstrap starting.
I20240502 21:28:58.039963 3507 tablet_bootstrap.cc:654] T 72355dd734ea4aeba031171e8c39696d P 521bb93035324cecbebed2f6f2a630f8: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:58.049012 3512 tablet_bootstrap.cc:492] T e45af71e3fc14c579b32ba4dbe2adf78 P a62a935ff43d494ca35d0c417943133d: Bootstrap starting.
I20240502 21:28:58.049043 3506 tablet_bootstrap.cc:492] T e45af71e3fc14c579b32ba4dbe2adf78 P 6ab8df3c796f4b6cb3e29d5045ad3ea0: No bootstrap required, opened a new log
I20240502 21:28:58.049325 3506 ts_tablet_manager.cc:1367] T e45af71e3fc14c579b32ba4dbe2adf78 P 6ab8df3c796f4b6cb3e29d5045ad3ea0: Time spent bootstrapping tablet: real 0.024s user 0.001s sys 0.003s
I20240502 21:28:58.050123 3506 raft_consensus.cc:348] T e45af71e3fc14c579b32ba4dbe2adf78 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:28:58.050468 3506 raft_consensus.cc:374] T e45af71e3fc14c579b32ba4dbe2adf78 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:58.050616 3506 raft_consensus.cc:729] T e45af71e3fc14c579b32ba4dbe2adf78 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ab8df3c796f4b6cb3e29d5045ad3ea0, State: Initialized, Role: FOLLOWER
I20240502 21:28:58.050957 3506 consensus_queue.cc:260] T e45af71e3fc14c579b32ba4dbe2adf78 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:28:58.051790 3506 ts_tablet_manager.cc:1398] T e45af71e3fc14c579b32ba4dbe2adf78 P 6ab8df3c796f4b6cb3e29d5045ad3ea0: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 21:28:58.052019 3507 tablet_bootstrap.cc:492] T 72355dd734ea4aeba031171e8c39696d P 521bb93035324cecbebed2f6f2a630f8: No bootstrap required, opened a new log
I20240502 21:28:58.052165 3507 ts_tablet_manager.cc:1367] T 72355dd734ea4aeba031171e8c39696d P 521bb93035324cecbebed2f6f2a630f8: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.003s
I20240502 21:28:58.052294 3506 tablet_bootstrap.cc:492] T 4dad8773f5b54efa9ff527252bac24aa P 6ab8df3c796f4b6cb3e29d5045ad3ea0: Bootstrap starting.
I20240502 21:28:58.053599 3507 raft_consensus.cc:348] T 72355dd734ea4aeba031171e8c39696d P 521bb93035324cecbebed2f6f2a630f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:28:58.053869 3507 raft_consensus.cc:374] T 72355dd734ea4aeba031171e8c39696d P 521bb93035324cecbebed2f6f2a630f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:58.053954 3511 tablet_bootstrap.cc:492] T e45af71e3fc14c579b32ba4dbe2adf78 P 153026dd7f8943cc93fedf6f48450fb4: Bootstrap starting.
I20240502 21:28:58.053997 3506 tablet_bootstrap.cc:654] T 4dad8773f5b54efa9ff527252bac24aa P 6ab8df3c796f4b6cb3e29d5045ad3ea0: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:58.053972 3507 raft_consensus.cc:729] T 72355dd734ea4aeba031171e8c39696d P 521bb93035324cecbebed2f6f2a630f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 521bb93035324cecbebed2f6f2a630f8, State: Initialized, Role: FOLLOWER
I20240502 21:28:58.055410 3507 consensus_queue.cc:260] T 72355dd734ea4aeba031171e8c39696d P 521bb93035324cecbebed2f6f2a630f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:28:58.057153 3507 ts_tablet_manager.cc:1398] T 72355dd734ea4aeba031171e8c39696d P 521bb93035324cecbebed2f6f2a630f8: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240502 21:28:58.065234 3512 tablet_bootstrap.cc:654] T e45af71e3fc14c579b32ba4dbe2adf78 P a62a935ff43d494ca35d0c417943133d: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:58.066273 3511 tablet_bootstrap.cc:654] T e45af71e3fc14c579b32ba4dbe2adf78 P 153026dd7f8943cc93fedf6f48450fb4: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:58.066475 3508 tablet_bootstrap.cc:492] T 72355dd734ea4aeba031171e8c39696d P 7d1e5c1b047443cbb28d7b0a9744babf: No bootstrap required, opened a new log
I20240502 21:28:58.066648 3508 ts_tablet_manager.cc:1367] T 72355dd734ea4aeba031171e8c39696d P 7d1e5c1b047443cbb28d7b0a9744babf: Time spent bootstrapping tablet: real 0.046s user 0.007s sys 0.009s
I20240502 21:28:58.067502 3508 raft_consensus.cc:348] T 72355dd734ea4aeba031171e8c39696d P 7d1e5c1b047443cbb28d7b0a9744babf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:28:58.067821 3508 raft_consensus.cc:374] T 72355dd734ea4aeba031171e8c39696d P 7d1e5c1b047443cbb28d7b0a9744babf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:58.067929 3508 raft_consensus.cc:729] T 72355dd734ea4aeba031171e8c39696d P 7d1e5c1b047443cbb28d7b0a9744babf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d1e5c1b047443cbb28d7b0a9744babf, State: Initialized, Role: FOLLOWER
I20240502 21:28:58.068234 3508 consensus_queue.cc:260] T 72355dd734ea4aeba031171e8c39696d P 7d1e5c1b047443cbb28d7b0a9744babf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:28:58.069082 3508 ts_tablet_manager.cc:1398] T 72355dd734ea4aeba031171e8c39696d P 7d1e5c1b047443cbb28d7b0a9744babf: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240502 21:28:58.069697 3508 tablet_bootstrap.cc:492] T 4dad8773f5b54efa9ff527252bac24aa P 7d1e5c1b047443cbb28d7b0a9744babf: Bootstrap starting.
I20240502 21:28:58.070598 3505 tablet_bootstrap.cc:492] T e45af71e3fc14c579b32ba4dbe2adf78 P e632c553ff4341328acf1cbaca678392: No bootstrap required, opened a new log
I20240502 21:28:58.070827 3505 ts_tablet_manager.cc:1367] T e45af71e3fc14c579b32ba4dbe2adf78 P e632c553ff4341328acf1cbaca678392: Time spent bootstrapping tablet: real 0.049s user 0.011s sys 0.005s
I20240502 21:28:58.071426 3508 tablet_bootstrap.cc:654] T 4dad8773f5b54efa9ff527252bac24aa P 7d1e5c1b047443cbb28d7b0a9744babf: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:58.072604 3505 raft_consensus.cc:348] T e45af71e3fc14c579b32ba4dbe2adf78 P e632c553ff4341328acf1cbaca678392 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:28:58.073101 3505 raft_consensus.cc:374] T e45af71e3fc14c579b32ba4dbe2adf78 P e632c553ff4341328acf1cbaca678392 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:58.073251 3505 raft_consensus.cc:729] T e45af71e3fc14c579b32ba4dbe2adf78 P e632c553ff4341328acf1cbaca678392 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e632c553ff4341328acf1cbaca678392, State: Initialized, Role: FOLLOWER
I20240502 21:28:58.073588 3505 consensus_queue.cc:260] T e45af71e3fc14c579b32ba4dbe2adf78 P e632c553ff4341328acf1cbaca678392 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:28:58.074580 3506 tablet_bootstrap.cc:492] T 4dad8773f5b54efa9ff527252bac24aa P 6ab8df3c796f4b6cb3e29d5045ad3ea0: No bootstrap required, opened a new log
I20240502 21:28:58.074769 3506 ts_tablet_manager.cc:1367] T 4dad8773f5b54efa9ff527252bac24aa P 6ab8df3c796f4b6cb3e29d5045ad3ea0: Time spent bootstrapping tablet: real 0.023s user 0.000s sys 0.004s
I20240502 21:28:58.074910 3513 tablet_bootstrap.cc:492] T e45af71e3fc14c579b32ba4dbe2adf78 P 16b3a337bc5e462f83ba895ad8660deb: Bootstrap starting.
I20240502 21:28:58.075436 3506 raft_consensus.cc:348] T 4dad8773f5b54efa9ff527252bac24aa P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:28:58.075654 3506 raft_consensus.cc:374] T 4dad8773f5b54efa9ff527252bac24aa P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:58.075749 3506 raft_consensus.cc:729] T 4dad8773f5b54efa9ff527252bac24aa P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ab8df3c796f4b6cb3e29d5045ad3ea0, State: Initialized, Role: FOLLOWER
I20240502 21:28:58.075470 3505 ts_tablet_manager.cc:1398] T e45af71e3fc14c579b32ba4dbe2adf78 P e632c553ff4341328acf1cbaca678392: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 21:28:58.075964 3506 consensus_queue.cc:260] T 4dad8773f5b54efa9ff527252bac24aa P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:28:58.076258 3505 tablet_bootstrap.cc:492] T 4dad8773f5b54efa9ff527252bac24aa P e632c553ff4341328acf1cbaca678392: Bootstrap starting.
I20240502 21:28:58.076622 3506 ts_tablet_manager.cc:1398] T 4dad8773f5b54efa9ff527252bac24aa P 6ab8df3c796f4b6cb3e29d5045ad3ea0: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 21:28:58.078279 3513 tablet_bootstrap.cc:654] T e45af71e3fc14c579b32ba4dbe2adf78 P 16b3a337bc5e462f83ba895ad8660deb: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:58.080602 3505 tablet_bootstrap.cc:654] T 4dad8773f5b54efa9ff527252bac24aa P e632c553ff4341328acf1cbaca678392: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:58.090862 3511 tablet_bootstrap.cc:492] T e45af71e3fc14c579b32ba4dbe2adf78 P 153026dd7f8943cc93fedf6f48450fb4: No bootstrap required, opened a new log
I20240502 21:28:58.092032 3508 tablet_bootstrap.cc:492] T 4dad8773f5b54efa9ff527252bac24aa P 7d1e5c1b047443cbb28d7b0a9744babf: No bootstrap required, opened a new log
I20240502 21:28:58.092041 3512 tablet_bootstrap.cc:492] T e45af71e3fc14c579b32ba4dbe2adf78 P a62a935ff43d494ca35d0c417943133d: No bootstrap required, opened a new log
I20240502 21:28:58.092252 3508 ts_tablet_manager.cc:1367] T 4dad8773f5b54efa9ff527252bac24aa P 7d1e5c1b047443cbb28d7b0a9744babf: Time spent bootstrapping tablet: real 0.023s user 0.006s sys 0.003s
I20240502 21:28:58.093012 3511 ts_tablet_manager.cc:1367] T e45af71e3fc14c579b32ba4dbe2adf78 P 153026dd7f8943cc93fedf6f48450fb4: Time spent bootstrapping tablet: real 0.039s user 0.003s sys 0.005s
I20240502 21:28:58.093120 3508 raft_consensus.cc:348] T 4dad8773f5b54efa9ff527252bac24aa P 7d1e5c1b047443cbb28d7b0a9744babf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:28:58.093452 3508 raft_consensus.cc:374] T 4dad8773f5b54efa9ff527252bac24aa P 7d1e5c1b047443cbb28d7b0a9744babf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:58.093539 3508 raft_consensus.cc:729] T 4dad8773f5b54efa9ff527252bac24aa P 7d1e5c1b047443cbb28d7b0a9744babf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d1e5c1b047443cbb28d7b0a9744babf, State: Initialized, Role: FOLLOWER
I20240502 21:28:58.093369 3512 ts_tablet_manager.cc:1367] T e45af71e3fc14c579b32ba4dbe2adf78 P a62a935ff43d494ca35d0c417943133d: Time spent bootstrapping tablet: real 0.044s user 0.002s sys 0.005s
I20240502 21:28:58.093808 3508 consensus_queue.cc:260] T 4dad8773f5b54efa9ff527252bac24aa P 7d1e5c1b047443cbb28d7b0a9744babf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:28:58.094583 3508 ts_tablet_manager.cc:1398] T 4dad8773f5b54efa9ff527252bac24aa P 7d1e5c1b047443cbb28d7b0a9744babf: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20240502 21:28:58.094942 3508 tablet_bootstrap.cc:492] T e45af71e3fc14c579b32ba4dbe2adf78 P 7d1e5c1b047443cbb28d7b0a9744babf: Bootstrap starting.
I20240502 21:28:58.095911 3511 raft_consensus.cc:348] T e45af71e3fc14c579b32ba4dbe2adf78 P 153026dd7f8943cc93fedf6f48450fb4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:28:58.097174 3512 raft_consensus.cc:348] T e45af71e3fc14c579b32ba4dbe2adf78 P a62a935ff43d494ca35d0c417943133d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:28:58.098181 3508 tablet_bootstrap.cc:654] T e45af71e3fc14c579b32ba4dbe2adf78 P 7d1e5c1b047443cbb28d7b0a9744babf: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:58.099231 3505 tablet_bootstrap.cc:492] T 4dad8773f5b54efa9ff527252bac24aa P e632c553ff4341328acf1cbaca678392: No bootstrap required, opened a new log
I20240502 21:28:58.099493 3505 ts_tablet_manager.cc:1367] T 4dad8773f5b54efa9ff527252bac24aa P e632c553ff4341328acf1cbaca678392: Time spent bootstrapping tablet: real 0.023s user 0.000s sys 0.005s
I20240502 21:28:58.099647 3511 raft_consensus.cc:374] T e45af71e3fc14c579b32ba4dbe2adf78 P 153026dd7f8943cc93fedf6f48450fb4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:58.099768 3511 raft_consensus.cc:729] T e45af71e3fc14c579b32ba4dbe2adf78 P 153026dd7f8943cc93fedf6f48450fb4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 153026dd7f8943cc93fedf6f48450fb4, State: Initialized, Role: FOLLOWER
I20240502 21:28:58.100085 3508 tablet_bootstrap.cc:492] T e45af71e3fc14c579b32ba4dbe2adf78 P 7d1e5c1b047443cbb28d7b0a9744babf: No bootstrap required, opened a new log
I20240502 21:28:58.100234 3508 ts_tablet_manager.cc:1367] T e45af71e3fc14c579b32ba4dbe2adf78 P 7d1e5c1b047443cbb28d7b0a9744babf: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.002s
I20240502 21:28:58.100039 3511 consensus_queue.cc:260] T e45af71e3fc14c579b32ba4dbe2adf78 P 153026dd7f8943cc93fedf6f48450fb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:28:58.100939 3508 raft_consensus.cc:348] T e45af71e3fc14c579b32ba4dbe2adf78 P 7d1e5c1b047443cbb28d7b0a9744babf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:28:58.101169 3508 raft_consensus.cc:374] T e45af71e3fc14c579b32ba4dbe2adf78 P 7d1e5c1b047443cbb28d7b0a9744babf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:58.101259 3508 raft_consensus.cc:729] T e45af71e3fc14c579b32ba4dbe2adf78 P 7d1e5c1b047443cbb28d7b0a9744babf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d1e5c1b047443cbb28d7b0a9744babf, State: Initialized, Role: FOLLOWER
I20240502 21:28:58.101482 3508 consensus_queue.cc:260] T e45af71e3fc14c579b32ba4dbe2adf78 P 7d1e5c1b047443cbb28d7b0a9744babf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:28:58.101687 3467 raft_consensus.cc:482] T e45af71e3fc14c579b32ba4dbe2adf78 P 521bb93035324cecbebed2f6f2a630f8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:28:58.101850 3467 raft_consensus.cc:504] T e45af71e3fc14c579b32ba4dbe2adf78 P 521bb93035324cecbebed2f6f2a630f8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:28:58.101938 3505 raft_consensus.cc:348] T 4dad8773f5b54efa9ff527252bac24aa P e632c553ff4341328acf1cbaca678392 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:28:58.102423 3505 raft_consensus.cc:374] T 4dad8773f5b54efa9ff527252bac24aa P e632c553ff4341328acf1cbaca678392 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:58.102557 3505 raft_consensus.cc:729] T 4dad8773f5b54efa9ff527252bac24aa P e632c553ff4341328acf1cbaca678392 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e632c553ff4341328acf1cbaca678392, State: Initialized, Role: FOLLOWER
I20240502 21:28:58.103106 3467 leader_election.cc:288] T e45af71e3fc14c579b32ba4dbe2adf78 P 521bb93035324cecbebed2f6f2a630f8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e632c553ff4341328acf1cbaca678392 (127.15.255.133:37439), 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132:44947), 6ab8df3c796f4b6cb3e29d5045ad3ea0 (127.15.255.135:34701), a62a935ff43d494ca35d0c417943133d (127.15.255.130:37991), 7d1e5c1b047443cbb28d7b0a9744babf (127.15.255.129:37897), 16b3a337bc5e462f83ba895ad8660deb (127.15.255.131:36033)
I20240502 21:28:58.102981 3505 consensus_queue.cc:260] T 4dad8773f5b54efa9ff527252bac24aa P e632c553ff4341328acf1cbaca678392 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:28:58.104271 3508 ts_tablet_manager.cc:1398] T e45af71e3fc14c579b32ba4dbe2adf78 P 7d1e5c1b047443cbb28d7b0a9744babf: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240502 21:28:58.104390 2227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e45af71e3fc14c579b32ba4dbe2adf78" candidate_uuid: "521bb93035324cecbebed2f6f2a630f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" is_pre_election: true
I20240502 21:28:58.104631 1415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e45af71e3fc14c579b32ba4dbe2adf78" candidate_uuid: "521bb93035324cecbebed2f6f2a630f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" is_pre_election: true
I20240502 21:28:58.104696 2227 raft_consensus.cc:2454] T e45af71e3fc14c579b32ba4dbe2adf78 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 521bb93035324cecbebed2f6f2a630f8 in term 0.
I20240502 21:28:58.104843 1415 raft_consensus.cc:2454] T e45af71e3fc14c579b32ba4dbe2adf78 P 7d1e5c1b047443cbb28d7b0a9744babf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 521bb93035324cecbebed2f6f2a630f8 in term 0.
I20240502 21:28:58.105198 1685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e45af71e3fc14c579b32ba4dbe2adf78" candidate_uuid: "521bb93035324cecbebed2f6f2a630f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16b3a337bc5e462f83ba895ad8660deb" is_pre_election: true
I20240502 21:28:58.105610 1818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e45af71e3fc14c579b32ba4dbe2adf78" candidate_uuid: "521bb93035324cecbebed2f6f2a630f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "153026dd7f8943cc93fedf6f48450fb4" is_pre_election: true
I20240502 21:28:58.105829 1818 raft_consensus.cc:2454] T e45af71e3fc14c579b32ba4dbe2adf78 P 153026dd7f8943cc93fedf6f48450fb4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 521bb93035324cecbebed2f6f2a630f8 in term 0.
I20240502 21:28:58.104069 1955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e45af71e3fc14c579b32ba4dbe2adf78" candidate_uuid: "521bb93035324cecbebed2f6f2a630f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e632c553ff4341328acf1cbaca678392" is_pre_election: true
W20240502 21:28:58.106068 2025 leader_election.cc:341] T e45af71e3fc14c579b32ba4dbe2adf78 P 521bb93035324cecbebed2f6f2a630f8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 16b3a337bc5e462f83ba895ad8660deb (127.15.255.131:36033): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:28:58.106062 1955 raft_consensus.cc:2454] T e45af71e3fc14c579b32ba4dbe2adf78 P e632c553ff4341328acf1cbaca678392 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 521bb93035324cecbebed2f6f2a630f8 in term 0.
I20240502 21:28:58.106737 2023 leader_election.cc:302] T e45af71e3fc14c579b32ba4dbe2adf78 P 521bb93035324cecbebed2f6f2a630f8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 5 responses out of 7 voters: 4 yes votes; 1 no votes. yes voters: 153026dd7f8943cc93fedf6f48450fb4, 521bb93035324cecbebed2f6f2a630f8, 7d1e5c1b047443cbb28d7b0a9744babf, e632c553ff4341328acf1cbaca678392; no voters: 16b3a337bc5e462f83ba895ad8660deb
I20240502 21:28:58.107208 3467 raft_consensus.cc:2781] T e45af71e3fc14c579b32ba4dbe2adf78 P 521bb93035324cecbebed2f6f2a630f8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:28:58.107323 3467 raft_consensus.cc:482] T e45af71e3fc14c579b32ba4dbe2adf78 P 521bb93035324cecbebed2f6f2a630f8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:28:58.107417 3467 raft_consensus.cc:3037] T e45af71e3fc14c579b32ba4dbe2adf78 P 521bb93035324cecbebed2f6f2a630f8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:58.107460 3505 ts_tablet_manager.cc:1398] T 4dad8773f5b54efa9ff527252bac24aa P e632c553ff4341328acf1cbaca678392: Time spent starting tablet: real 0.008s user 0.000s sys 0.002s
I20240502 21:28:58.107334 3512 raft_consensus.cc:374] T e45af71e3fc14c579b32ba4dbe2adf78 P a62a935ff43d494ca35d0c417943133d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:58.108034 3505 tablet_bootstrap.cc:492] T 72355dd734ea4aeba031171e8c39696d P e632c553ff4341328acf1cbaca678392: Bootstrap starting.
I20240502 21:28:58.108078 3512 raft_consensus.cc:729] T e45af71e3fc14c579b32ba4dbe2adf78 P a62a935ff43d494ca35d0c417943133d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a62a935ff43d494ca35d0c417943133d, State: Initialized, Role: FOLLOWER
I20240502 21:28:58.108484 3512 consensus_queue.cc:260] T e45af71e3fc14c579b32ba4dbe2adf78 P a62a935ff43d494ca35d0c417943133d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:28:58.104341 1549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e45af71e3fc14c579b32ba4dbe2adf78" candidate_uuid: "521bb93035324cecbebed2f6f2a630f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a62a935ff43d494ca35d0c417943133d" is_pre_election: true
I20240502 21:28:58.109637 3511 ts_tablet_manager.cc:1398] T e45af71e3fc14c579b32ba4dbe2adf78 P 153026dd7f8943cc93fedf6f48450fb4: Time spent starting tablet: real 0.016s user 0.001s sys 0.001s
I20240502 21:28:58.109632 1549 raft_consensus.cc:2454] T e45af71e3fc14c579b32ba4dbe2adf78 P a62a935ff43d494ca35d0c417943133d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 521bb93035324cecbebed2f6f2a630f8 in term 0.
I20240502 21:28:58.111099 3467 raft_consensus.cc:504] T e45af71e3fc14c579b32ba4dbe2adf78 P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:28:58.112535 1818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e45af71e3fc14c579b32ba4dbe2adf78" candidate_uuid: "521bb93035324cecbebed2f6f2a630f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "153026dd7f8943cc93fedf6f48450fb4"
I20240502 21:28:58.112730 1818 raft_consensus.cc:3037] T e45af71e3fc14c579b32ba4dbe2adf78 P 153026dd7f8943cc93fedf6f48450fb4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:58.113360 3511 tablet_bootstrap.cc:492] T 4dad8773f5b54efa9ff527252bac24aa P 153026dd7f8943cc93fedf6f48450fb4: Bootstrap starting.
I20240502 21:28:58.113454 3512 ts_tablet_manager.cc:1398] T e45af71e3fc14c579b32ba4dbe2adf78 P a62a935ff43d494ca35d0c417943133d: Time spent starting tablet: real 0.016s user 0.002s sys 0.000s
I20240502 21:28:58.113440 1955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e45af71e3fc14c579b32ba4dbe2adf78" candidate_uuid: "521bb93035324cecbebed2f6f2a630f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e632c553ff4341328acf1cbaca678392"
I20240502 21:28:58.113672 1955 raft_consensus.cc:3037] T e45af71e3fc14c579b32ba4dbe2adf78 P e632c553ff4341328acf1cbaca678392 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:58.114001 3512 tablet_bootstrap.cc:492] T 72355dd734ea4aeba031171e8c39696d P a62a935ff43d494ca35d0c417943133d: Bootstrap starting.
I20240502 21:28:58.115622 1549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e45af71e3fc14c579b32ba4dbe2adf78" candidate_uuid: "521bb93035324cecbebed2f6f2a630f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a62a935ff43d494ca35d0c417943133d"
I20240502 21:28:58.114846 1818 raft_consensus.cc:2454] T e45af71e3fc14c579b32ba4dbe2adf78 P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 521bb93035324cecbebed2f6f2a630f8 in term 1.
I20240502 21:28:58.115821 1549 raft_consensus.cc:3037] T e45af71e3fc14c579b32ba4dbe2adf78 P a62a935ff43d494ca35d0c417943133d [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:58.115810 1415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e45af71e3fc14c579b32ba4dbe2adf78" candidate_uuid: "521bb93035324cecbebed2f6f2a630f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d1e5c1b047443cbb28d7b0a9744babf"
I20240502 21:28:58.116009 1415 raft_consensus.cc:3037] T e45af71e3fc14c579b32ba4dbe2adf78 P 7d1e5c1b047443cbb28d7b0a9744babf [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:58.116328 1955 raft_consensus.cc:2454] T e45af71e3fc14c579b32ba4dbe2adf78 P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 521bb93035324cecbebed2f6f2a630f8 in term 1.
I20240502 21:28:58.117192 2227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e45af71e3fc14c579b32ba4dbe2adf78" candidate_uuid: "521bb93035324cecbebed2f6f2a630f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0"
I20240502 21:28:58.117405 2227 raft_consensus.cc:3037] T e45af71e3fc14c579b32ba4dbe2adf78 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:58.117952 3511 tablet_bootstrap.cc:654] T 4dad8773f5b54efa9ff527252bac24aa P 153026dd7f8943cc93fedf6f48450fb4: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:58.118571 1549 raft_consensus.cc:2454] T e45af71e3fc14c579b32ba4dbe2adf78 P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 521bb93035324cecbebed2f6f2a630f8 in term 1.
I20240502 21:28:58.115830 3512 tablet_bootstrap.cc:654] T 72355dd734ea4aeba031171e8c39696d P a62a935ff43d494ca35d0c417943133d: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:58.119864 3513 tablet_bootstrap.cc:492] T e45af71e3fc14c579b32ba4dbe2adf78 P 16b3a337bc5e462f83ba895ad8660deb: No bootstrap required, opened a new log
I20240502 21:28:58.120025 3513 ts_tablet_manager.cc:1367] T e45af71e3fc14c579b32ba4dbe2adf78 P 16b3a337bc5e462f83ba895ad8660deb: Time spent bootstrapping tablet: real 0.045s user 0.003s sys 0.002s
I20240502 21:28:58.120056 2227 raft_consensus.cc:2454] T e45af71e3fc14c579b32ba4dbe2adf78 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 521bb93035324cecbebed2f6f2a630f8 in term 1.
I20240502 21:28:58.118697 1415 raft_consensus.cc:2454] T e45af71e3fc14c579b32ba4dbe2adf78 P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 521bb93035324cecbebed2f6f2a630f8 in term 1.
I20240502 21:28:58.120839 3487 raft_consensus.cc:482] T 4dad8773f5b54efa9ff527252bac24aa P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:28:58.121018 3487 raft_consensus.cc:504] T 4dad8773f5b54efa9ff527252bac24aa P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:28:58.121356 1685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e45af71e3fc14c579b32ba4dbe2adf78" candidate_uuid: "521bb93035324cecbebed2f6f2a630f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16b3a337bc5e462f83ba895ad8660deb"
I20240502 21:28:58.120883 3513 raft_consensus.cc:348] T e45af71e3fc14c579b32ba4dbe2adf78 P 16b3a337bc5e462f83ba895ad8660deb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:28:58.121518 3513 raft_consensus.cc:374] T e45af71e3fc14c579b32ba4dbe2adf78 P 16b3a337bc5e462f83ba895ad8660deb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:58.121601 3513 raft_consensus.cc:729] T e45af71e3fc14c579b32ba4dbe2adf78 P 16b3a337bc5e462f83ba895ad8660deb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16b3a337bc5e462f83ba895ad8660deb, State: Initialized, Role: FOLLOWER
I20240502 21:28:58.121588 3467 leader_election.cc:288] T e45af71e3fc14c579b32ba4dbe2adf78 P 521bb93035324cecbebed2f6f2a630f8 [CANDIDATE]: Term 1 election: Requested vote from peers e632c553ff4341328acf1cbaca678392 (127.15.255.133:37439), 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132:44947), 6ab8df3c796f4b6cb3e29d5045ad3ea0 (127.15.255.135:34701), a62a935ff43d494ca35d0c417943133d (127.15.255.130:37991), 7d1e5c1b047443cbb28d7b0a9744babf (127.15.255.129:37897), 16b3a337bc5e462f83ba895ad8660deb (127.15.255.131:36033)
I20240502 21:28:58.121845 3513 consensus_queue.cc:260] T e45af71e3fc14c579b32ba4dbe2adf78 P 16b3a337bc5e462f83ba895ad8660deb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:28:58.122201 1685 raft_consensus.cc:3037] T e45af71e3fc14c579b32ba4dbe2adf78 P 16b3a337bc5e462f83ba895ad8660deb [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:58.122450 3487 leader_election.cc:288] T 4dad8773f5b54efa9ff527252bac24aa P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e632c553ff4341328acf1cbaca678392 (127.15.255.133:37439), 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132:44947), 521bb93035324cecbebed2f6f2a630f8 (127.15.255.134:40821), 7d1e5c1b047443cbb28d7b0a9744babf (127.15.255.129:37897), a62a935ff43d494ca35d0c417943133d (127.15.255.130:37991), 16b3a337bc5e462f83ba895ad8660deb (127.15.255.131:36033)
I20240502 21:28:58.123718 1818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4dad8773f5b54efa9ff527252bac24aa" candidate_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "153026dd7f8943cc93fedf6f48450fb4" is_pre_election: true
I20240502 21:28:58.123811 3442 raft_consensus.cc:482] T 4dad8773f5b54efa9ff527252bac24aa P e632c553ff4341328acf1cbaca678392 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:28:58.123950 3442 raft_consensus.cc:504] T 4dad8773f5b54efa9ff527252bac24aa P e632c553ff4341328acf1cbaca678392 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
W20240502 21:28:58.124368 2160 leader_election.cc:341] T 4dad8773f5b54efa9ff527252bac24aa P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132:44947): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:28:58.124938 1549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4dad8773f5b54efa9ff527252bac24aa" candidate_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a62a935ff43d494ca35d0c417943133d" is_pre_election: true
I20240502 21:28:58.125203 1685 raft_consensus.cc:2454] T e45af71e3fc14c579b32ba4dbe2adf78 P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 521bb93035324cecbebed2f6f2a630f8 in term 1.
I20240502 21:28:58.125672 1954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4dad8773f5b54efa9ff527252bac24aa" candidate_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e632c553ff4341328acf1cbaca678392" is_pre_election: true
I20240502 21:28:58.125916 1954 raft_consensus.cc:2454] T 4dad8773f5b54efa9ff527252bac24aa P e632c553ff4341328acf1cbaca678392 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ab8df3c796f4b6cb3e29d5045ad3ea0 in term 0.
I20240502 21:28:58.126602 3442 leader_election.cc:288] T 4dad8773f5b54efa9ff527252bac24aa P e632c553ff4341328acf1cbaca678392 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132:44947), 521bb93035324cecbebed2f6f2a630f8 (127.15.255.134:40821), 6ab8df3c796f4b6cb3e29d5045ad3ea0 (127.15.255.135:34701), 7d1e5c1b047443cbb28d7b0a9744babf (127.15.255.129:37897), a62a935ff43d494ca35d0c417943133d (127.15.255.130:37991), 16b3a337bc5e462f83ba895ad8660deb (127.15.255.131:36033)
I20240502 21:28:58.126643 1818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4dad8773f5b54efa9ff527252bac24aa" candidate_uuid: "e632c553ff4341328acf1cbaca678392" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "153026dd7f8943cc93fedf6f48450fb4" is_pre_election: true
I20240502 21:28:58.127322 2025 leader_election.cc:302] T e45af71e3fc14c579b32ba4dbe2adf78 P 521bb93035324cecbebed2f6f2a630f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 153026dd7f8943cc93fedf6f48450fb4, 16b3a337bc5e462f83ba895ad8660deb, 521bb93035324cecbebed2f6f2a630f8, a62a935ff43d494ca35d0c417943133d; no voters:
I20240502 21:28:58.127730 1414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4dad8773f5b54efa9ff527252bac24aa" candidate_uuid: "e632c553ff4341328acf1cbaca678392" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" is_pre_election: true
I20240502 21:28:58.127951 1414 raft_consensus.cc:2454] T 4dad8773f5b54efa9ff527252bac24aa P 7d1e5c1b047443cbb28d7b0a9744babf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e632c553ff4341328acf1cbaca678392 in term 0.
I20240502 21:28:58.128321 2088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4dad8773f5b54efa9ff527252bac24aa" candidate_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "521bb93035324cecbebed2f6f2a630f8" is_pre_election: true
I20240502 21:28:58.128504 3471 raft_consensus.cc:2781] T e45af71e3fc14c579b32ba4dbe2adf78 P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:28:58.128582 2088 raft_consensus.cc:2454] T 4dad8773f5b54efa9ff527252bac24aa P 521bb93035324cecbebed2f6f2a630f8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ab8df3c796f4b6cb3e29d5045ad3ea0 in term 0.
I20240502 21:28:58.128651 3471 raft_consensus.cc:686] T e45af71e3fc14c579b32ba4dbe2adf78 P 521bb93035324cecbebed2f6f2a630f8 [term 1 LEADER]: Becoming Leader. State: Replica: 521bb93035324cecbebed2f6f2a630f8, State: Running, Role: LEADER
I20240502 21:28:58.129033 3471 consensus_queue.cc:237] T e45af71e3fc14c579b32ba4dbe2adf78 P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
W20240502 21:28:58.129335 2159 leader_election.cc:341] T 4dad8773f5b54efa9ff527252bac24aa P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a62a935ff43d494ca35d0c417943133d (127.15.255.130:37991): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:28:58.129871 2227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4dad8773f5b54efa9ff527252bac24aa" candidate_uuid: "e632c553ff4341328acf1cbaca678392" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" is_pre_election: true
I20240502 21:28:58.130110 2227 raft_consensus.cc:2454] T 4dad8773f5b54efa9ff527252bac24aa P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e632c553ff4341328acf1cbaca678392 in term 0.
I20240502 21:28:58.131407 1415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4dad8773f5b54efa9ff527252bac24aa" candidate_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" is_pre_election: true
I20240502 21:28:58.131637 1415 raft_consensus.cc:2454] T 4dad8773f5b54efa9ff527252bac24aa P 7d1e5c1b047443cbb28d7b0a9744babf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ab8df3c796f4b6cb3e29d5045ad3ea0 in term 0.
I20240502 21:28:58.127317 2089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4dad8773f5b54efa9ff527252bac24aa" candidate_uuid: "e632c553ff4341328acf1cbaca678392" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "521bb93035324cecbebed2f6f2a630f8" is_pre_election: true
I20240502 21:28:58.131901 2089 raft_consensus.cc:2454] T 4dad8773f5b54efa9ff527252bac24aa P 521bb93035324cecbebed2f6f2a630f8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e632c553ff4341328acf1cbaca678392 in term 0.
I20240502 21:28:58.132380 1884 leader_election.cc:302] T 4dad8773f5b54efa9ff527252bac24aa P e632c553ff4341328acf1cbaca678392 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 521bb93035324cecbebed2f6f2a630f8, 6ab8df3c796f4b6cb3e29d5045ad3ea0, 7d1e5c1b047443cbb28d7b0a9744babf, e632c553ff4341328acf1cbaca678392; no voters:
I20240502 21:28:58.132725 3442 raft_consensus.cc:2781] T 4dad8773f5b54efa9ff527252bac24aa P e632c553ff4341328acf1cbaca678392 [term 0 FOLLOWER]: Leader pre-election won for term 1
W20240502 21:28:58.132791 1886 leader_election.cc:341] T 4dad8773f5b54efa9ff527252bac24aa P e632c553ff4341328acf1cbaca678392 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132:44947): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:28:58.132858 3442 raft_consensus.cc:482] T 4dad8773f5b54efa9ff527252bac24aa P e632c553ff4341328acf1cbaca678392 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:28:58.132946 3442 raft_consensus.cc:3037] T 4dad8773f5b54efa9ff527252bac24aa P e632c553ff4341328acf1cbaca678392 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:58.133316 1684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4dad8773f5b54efa9ff527252bac24aa" candidate_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16b3a337bc5e462f83ba895ad8660deb" is_pre_election: true
W20240502 21:28:58.133885 2161 leader_election.cc:341] T 4dad8773f5b54efa9ff527252bac24aa P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 16b3a337bc5e462f83ba895ad8660deb (127.15.255.131:36033): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:28:58.134090 1549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4dad8773f5b54efa9ff527252bac24aa" candidate_uuid: "e632c553ff4341328acf1cbaca678392" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a62a935ff43d494ca35d0c417943133d" is_pre_election: true
W20240502 21:28:58.134637 1885 leader_election.cc:341] T 4dad8773f5b54efa9ff527252bac24aa P e632c553ff4341328acf1cbaca678392 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a62a935ff43d494ca35d0c417943133d (127.15.255.130:37991): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:28:58.135143 2158 leader_election.cc:302] T 4dad8773f5b54efa9ff527252bac24aa P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 7 responses out of 7 voters: 4 yes votes; 3 no votes. yes voters: 521bb93035324cecbebed2f6f2a630f8, 6ab8df3c796f4b6cb3e29d5045ad3ea0, 7d1e5c1b047443cbb28d7b0a9744babf, e632c553ff4341328acf1cbaca678392; no voters: 153026dd7f8943cc93fedf6f48450fb4, 16b3a337bc5e462f83ba895ad8660deb, a62a935ff43d494ca35d0c417943133d
I20240502 21:28:58.136003 1684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4dad8773f5b54efa9ff527252bac24aa" candidate_uuid: "e632c553ff4341328acf1cbaca678392" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16b3a337bc5e462f83ba895ad8660deb" is_pre_election: true
I20240502 21:28:58.136194 3487 raft_consensus.cc:2781] T 4dad8773f5b54efa9ff527252bac24aa P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:28:58.136308 3487 raft_consensus.cc:482] T 4dad8773f5b54efa9ff527252bac24aa P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:28:58.136413 3487 raft_consensus.cc:3037] T 4dad8773f5b54efa9ff527252bac24aa P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 0 FOLLOWER]: Advancing to term 1
W20240502 21:28:58.136595 1887 leader_election.cc:341] T 4dad8773f5b54efa9ff527252bac24aa P e632c553ff4341328acf1cbaca678392 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 16b3a337bc5e462f83ba895ad8660deb (127.15.255.131:36033): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:28:58.139799 3513 ts_tablet_manager.cc:1398] T e45af71e3fc14c579b32ba4dbe2adf78 P 16b3a337bc5e462f83ba895ad8660deb: Time spent starting tablet: real 0.020s user 0.014s sys 0.005s
I20240502 21:28:58.140255 3513 tablet_bootstrap.cc:492] T 4dad8773f5b54efa9ff527252bac24aa P 16b3a337bc5e462f83ba895ad8660deb: Bootstrap starting.
I20240502 21:28:58.141157 1267 catalog_manager.cc:5484] T e45af71e3fc14c579b32ba4dbe2adf78 P 521bb93035324cecbebed2f6f2a630f8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 521bb93035324cecbebed2f6f2a630f8 (127.15.255.134). New cstate: current_term: 1 leader_uuid: "521bb93035324cecbebed2f6f2a630f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } health_report { overall_health: UNKNOWN } } }
I20240502 21:28:58.142088 3513 tablet_bootstrap.cc:654] T 4dad8773f5b54efa9ff527252bac24aa P 16b3a337bc5e462f83ba895ad8660deb: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:58.144861 3513 tablet_bootstrap.cc:492] T 4dad8773f5b54efa9ff527252bac24aa P 16b3a337bc5e462f83ba895ad8660deb: No bootstrap required, opened a new log
I20240502 21:28:58.145092 3513 ts_tablet_manager.cc:1367] T 4dad8773f5b54efa9ff527252bac24aa P 16b3a337bc5e462f83ba895ad8660deb: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.001s
I20240502 21:28:58.145890 3513 raft_consensus.cc:348] T 4dad8773f5b54efa9ff527252bac24aa P 16b3a337bc5e462f83ba895ad8660deb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:28:58.146219 3513 raft_consensus.cc:374] T 4dad8773f5b54efa9ff527252bac24aa P 16b3a337bc5e462f83ba895ad8660deb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:58.146034 3442 raft_consensus.cc:504] T 4dad8773f5b54efa9ff527252bac24aa P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:28:58.146356 3513 raft_consensus.cc:729] T 4dad8773f5b54efa9ff527252bac24aa P 16b3a337bc5e462f83ba895ad8660deb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16b3a337bc5e462f83ba895ad8660deb, State: Initialized, Role: FOLLOWER
I20240502 21:28:58.146670 3513 consensus_queue.cc:260] T 4dad8773f5b54efa9ff527252bac24aa P 16b3a337bc5e462f83ba895ad8660deb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:28:58.147056 3487 raft_consensus.cc:504] T 4dad8773f5b54efa9ff527252bac24aa P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:28:58.147298 3513 ts_tablet_manager.cc:1398] T 4dad8773f5b54efa9ff527252bac24aa P 16b3a337bc5e462f83ba895ad8660deb: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 21:28:58.147688 3513 tablet_bootstrap.cc:492] T 72355dd734ea4aeba031171e8c39696d P 16b3a337bc5e462f83ba895ad8660deb: Bootstrap starting.
I20240502 21:28:58.148743 1818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4dad8773f5b54efa9ff527252bac24aa" candidate_uuid: "e632c553ff4341328acf1cbaca678392" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "153026dd7f8943cc93fedf6f48450fb4"
I20240502 21:28:58.149139 3513 tablet_bootstrap.cc:654] T 72355dd734ea4aeba031171e8c39696d P 16b3a337bc5e462f83ba895ad8660deb: Neither blocks nor log segments found. Creating new log.
W20240502 21:28:58.149369 1886 leader_election.cc:341] T 4dad8773f5b54efa9ff527252bac24aa P e632c553ff4341328acf1cbaca678392 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132:44947): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:28:58.150557 1955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4dad8773f5b54efa9ff527252bac24aa" candidate_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e632c553ff4341328acf1cbaca678392"
I20240502 21:28:58.150867 1955 raft_consensus.cc:2379] T 4dad8773f5b54efa9ff527252bac24aa P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 6ab8df3c796f4b6cb3e29d5045ad3ea0 in current term 1: Already voted for candidate e632c553ff4341328acf1cbaca678392 in this term.
I20240502 21:28:58.152060 2227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4dad8773f5b54efa9ff527252bac24aa" candidate_uuid: "e632c553ff4341328acf1cbaca678392" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0"
I20240502 21:28:58.152230 2089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4dad8773f5b54efa9ff527252bac24aa" candidate_uuid: "e632c553ff4341328acf1cbaca678392" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "521bb93035324cecbebed2f6f2a630f8"
I20240502 21:28:58.152495 2227 raft_consensus.cc:2379] T 4dad8773f5b54efa9ff527252bac24aa P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate e632c553ff4341328acf1cbaca678392 in current term 1: Already voted for candidate 6ab8df3c796f4b6cb3e29d5045ad3ea0 in this term.
I20240502 21:28:58.152518 2089 raft_consensus.cc:3037] T 4dad8773f5b54efa9ff527252bac24aa P 521bb93035324cecbebed2f6f2a630f8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:58.153717 1415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4dad8773f5b54efa9ff527252bac24aa" candidate_uuid: "e632c553ff4341328acf1cbaca678392" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d1e5c1b047443cbb28d7b0a9744babf"
I20240502 21:28:58.154024 1415 raft_consensus.cc:3037] T 4dad8773f5b54efa9ff527252bac24aa P 7d1e5c1b047443cbb28d7b0a9744babf [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:58.155395 2089 raft_consensus.cc:2454] T 4dad8773f5b54efa9ff527252bac24aa P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e632c553ff4341328acf1cbaca678392 in term 1.
I20240502 21:28:58.156996 1415 raft_consensus.cc:2454] T 4dad8773f5b54efa9ff527252bac24aa P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e632c553ff4341328acf1cbaca678392 in term 1.
I20240502 21:28:58.157271 3513 tablet_bootstrap.cc:492] T 72355dd734ea4aeba031171e8c39696d P 16b3a337bc5e462f83ba895ad8660deb: No bootstrap required, opened a new log
I20240502 21:28:58.157466 3513 ts_tablet_manager.cc:1367] T 72355dd734ea4aeba031171e8c39696d P 16b3a337bc5e462f83ba895ad8660deb: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.009s
I20240502 21:28:58.158156 3513 raft_consensus.cc:348] T 72355dd734ea4aeba031171e8c39696d P 16b3a337bc5e462f83ba895ad8660deb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:28:58.158373 3513 raft_consensus.cc:374] T 72355dd734ea4aeba031171e8c39696d P 16b3a337bc5e462f83ba895ad8660deb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:58.158493 3513 raft_consensus.cc:729] T 72355dd734ea4aeba031171e8c39696d P 16b3a337bc5e462f83ba895ad8660deb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16b3a337bc5e462f83ba895ad8660deb, State: Initialized, Role: FOLLOWER
I20240502 21:28:58.158771 3513 consensus_queue.cc:260] T 72355dd734ea4aeba031171e8c39696d P 16b3a337bc5e462f83ba895ad8660deb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:28:58.159423 1549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4dad8773f5b54efa9ff527252bac24aa" candidate_uuid: "e632c553ff4341328acf1cbaca678392" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a62a935ff43d494ca35d0c417943133d"
I20240502 21:28:58.159569 1684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4dad8773f5b54efa9ff527252bac24aa" candidate_uuid: "e632c553ff4341328acf1cbaca678392" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16b3a337bc5e462f83ba895ad8660deb"
I20240502 21:28:58.159804 1684 raft_consensus.cc:3037] T 4dad8773f5b54efa9ff527252bac24aa P 16b3a337bc5e462f83ba895ad8660deb [term 0 FOLLOWER]: Advancing to term 1
W20240502 21:28:58.160065 1885 leader_election.cc:341] T 4dad8773f5b54efa9ff527252bac24aa P e632c553ff4341328acf1cbaca678392 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a62a935ff43d494ca35d0c417943133d (127.15.255.130:37991): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:28:58.160260 3513 ts_tablet_manager.cc:1398] T 72355dd734ea4aeba031171e8c39696d P 16b3a337bc5e462f83ba895ad8660deb: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240502 21:28:58.160884 1818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4dad8773f5b54efa9ff527252bac24aa" candidate_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "153026dd7f8943cc93fedf6f48450fb4"
W20240502 21:28:58.161415 2160 leader_election.cc:341] T 4dad8773f5b54efa9ff527252bac24aa P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132:44947): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:28:58.162488 1684 raft_consensus.cc:2454] T 4dad8773f5b54efa9ff527252bac24aa P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e632c553ff4341328acf1cbaca678392 in term 1.
I20240502 21:28:58.162904 1887 leader_election.cc:302] T 4dad8773f5b54efa9ff527252bac24aa P e632c553ff4341328acf1cbaca678392 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 7 responses out of 7 voters: 4 yes votes; 3 no votes. yes voters: 16b3a337bc5e462f83ba895ad8660deb, 521bb93035324cecbebed2f6f2a630f8, 7d1e5c1b047443cbb28d7b0a9744babf, e632c553ff4341328acf1cbaca678392; no voters: 153026dd7f8943cc93fedf6f48450fb4, 6ab8df3c796f4b6cb3e29d5045ad3ea0, a62a935ff43d494ca35d0c417943133d
I20240502 21:28:58.162986 2089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4dad8773f5b54efa9ff527252bac24aa" candidate_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "521bb93035324cecbebed2f6f2a630f8"
I20240502 21:28:58.163203 2089 raft_consensus.cc:2379] T 4dad8773f5b54efa9ff527252bac24aa P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 6ab8df3c796f4b6cb3e29d5045ad3ea0 in current term 1: Already voted for candidate e632c553ff4341328acf1cbaca678392 in this term.
I20240502 21:28:58.163341 3443 raft_consensus.cc:2781] T 4dad8773f5b54efa9ff527252bac24aa P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:28:58.163717 3441 raft_consensus.cc:482] T 72355dd734ea4aeba031171e8c39696d P 7d1e5c1b047443cbb28d7b0a9744babf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:28:58.163883 3441 raft_consensus.cc:504] T 72355dd734ea4aeba031171e8c39696d P 7d1e5c1b047443cbb28d7b0a9744babf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:28:58.164220 1414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4dad8773f5b54efa9ff527252bac24aa" candidate_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d1e5c1b047443cbb28d7b0a9744babf"
I20240502 21:28:58.164572 1414 raft_consensus.cc:2379] T 4dad8773f5b54efa9ff527252bac24aa P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 6ab8df3c796f4b6cb3e29d5045ad3ea0 in current term 1: Already voted for candidate e632c553ff4341328acf1cbaca678392 in this term.
I20240502 21:28:58.165259 3441 leader_election.cc:288] T 72355dd734ea4aeba031171e8c39696d P 7d1e5c1b047443cbb28d7b0a9744babf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 521bb93035324cecbebed2f6f2a630f8 (127.15.255.134:40821), 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132:44947), e632c553ff4341328acf1cbaca678392 (127.15.255.133:37439), 6ab8df3c796f4b6cb3e29d5045ad3ea0 (127.15.255.135:34701), a62a935ff43d494ca35d0c417943133d (127.15.255.130:37991), 16b3a337bc5e462f83ba895ad8660deb (127.15.255.131:36033)
I20240502 21:28:58.165524 2089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72355dd734ea4aeba031171e8c39696d" candidate_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "521bb93035324cecbebed2f6f2a630f8" is_pre_election: true
I20240502 21:28:58.165717 2089 raft_consensus.cc:2454] T 72355dd734ea4aeba031171e8c39696d P 521bb93035324cecbebed2f6f2a630f8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d1e5c1b047443cbb28d7b0a9744babf in term 0.
I20240502 21:28:58.165755 1550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72355dd734ea4aeba031171e8c39696d" candidate_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a62a935ff43d494ca35d0c417943133d" is_pre_election: true
I20240502 21:28:58.166639 1549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4dad8773f5b54efa9ff527252bac24aa" candidate_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a62a935ff43d494ca35d0c417943133d"
I20240502 21:28:58.166616 2159 leader_election.cc:302] T 4dad8773f5b54efa9ff527252bac24aa P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 5 responses out of 7 voters: 1 yes votes; 4 no votes. yes voters: 6ab8df3c796f4b6cb3e29d5045ad3ea0; no voters: 153026dd7f8943cc93fedf6f48450fb4, 521bb93035324cecbebed2f6f2a630f8, 7d1e5c1b047443cbb28d7b0a9744babf, e632c553ff4341328acf1cbaca678392
I20240502 21:28:58.166962 3293 raft_consensus.cc:2726] T 4dad8773f5b54efa9ff527252bac24aa P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240502 21:28:58.167013 1685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72355dd734ea4aeba031171e8c39696d" candidate_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16b3a337bc5e462f83ba895ad8660deb" is_pre_election: true
I20240502 21:28:58.167210 1685 raft_consensus.cc:2454] T 72355dd734ea4aeba031171e8c39696d P 16b3a337bc5e462f83ba895ad8660deb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d1e5c1b047443cbb28d7b0a9744babf in term 0.
W20240502 21:28:58.167270 2159 leader_election.cc:341] T 4dad8773f5b54efa9ff527252bac24aa P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a62a935ff43d494ca35d0c417943133d (127.15.255.130:37991): Illegal state: must be running to vote when last-logged opid is not known
W20240502 21:28:58.167614 1348 leader_election.cc:341] T 72355dd734ea4aeba031171e8c39696d P 7d1e5c1b047443cbb28d7b0a9744babf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a62a935ff43d494ca35d0c417943133d (127.15.255.130:37991): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:28:58.167789 1818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72355dd734ea4aeba031171e8c39696d" candidate_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "153026dd7f8943cc93fedf6f48450fb4" is_pre_election: true
W20240502 21:28:58.168269 1349 leader_election.cc:341] T 72355dd734ea4aeba031171e8c39696d P 7d1e5c1b047443cbb28d7b0a9744babf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132:44947): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:28:58.168282 1684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4dad8773f5b54efa9ff527252bac24aa" candidate_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16b3a337bc5e462f83ba895ad8660deb"
I20240502 21:28:58.168641 1684 raft_consensus.cc:2379] T 4dad8773f5b54efa9ff527252bac24aa P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 6ab8df3c796f4b6cb3e29d5045ad3ea0 in current term 1: Already voted for candidate e632c553ff4341328acf1cbaca678392 in this term.
I20240502 21:28:58.169230 2227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72355dd734ea4aeba031171e8c39696d" candidate_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" is_pre_election: true
I20240502 21:28:58.169517 2227 raft_consensus.cc:2454] T 72355dd734ea4aeba031171e8c39696d P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d1e5c1b047443cbb28d7b0a9744babf in term 0.
I20240502 21:28:58.170377 1348 leader_election.cc:302] T 72355dd734ea4aeba031171e8c39696d P 7d1e5c1b047443cbb28d7b0a9744babf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 6 responses out of 7 voters: 4 yes votes; 2 no votes. yes voters: 16b3a337bc5e462f83ba895ad8660deb, 521bb93035324cecbebed2f6f2a630f8, 6ab8df3c796f4b6cb3e29d5045ad3ea0, 7d1e5c1b047443cbb28d7b0a9744babf; no voters: 153026dd7f8943cc93fedf6f48450fb4, a62a935ff43d494ca35d0c417943133d
I20240502 21:28:58.170778 3442 leader_election.cc:288] T 4dad8773f5b54efa9ff527252bac24aa P e632c553ff4341328acf1cbaca678392 [CANDIDATE]: Term 1 election: Requested vote from peers 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132:44947), 521bb93035324cecbebed2f6f2a630f8 (127.15.255.134:40821), 6ab8df3c796f4b6cb3e29d5045ad3ea0 (127.15.255.135:34701), 7d1e5c1b047443cbb28d7b0a9744babf (127.15.255.129:37897), a62a935ff43d494ca35d0c417943133d (127.15.255.130:37991), 16b3a337bc5e462f83ba895ad8660deb (127.15.255.131:36033)
I20240502 21:28:58.171178 3441 raft_consensus.cc:2781] T 72355dd734ea4aeba031171e8c39696d P 7d1e5c1b047443cbb28d7b0a9744babf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:28:58.171314 3441 raft_consensus.cc:482] T 72355dd734ea4aeba031171e8c39696d P 7d1e5c1b047443cbb28d7b0a9744babf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:28:58.171396 3441 raft_consensus.cc:3037] T 72355dd734ea4aeba031171e8c39696d P 7d1e5c1b047443cbb28d7b0a9744babf [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:58.174644 3441 raft_consensus.cc:504] T 72355dd734ea4aeba031171e8c39696d P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:28:58.177546 2089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72355dd734ea4aeba031171e8c39696d" candidate_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "521bb93035324cecbebed2f6f2a630f8"
I20240502 21:28:58.177690 1818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72355dd734ea4aeba031171e8c39696d" candidate_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "153026dd7f8943cc93fedf6f48450fb4"
I20240502 21:28:58.177781 2089 raft_consensus.cc:3037] T 72355dd734ea4aeba031171e8c39696d P 521bb93035324cecbebed2f6f2a630f8 [term 0 FOLLOWER]: Advancing to term 1
W20240502 21:28:58.178262 1349 leader_election.cc:341] T 72355dd734ea4aeba031171e8c39696d P 7d1e5c1b047443cbb28d7b0a9744babf [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132:44947): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:28:58.178884 3441 leader_election.cc:288] T 72355dd734ea4aeba031171e8c39696d P 7d1e5c1b047443cbb28d7b0a9744babf [CANDIDATE]: Term 1 election: Requested vote from peers 521bb93035324cecbebed2f6f2a630f8 (127.15.255.134:40821), 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132:44947), e632c553ff4341328acf1cbaca678392 (127.15.255.133:37439), 6ab8df3c796f4b6cb3e29d5045ad3ea0 (127.15.255.135:34701), a62a935ff43d494ca35d0c417943133d (127.15.255.130:37991), 16b3a337bc5e462f83ba895ad8660deb (127.15.255.131:36033)
I20240502 21:28:58.179128 3487 leader_election.cc:288] T 4dad8773f5b54efa9ff527252bac24aa P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [CANDIDATE]: Term 1 election: Requested vote from peers e632c553ff4341328acf1cbaca678392 (127.15.255.133:37439), 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132:44947), 521bb93035324cecbebed2f6f2a630f8 (127.15.255.134:40821), 7d1e5c1b047443cbb28d7b0a9744babf (127.15.255.129:37897), a62a935ff43d494ca35d0c417943133d (127.15.255.130:37991), 16b3a337bc5e462f83ba895ad8660deb (127.15.255.131:36033)
I20240502 21:28:58.179546 1549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72355dd734ea4aeba031171e8c39696d" candidate_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a62a935ff43d494ca35d0c417943133d"
I20240502 21:28:58.179890 3505 tablet_bootstrap.cc:654] T 72355dd734ea4aeba031171e8c39696d P e632c553ff4341328acf1cbaca678392: Neither blocks nor log segments found. Creating new log.
W20240502 21:28:58.180371 1348 leader_election.cc:341] T 72355dd734ea4aeba031171e8c39696d P 7d1e5c1b047443cbb28d7b0a9744babf [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a62a935ff43d494ca35d0c417943133d (127.15.255.130:37991): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:28:58.180724 1684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72355dd734ea4aeba031171e8c39696d" candidate_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16b3a337bc5e462f83ba895ad8660deb"
I20240502 21:28:58.180779 2089 raft_consensus.cc:2454] T 72355dd734ea4aeba031171e8c39696d P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d1e5c1b047443cbb28d7b0a9744babf in term 1.
I20240502 21:28:58.181010 1684 raft_consensus.cc:3037] T 72355dd734ea4aeba031171e8c39696d P 16b3a337bc5e462f83ba895ad8660deb [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:58.182337 3511 tablet_bootstrap.cc:492] T 4dad8773f5b54efa9ff527252bac24aa P 153026dd7f8943cc93fedf6f48450fb4: No bootstrap required, opened a new log
I20240502 21:28:58.182607 3511 ts_tablet_manager.cc:1367] T 4dad8773f5b54efa9ff527252bac24aa P 153026dd7f8943cc93fedf6f48450fb4: Time spent bootstrapping tablet: real 0.069s user 0.011s sys 0.011s
I20240502 21:28:58.182581 2227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72355dd734ea4aeba031171e8c39696d" candidate_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0"
I20240502 21:28:58.182855 2227 raft_consensus.cc:3037] T 72355dd734ea4aeba031171e8c39696d P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:58.183437 3443 raft_consensus.cc:686] T 4dad8773f5b54efa9ff527252bac24aa P e632c553ff4341328acf1cbaca678392 [term 1 LEADER]: Becoming Leader. State: Replica: e632c553ff4341328acf1cbaca678392, State: Running, Role: LEADER
I20240502 21:28:58.183578 3511 raft_consensus.cc:348] T 4dad8773f5b54efa9ff527252bac24aa P 153026dd7f8943cc93fedf6f48450fb4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:28:58.184031 3511 raft_consensus.cc:374] T 4dad8773f5b54efa9ff527252bac24aa P 153026dd7f8943cc93fedf6f48450fb4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:58.184163 3511 raft_consensus.cc:729] T 4dad8773f5b54efa9ff527252bac24aa P 153026dd7f8943cc93fedf6f48450fb4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 153026dd7f8943cc93fedf6f48450fb4, State: Initialized, Role: FOLLOWER
I20240502 21:28:58.183943 3443 consensus_queue.cc:237] T 4dad8773f5b54efa9ff527252bac24aa P e632c553ff4341328acf1cbaca678392 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:28:58.184962 3511 consensus_queue.cc:260] T 4dad8773f5b54efa9ff527252bac24aa P 153026dd7f8943cc93fedf6f48450fb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:28:58.187395 1684 raft_consensus.cc:2454] T 72355dd734ea4aeba031171e8c39696d P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d1e5c1b047443cbb28d7b0a9744babf in term 1.
I20240502 21:28:58.189210 3511 ts_tablet_manager.cc:1398] T 4dad8773f5b54efa9ff527252bac24aa P 153026dd7f8943cc93fedf6f48450fb4: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20240502 21:28:58.189750 2227 raft_consensus.cc:2454] T 72355dd734ea4aeba031171e8c39696d P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d1e5c1b047443cbb28d7b0a9744babf in term 1.
I20240502 21:28:58.190534 1954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72355dd734ea4aeba031171e8c39696d" candidate_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e632c553ff4341328acf1cbaca678392"
I20240502 21:28:58.190627 1955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72355dd734ea4aeba031171e8c39696d" candidate_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e632c553ff4341328acf1cbaca678392" is_pre_election: true
W20240502 21:28:58.191143 1347 leader_election.cc:341] T 72355dd734ea4aeba031171e8c39696d P 7d1e5c1b047443cbb28d7b0a9744babf [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer e632c553ff4341328acf1cbaca678392 (127.15.255.133:37439): Illegal state: must be running to vote when last-logged opid is not known
W20240502 21:28:58.191705 1347 leader_election.cc:341] T 72355dd734ea4aeba031171e8c39696d P 7d1e5c1b047443cbb28d7b0a9744babf [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e632c553ff4341328acf1cbaca678392 (127.15.255.133:37439): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:28:58.191767 1348 leader_election.cc:302] T 72355dd734ea4aeba031171e8c39696d P 7d1e5c1b047443cbb28d7b0a9744babf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 7 responses out of 7 voters: 4 yes votes; 3 no votes. yes voters: 16b3a337bc5e462f83ba895ad8660deb, 521bb93035324cecbebed2f6f2a630f8, 6ab8df3c796f4b6cb3e29d5045ad3ea0, 7d1e5c1b047443cbb28d7b0a9744babf; no voters: 153026dd7f8943cc93fedf6f48450fb4, a62a935ff43d494ca35d0c417943133d, e632c553ff4341328acf1cbaca678392
I20240502 21:28:58.192132 3441 raft_consensus.cc:2781] T 72355dd734ea4aeba031171e8c39696d P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:28:58.192298 3441 raft_consensus.cc:686] T 72355dd734ea4aeba031171e8c39696d P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 LEADER]: Becoming Leader. State: Replica: 7d1e5c1b047443cbb28d7b0a9744babf, State: Running, Role: LEADER
I20240502 21:28:58.192698 3441 consensus_queue.cc:237] T 72355dd734ea4aeba031171e8c39696d P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:28:58.193486 3512 tablet_bootstrap.cc:492] T 72355dd734ea4aeba031171e8c39696d P a62a935ff43d494ca35d0c417943133d: No bootstrap required, opened a new log
I20240502 21:28:58.193801 3512 ts_tablet_manager.cc:1367] T 72355dd734ea4aeba031171e8c39696d P a62a935ff43d494ca35d0c417943133d: Time spent bootstrapping tablet: real 0.080s user 0.011s sys 0.011s
I20240502 21:28:58.194692 3512 raft_consensus.cc:348] T 72355dd734ea4aeba031171e8c39696d P a62a935ff43d494ca35d0c417943133d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:28:58.195039 3512 raft_consensus.cc:374] T 72355dd734ea4aeba031171e8c39696d P a62a935ff43d494ca35d0c417943133d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:58.195137 3512 raft_consensus.cc:729] T 72355dd734ea4aeba031171e8c39696d P a62a935ff43d494ca35d0c417943133d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a62a935ff43d494ca35d0c417943133d, State: Initialized, Role: FOLLOWER
I20240502 21:28:58.195442 3512 consensus_queue.cc:260] T 72355dd734ea4aeba031171e8c39696d P a62a935ff43d494ca35d0c417943133d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:28:58.196908 3512 ts_tablet_manager.cc:1398] T 72355dd734ea4aeba031171e8c39696d P a62a935ff43d494ca35d0c417943133d: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240502 21:28:58.192411 3511 tablet_bootstrap.cc:492] T 72355dd734ea4aeba031171e8c39696d P 153026dd7f8943cc93fedf6f48450fb4: Bootstrap starting.
I20240502 21:28:58.199398 3511 tablet_bootstrap.cc:654] T 72355dd734ea4aeba031171e8c39696d P 153026dd7f8943cc93fedf6f48450fb4: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:58.199621 3512 tablet_bootstrap.cc:492] T 4dad8773f5b54efa9ff527252bac24aa P a62a935ff43d494ca35d0c417943133d: Bootstrap starting.
I20240502 21:28:58.202800 3512 tablet_bootstrap.cc:654] T 4dad8773f5b54efa9ff527252bac24aa P a62a935ff43d494ca35d0c417943133d: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:58.204248 1267 catalog_manager.cc:5484] T 4dad8773f5b54efa9ff527252bac24aa P e632c553ff4341328acf1cbaca678392 reported cstate change: term changed from 0 to 1, leader changed from <none> to e632c553ff4341328acf1cbaca678392 (127.15.255.133). New cstate: current_term: 1 leader_uuid: "e632c553ff4341328acf1cbaca678392" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } health_report { overall_health: UNKNOWN } } }
I20240502 21:28:58.204116 1263 catalog_manager.cc:5484] T 72355dd734ea4aeba031171e8c39696d P 7d1e5c1b047443cbb28d7b0a9744babf reported cstate change: term changed from 0 to 1, leader changed from <none> to 7d1e5c1b047443cbb28d7b0a9744babf (127.15.255.129). New cstate: current_term: 1 leader_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } health_report { overall_health: UNKNOWN } } }
I20240502 21:28:58.205628 3512 tablet_bootstrap.cc:492] T 4dad8773f5b54efa9ff527252bac24aa P a62a935ff43d494ca35d0c417943133d: No bootstrap required, opened a new log
I20240502 21:28:58.205785 3512 ts_tablet_manager.cc:1367] T 4dad8773f5b54efa9ff527252bac24aa P a62a935ff43d494ca35d0c417943133d: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.003s
I20240502 21:28:58.205991 3505 tablet_bootstrap.cc:492] T 72355dd734ea4aeba031171e8c39696d P e632c553ff4341328acf1cbaca678392: No bootstrap required, opened a new log
I20240502 21:28:58.206138 3505 ts_tablet_manager.cc:1367] T 72355dd734ea4aeba031171e8c39696d P e632c553ff4341328acf1cbaca678392: Time spent bootstrapping tablet: real 0.098s user 0.011s sys 0.012s
I20240502 21:28:58.206537 3512 raft_consensus.cc:348] T 4dad8773f5b54efa9ff527252bac24aa P a62a935ff43d494ca35d0c417943133d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:28:58.206784 3512 raft_consensus.cc:374] T 4dad8773f5b54efa9ff527252bac24aa P a62a935ff43d494ca35d0c417943133d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:58.206872 3512 raft_consensus.cc:729] T 4dad8773f5b54efa9ff527252bac24aa P a62a935ff43d494ca35d0c417943133d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a62a935ff43d494ca35d0c417943133d, State: Initialized, Role: FOLLOWER
I20240502 21:28:58.206904 3505 raft_consensus.cc:348] T 72355dd734ea4aeba031171e8c39696d P e632c553ff4341328acf1cbaca678392 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:28:58.207154 3505 raft_consensus.cc:374] T 72355dd734ea4aeba031171e8c39696d P e632c553ff4341328acf1cbaca678392 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:58.207260 3505 raft_consensus.cc:729] T 72355dd734ea4aeba031171e8c39696d P e632c553ff4341328acf1cbaca678392 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e632c553ff4341328acf1cbaca678392, State: Initialized, Role: FOLLOWER
I20240502 21:28:58.207155 3512 consensus_queue.cc:260] T 4dad8773f5b54efa9ff527252bac24aa P a62a935ff43d494ca35d0c417943133d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:28:58.207525 3505 consensus_queue.cc:260] T 72355dd734ea4aeba031171e8c39696d P e632c553ff4341328acf1cbaca678392 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:28:58.207945 3512 ts_tablet_manager.cc:1398] T 4dad8773f5b54efa9ff527252bac24aa P a62a935ff43d494ca35d0c417943133d: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 21:28:58.208228 3505 ts_tablet_manager.cc:1398] T 72355dd734ea4aeba031171e8c39696d P e632c553ff4341328acf1cbaca678392: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 21:28:58.212831 3511 tablet_bootstrap.cc:492] T 72355dd734ea4aeba031171e8c39696d P 153026dd7f8943cc93fedf6f48450fb4: No bootstrap required, opened a new log
I20240502 21:28:58.213102 3511 ts_tablet_manager.cc:1367] T 72355dd734ea4aeba031171e8c39696d P 153026dd7f8943cc93fedf6f48450fb4: Time spent bootstrapping tablet: real 0.021s user 0.002s sys 0.004s
I20240502 21:28:58.213932 3511 raft_consensus.cc:348] T 72355dd734ea4aeba031171e8c39696d P 153026dd7f8943cc93fedf6f48450fb4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:28:58.214226 3511 raft_consensus.cc:374] T 72355dd734ea4aeba031171e8c39696d P 153026dd7f8943cc93fedf6f48450fb4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:58.214320 3511 raft_consensus.cc:729] T 72355dd734ea4aeba031171e8c39696d P 153026dd7f8943cc93fedf6f48450fb4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 153026dd7f8943cc93fedf6f48450fb4, State: Initialized, Role: FOLLOWER
I20240502 21:28:58.214578 3511 consensus_queue.cc:260] T 72355dd734ea4aeba031171e8c39696d P 153026dd7f8943cc93fedf6f48450fb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:28:58.215209 3511 ts_tablet_manager.cc:1398] T 72355dd734ea4aeba031171e8c39696d P 153026dd7f8943cc93fedf6f48450fb4: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:28:58.224386 3443 raft_consensus.cc:482] T 72355dd734ea4aeba031171e8c39696d P e632c553ff4341328acf1cbaca678392 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:28:58.224592 3443 raft_consensus.cc:504] T 72355dd734ea4aeba031171e8c39696d P e632c553ff4341328acf1cbaca678392 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:28:58.225687 3443 leader_election.cc:288] T 72355dd734ea4aeba031171e8c39696d P e632c553ff4341328acf1cbaca678392 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 521bb93035324cecbebed2f6f2a630f8 (127.15.255.134:40821), 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132:44947), 6ab8df3c796f4b6cb3e29d5045ad3ea0 (127.15.255.135:34701), 7d1e5c1b047443cbb28d7b0a9744babf (127.15.255.129:37897), a62a935ff43d494ca35d0c417943133d (127.15.255.130:37991), 16b3a337bc5e462f83ba895ad8660deb (127.15.255.131:36033)
I20240502 21:28:58.225804 2089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72355dd734ea4aeba031171e8c39696d" candidate_uuid: "e632c553ff4341328acf1cbaca678392" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "521bb93035324cecbebed2f6f2a630f8" is_pre_election: true
I20240502 21:28:58.226073 2089 raft_consensus.cc:2379] T 72355dd734ea4aeba031171e8c39696d P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e632c553ff4341328acf1cbaca678392 in current term 1: Already voted for candidate 7d1e5c1b047443cbb28d7b0a9744babf in this term.
I20240502 21:28:58.226326 1818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72355dd734ea4aeba031171e8c39696d" candidate_uuid: "e632c553ff4341328acf1cbaca678392" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "153026dd7f8943cc93fedf6f48450fb4" is_pre_election: true
I20240502 21:28:58.226317 1549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72355dd734ea4aeba031171e8c39696d" candidate_uuid: "e632c553ff4341328acf1cbaca678392" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a62a935ff43d494ca35d0c417943133d" is_pre_election: true
I20240502 21:28:58.226542 1549 raft_consensus.cc:2454] T 72355dd734ea4aeba031171e8c39696d P a62a935ff43d494ca35d0c417943133d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e632c553ff4341328acf1cbaca678392 in term 0.
I20240502 21:28:58.226550 1818 raft_consensus.cc:2454] T 72355dd734ea4aeba031171e8c39696d P 153026dd7f8943cc93fedf6f48450fb4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e632c553ff4341328acf1cbaca678392 in term 0.
I20240502 21:28:58.226897 2227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72355dd734ea4aeba031171e8c39696d" candidate_uuid: "e632c553ff4341328acf1cbaca678392" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" is_pre_election: true
I20240502 21:28:58.227211 2227 raft_consensus.cc:2379] T 72355dd734ea4aeba031171e8c39696d P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e632c553ff4341328acf1cbaca678392 in current term 1: Already voted for candidate 7d1e5c1b047443cbb28d7b0a9744babf in this term.
I20240502 21:28:58.227566 1415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72355dd734ea4aeba031171e8c39696d" candidate_uuid: "e632c553ff4341328acf1cbaca678392" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" is_pre_election: true
I20240502 21:28:58.228086 1684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72355dd734ea4aeba031171e8c39696d" candidate_uuid: "e632c553ff4341328acf1cbaca678392" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16b3a337bc5e462f83ba895ad8660deb" is_pre_election: true
I20240502 21:28:58.228359 1684 raft_consensus.cc:2379] T 72355dd734ea4aeba031171e8c39696d P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e632c553ff4341328acf1cbaca678392 in current term 1: Already voted for candidate 7d1e5c1b047443cbb28d7b0a9744babf in this term.
I20240502 21:28:58.229107 1885 leader_election.cc:302] T 72355dd734ea4aeba031171e8c39696d P e632c553ff4341328acf1cbaca678392 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 7 responses out of 7 voters: 3 yes votes; 4 no votes. yes voters: 153026dd7f8943cc93fedf6f48450fb4, a62a935ff43d494ca35d0c417943133d, e632c553ff4341328acf1cbaca678392; no voters: 16b3a337bc5e462f83ba895ad8660deb, 521bb93035324cecbebed2f6f2a630f8, 6ab8df3c796f4b6cb3e29d5045ad3ea0, 7d1e5c1b047443cbb28d7b0a9744babf
I20240502 21:28:58.229655 3443 raft_consensus.cc:3037] T 72355dd734ea4aeba031171e8c39696d P e632c553ff4341328acf1cbaca678392 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:58.232270 3443 raft_consensus.cc:2726] T 72355dd734ea4aeba031171e8c39696d P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 21:28:58.498879 3357 raft_consensus.cc:482] T 4dad8773f5b54efa9ff527252bac24aa P 153026dd7f8943cc93fedf6f48450fb4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:28:58.499095 3357 raft_consensus.cc:504] T 4dad8773f5b54efa9ff527252bac24aa P 153026dd7f8943cc93fedf6f48450fb4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:28:58.500341 3357 leader_election.cc:288] T 4dad8773f5b54efa9ff527252bac24aa P 153026dd7f8943cc93fedf6f48450fb4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e632c553ff4341328acf1cbaca678392 (127.15.255.133:37439), 521bb93035324cecbebed2f6f2a630f8 (127.15.255.134:40821), 6ab8df3c796f4b6cb3e29d5045ad3ea0 (127.15.255.135:34701), 7d1e5c1b047443cbb28d7b0a9744babf (127.15.255.129:37897), a62a935ff43d494ca35d0c417943133d (127.15.255.130:37991), 16b3a337bc5e462f83ba895ad8660deb (127.15.255.131:36033)
I20240502 21:28:58.500510 2089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4dad8773f5b54efa9ff527252bac24aa" candidate_uuid: "153026dd7f8943cc93fedf6f48450fb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "521bb93035324cecbebed2f6f2a630f8" is_pre_election: true
I20240502 21:28:58.500779 2089 raft_consensus.cc:2379] T 4dad8773f5b54efa9ff527252bac24aa P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 153026dd7f8943cc93fedf6f48450fb4 in current term 1: Already voted for candidate e632c553ff4341328acf1cbaca678392 in this term.
I20240502 21:28:58.500800 1955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4dad8773f5b54efa9ff527252bac24aa" candidate_uuid: "153026dd7f8943cc93fedf6f48450fb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e632c553ff4341328acf1cbaca678392" is_pre_election: true
I20240502 21:28:58.500946 1415 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4dad8773f5b54efa9ff527252bac24aa" candidate_uuid: "153026dd7f8943cc93fedf6f48450fb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" is_pre_election: true
I20240502 21:28:58.501241 1415 raft_consensus.cc:2379] T 4dad8773f5b54efa9ff527252bac24aa P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 153026dd7f8943cc93fedf6f48450fb4 in current term 1: Already voted for candidate e632c553ff4341328acf1cbaca678392 in this term.
I20240502 21:28:58.501668 2227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4dad8773f5b54efa9ff527252bac24aa" candidate_uuid: "153026dd7f8943cc93fedf6f48450fb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" is_pre_election: true
I20240502 21:28:58.501937 2227 raft_consensus.cc:2379] T 4dad8773f5b54efa9ff527252bac24aa P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 153026dd7f8943cc93fedf6f48450fb4 in current term 1: Already voted for candidate 6ab8df3c796f4b6cb3e29d5045ad3ea0 in this term.
I20240502 21:28:58.501886 1684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4dad8773f5b54efa9ff527252bac24aa" candidate_uuid: "153026dd7f8943cc93fedf6f48450fb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16b3a337bc5e462f83ba895ad8660deb" is_pre_election: true
I20240502 21:28:58.502116 1684 raft_consensus.cc:2379] T 4dad8773f5b54efa9ff527252bac24aa P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 153026dd7f8943cc93fedf6f48450fb4 in current term 1: Already voted for candidate e632c553ff4341328acf1cbaca678392 in this term.
I20240502 21:28:58.502383 1752 leader_election.cc:302] T 4dad8773f5b54efa9ff527252bac24aa P 153026dd7f8943cc93fedf6f48450fb4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 5 responses out of 7 voters: 1 yes votes; 4 no votes. yes voters: 153026dd7f8943cc93fedf6f48450fb4; no voters: 521bb93035324cecbebed2f6f2a630f8, 6ab8df3c796f4b6cb3e29d5045ad3ea0, 7d1e5c1b047443cbb28d7b0a9744babf, e632c553ff4341328acf1cbaca678392
I20240502 21:28:58.502471 1549 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4dad8773f5b54efa9ff527252bac24aa" candidate_uuid: "153026dd7f8943cc93fedf6f48450fb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a62a935ff43d494ca35d0c417943133d" is_pre_election: true
I20240502 21:28:58.502724 3357 raft_consensus.cc:3037] T 4dad8773f5b54efa9ff527252bac24aa P 153026dd7f8943cc93fedf6f48450fb4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:58.502785 1549 raft_consensus.cc:2454] T 4dad8773f5b54efa9ff527252bac24aa P a62a935ff43d494ca35d0c417943133d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 153026dd7f8943cc93fedf6f48450fb4 in term 0.
I20240502 21:28:58.504901 3357 raft_consensus.cc:2726] T 4dad8773f5b54efa9ff527252bac24aa P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 21:28:58.512528 3471 consensus_queue.cc:1035] T e45af71e3fc14c579b32ba4dbe2adf78 P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:28:58.518365 3471 consensus_queue.cc:1035] T e45af71e3fc14c579b32ba4dbe2adf78 P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:28:58.553998 3471 consensus_queue.cc:1035] T e45af71e3fc14c579b32ba4dbe2adf78 P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:28:58.560472 3545 consensus_queue.cc:1035] T e45af71e3fc14c579b32ba4dbe2adf78 P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:28:58.560781 3436 consensus_queue.cc:1035] T e45af71e3fc14c579b32ba4dbe2adf78 P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:28:58.566156 3471 consensus_queue.cc:1035] T e45af71e3fc14c579b32ba4dbe2adf78 P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:28:58.591274 3443 consensus_queue.cc:1035] T 4dad8773f5b54efa9ff527252bac24aa P e632c553ff4341328acf1cbaca678392 [LEADER]: Connected to new peer: Peer: permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:28:58.621347 3441 consensus_queue.cc:1035] T 72355dd734ea4aeba031171e8c39696d P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 21:28:58.634322 3539 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:28:58.634642 3539 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:28:58.642961 3539 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_5'
I20240502 21:28:58.695232 3443 consensus_queue.cc:1035] T 4dad8773f5b54efa9ff527252bac24aa P e632c553ff4341328acf1cbaca678392 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:28:58.708581 3441 consensus_queue.cc:1035] T 72355dd734ea4aeba031171e8c39696d P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:28:58.713476 3441 consensus_queue.cc:1035] T 72355dd734ea4aeba031171e8c39696d P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:28:58.719826 3441 consensus_queue.cc:1035] T 72355dd734ea4aeba031171e8c39696d P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:28:58.726689 1819 raft_consensus.cc:3037] T 72355dd734ea4aeba031171e8c39696d P 153026dd7f8943cc93fedf6f48450fb4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:58.731577 3441 consensus_queue.cc:1035] T 72355dd734ea4aeba031171e8c39696d P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:28:58.747282 1549 raft_consensus.cc:3037] T 72355dd734ea4aeba031171e8c39696d P a62a935ff43d494ca35d0c417943133d [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:58.750890 3122 consensus_queue.cc:1035] T 72355dd734ea4aeba031171e8c39696d P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:28:58.773384 3443 consensus_queue.cc:1035] T 4dad8773f5b54efa9ff527252bac24aa P e632c553ff4341328acf1cbaca678392 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:28:58.777609 3443 consensus_queue.cc:1035] T 4dad8773f5b54efa9ff527252bac24aa P e632c553ff4341328acf1cbaca678392 [LEADER]: Connected to new peer: Peer: permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:28:58.783239 3443 consensus_queue.cc:1035] T 4dad8773f5b54efa9ff527252bac24aa P e632c553ff4341328acf1cbaca678392 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:28:58.784986 1549 raft_consensus.cc:3037] T 4dad8773f5b54efa9ff527252bac24aa P a62a935ff43d494ca35d0c417943133d [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:58.788995 3443 consensus_queue.cc:1035] T 4dad8773f5b54efa9ff527252bac24aa P e632c553ff4341328acf1cbaca678392 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20240502 21:28:58.895931 1886 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.132:44947, user_credentials={real_user=slave}} blocked reactor thread for 70617us
W20240502 21:28:58.903878 1884 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.134:40821, user_credentials={real_user=slave}} blocked reactor thread for 71155.1us
INSERT report
rows total: 2000
time total: 491.405 ms
time per row: 0.245703 ms
I20240502 21:28:59.276484 1267 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44934:
name: "rebalance_test_table_6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 7
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
W20240502 21:28:59.277172 1267 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table rebalance_test_table_6 in case of a server failure: 8 tablet servers would be needed, 7 are available. Consider bringing up more tablet servers.
I20240502 21:28:59.286768 1392 tablet_service.cc:1467] Processing CreateTablet for tablet ef6c97c8da364cae96c34d33b20154fe (DEFAULT_TABLE table=rebalance_test_table_6 [id=552d53f5242c4a2eb8e89b63e8eb6a7a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:59.286990 1664 tablet_service.cc:1467] Processing CreateTablet for tablet ef6c97c8da364cae96c34d33b20154fe (DEFAULT_TABLE table=rebalance_test_table_6 [id=552d53f5242c4a2eb8e89b63e8eb6a7a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:59.287286 1392 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef6c97c8da364cae96c34d33b20154fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:59.287468 1664 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef6c97c8da364cae96c34d33b20154fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:59.287429 1931 tablet_service.cc:1467] Processing CreateTablet for tablet ef6c97c8da364cae96c34d33b20154fe (DEFAULT_TABLE table=rebalance_test_table_6 [id=552d53f5242c4a2eb8e89b63e8eb6a7a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:59.287834 1931 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef6c97c8da364cae96c34d33b20154fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:59.291662 2068 tablet_service.cc:1467] Processing CreateTablet for tablet ef6c97c8da364cae96c34d33b20154fe (DEFAULT_TABLE table=rebalance_test_table_6 [id=552d53f5242c4a2eb8e89b63e8eb6a7a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:59.292065 2068 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef6c97c8da364cae96c34d33b20154fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:59.291957 1800 tablet_service.cc:1467] Processing CreateTablet for tablet ef6c97c8da364cae96c34d33b20154fe (DEFAULT_TABLE table=rebalance_test_table_6 [id=552d53f5242c4a2eb8e89b63e8eb6a7a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:59.292114 1527 tablet_service.cc:1467] Processing CreateTablet for tablet ef6c97c8da364cae96c34d33b20154fe (DEFAULT_TABLE table=rebalance_test_table_6 [id=552d53f5242c4a2eb8e89b63e8eb6a7a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:59.292450 1800 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef6c97c8da364cae96c34d33b20154fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:59.292613 1527 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef6c97c8da364cae96c34d33b20154fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:59.299026 1799 tablet_service.cc:1467] Processing CreateTablet for tablet 4555e51179344acda81670fc9019d968 (DEFAULT_TABLE table=rebalance_test_table_6 [id=552d53f5242c4a2eb8e89b63e8eb6a7a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:59.299482 1799 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4555e51179344acda81670fc9019d968. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:59.301127 2205 tablet_service.cc:1467] Processing CreateTablet for tablet ef6c97c8da364cae96c34d33b20154fe (DEFAULT_TABLE table=rebalance_test_table_6 [id=552d53f5242c4a2eb8e89b63e8eb6a7a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:59.301573 2205 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef6c97c8da364cae96c34d33b20154fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:59.302937 1798 tablet_service.cc:1467] Processing CreateTablet for tablet f8de1fd0869348e6b0aa131b958ce822 (DEFAULT_TABLE table=rebalance_test_table_6 [id=552d53f5242c4a2eb8e89b63e8eb6a7a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:59.303050 1528 tablet_service.cc:1467] Processing CreateTablet for tablet 4555e51179344acda81670fc9019d968 (DEFAULT_TABLE table=rebalance_test_table_6 [id=552d53f5242c4a2eb8e89b63e8eb6a7a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:59.303308 1798 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8de1fd0869348e6b0aa131b958ce822. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:59.303452 1528 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4555e51179344acda81670fc9019d968. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:59.305048 1393 tablet_service.cc:1467] Processing CreateTablet for tablet 4555e51179344acda81670fc9019d968 (DEFAULT_TABLE table=rebalance_test_table_6 [id=552d53f5242c4a2eb8e89b63e8eb6a7a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:59.305475 1393 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4555e51179344acda81670fc9019d968. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:59.309618 2067 tablet_service.cc:1467] Processing CreateTablet for tablet 4555e51179344acda81670fc9019d968 (DEFAULT_TABLE table=rebalance_test_table_6 [id=552d53f5242c4a2eb8e89b63e8eb6a7a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:59.309768 2206 tablet_service.cc:1467] Processing CreateTablet for tablet 4555e51179344acda81670fc9019d968 (DEFAULT_TABLE table=rebalance_test_table_6 [id=552d53f5242c4a2eb8e89b63e8eb6a7a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:59.310087 2067 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4555e51179344acda81670fc9019d968. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:59.310127 2206 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4555e51179344acda81670fc9019d968. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:59.313175 1932 tablet_service.cc:1467] Processing CreateTablet for tablet 4555e51179344acda81670fc9019d968 (DEFAULT_TABLE table=rebalance_test_table_6 [id=552d53f5242c4a2eb8e89b63e8eb6a7a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:59.313585 1932 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4555e51179344acda81670fc9019d968. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:59.314095 2069 tablet_service.cc:1467] Processing CreateTablet for tablet f8de1fd0869348e6b0aa131b958ce822 (DEFAULT_TABLE table=rebalance_test_table_6 [id=552d53f5242c4a2eb8e89b63e8eb6a7a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:59.314460 2069 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8de1fd0869348e6b0aa131b958ce822. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:59.315733 1526 tablet_service.cc:1467] Processing CreateTablet for tablet f8de1fd0869348e6b0aa131b958ce822 (DEFAULT_TABLE table=rebalance_test_table_6 [id=552d53f5242c4a2eb8e89b63e8eb6a7a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:59.316131 1526 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8de1fd0869348e6b0aa131b958ce822. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:59.316871 1933 tablet_service.cc:1467] Processing CreateTablet for tablet f8de1fd0869348e6b0aa131b958ce822 (DEFAULT_TABLE table=rebalance_test_table_6 [id=552d53f5242c4a2eb8e89b63e8eb6a7a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:59.317225 1933 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8de1fd0869348e6b0aa131b958ce822. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:59.320299 1394 tablet_service.cc:1467] Processing CreateTablet for tablet f8de1fd0869348e6b0aa131b958ce822 (DEFAULT_TABLE table=rebalance_test_table_6 [id=552d53f5242c4a2eb8e89b63e8eb6a7a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:59.320955 1394 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8de1fd0869348e6b0aa131b958ce822. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:59.321000 2207 tablet_service.cc:1467] Processing CreateTablet for tablet f8de1fd0869348e6b0aa131b958ce822 (DEFAULT_TABLE table=rebalance_test_table_6 [id=552d53f5242c4a2eb8e89b63e8eb6a7a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:59.321451 2207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8de1fd0869348e6b0aa131b958ce822. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:59.326324 1662 tablet_service.cc:1467] Processing CreateTablet for tablet 4555e51179344acda81670fc9019d968 (DEFAULT_TABLE table=rebalance_test_table_6 [id=552d53f5242c4a2eb8e89b63e8eb6a7a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:59.326762 1662 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4555e51179344acda81670fc9019d968. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:59.330906 1663 tablet_service.cc:1467] Processing CreateTablet for tablet f8de1fd0869348e6b0aa131b958ce822 (DEFAULT_TABLE table=rebalance_test_table_6 [id=552d53f5242c4a2eb8e89b63e8eb6a7a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 21:28:59.331383 1663 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8de1fd0869348e6b0aa131b958ce822. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:28:59.342869 3647 tablet_bootstrap.cc:492] T ef6c97c8da364cae96c34d33b20154fe P 521bb93035324cecbebed2f6f2a630f8: Bootstrap starting.
I20240502 21:28:59.345160 3648 tablet_bootstrap.cc:492] T f8de1fd0869348e6b0aa131b958ce822 P 153026dd7f8943cc93fedf6f48450fb4: Bootstrap starting.
I20240502 21:28:59.346375 3647 tablet_bootstrap.cc:654] T ef6c97c8da364cae96c34d33b20154fe P 521bb93035324cecbebed2f6f2a630f8: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:59.348311 3648 tablet_bootstrap.cc:654] T f8de1fd0869348e6b0aa131b958ce822 P 153026dd7f8943cc93fedf6f48450fb4: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:59.351612 3646 tablet_bootstrap.cc:492] T ef6c97c8da364cae96c34d33b20154fe P e632c553ff4341328acf1cbaca678392: Bootstrap starting.
I20240502 21:28:59.349285 3651 tablet_bootstrap.cc:492] T f8de1fd0869348e6b0aa131b958ce822 P 6ab8df3c796f4b6cb3e29d5045ad3ea0: Bootstrap starting.
I20240502 21:28:59.354635 3651 tablet_bootstrap.cc:654] T f8de1fd0869348e6b0aa131b958ce822 P 6ab8df3c796f4b6cb3e29d5045ad3ea0: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:59.355404 3645 tablet_bootstrap.cc:492] T ef6c97c8da364cae96c34d33b20154fe P 16b3a337bc5e462f83ba895ad8660deb: Bootstrap starting.
I20240502 21:28:59.358469 3646 tablet_bootstrap.cc:654] T ef6c97c8da364cae96c34d33b20154fe P e632c553ff4341328acf1cbaca678392: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:59.360070 3645 tablet_bootstrap.cc:654] T ef6c97c8da364cae96c34d33b20154fe P 16b3a337bc5e462f83ba895ad8660deb: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:59.363404 3648 tablet_bootstrap.cc:492] T f8de1fd0869348e6b0aa131b958ce822 P 153026dd7f8943cc93fedf6f48450fb4: No bootstrap required, opened a new log
I20240502 21:28:59.363601 3648 ts_tablet_manager.cc:1367] T f8de1fd0869348e6b0aa131b958ce822 P 153026dd7f8943cc93fedf6f48450fb4: Time spent bootstrapping tablet: real 0.019s user 0.002s sys 0.003s
I20240502 21:28:59.363903 3650 tablet_bootstrap.cc:492] T f8de1fd0869348e6b0aa131b958ce822 P 7d1e5c1b047443cbb28d7b0a9744babf: Bootstrap starting.
I20240502 21:28:59.364490 3648 raft_consensus.cc:348] T f8de1fd0869348e6b0aa131b958ce822 P 153026dd7f8943cc93fedf6f48450fb4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:28:59.365878 3650 tablet_bootstrap.cc:654] T f8de1fd0869348e6b0aa131b958ce822 P 7d1e5c1b047443cbb28d7b0a9744babf: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:59.366089 3649 tablet_bootstrap.cc:492] T 4555e51179344acda81670fc9019d968 P a62a935ff43d494ca35d0c417943133d: Bootstrap starting.
I20240502 21:28:59.367647 3648 raft_consensus.cc:374] T f8de1fd0869348e6b0aa131b958ce822 P 153026dd7f8943cc93fedf6f48450fb4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:59.368002 3648 raft_consensus.cc:729] T f8de1fd0869348e6b0aa131b958ce822 P 153026dd7f8943cc93fedf6f48450fb4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 153026dd7f8943cc93fedf6f48450fb4, State: Initialized, Role: FOLLOWER
I20240502 21:28:59.369623 3649 tablet_bootstrap.cc:654] T 4555e51179344acda81670fc9019d968 P a62a935ff43d494ca35d0c417943133d: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:59.370157 3648 consensus_queue.cc:260] T f8de1fd0869348e6b0aa131b958ce822 P 153026dd7f8943cc93fedf6f48450fb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:28:59.370986 3647 tablet_bootstrap.cc:492] T ef6c97c8da364cae96c34d33b20154fe P 521bb93035324cecbebed2f6f2a630f8: No bootstrap required, opened a new log
I20240502 21:28:59.371214 3647 ts_tablet_manager.cc:1367] T ef6c97c8da364cae96c34d33b20154fe P 521bb93035324cecbebed2f6f2a630f8: Time spent bootstrapping tablet: real 0.028s user 0.006s sys 0.006s
I20240502 21:28:59.373065 3646 tablet_bootstrap.cc:492] T ef6c97c8da364cae96c34d33b20154fe P e632c553ff4341328acf1cbaca678392: No bootstrap required, opened a new log
I20240502 21:28:59.372136 3647 raft_consensus.cc:348] T ef6c97c8da364cae96c34d33b20154fe P 521bb93035324cecbebed2f6f2a630f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } }
I20240502 21:28:59.374732 3647 raft_consensus.cc:374] T ef6c97c8da364cae96c34d33b20154fe P 521bb93035324cecbebed2f6f2a630f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:59.374890 3647 raft_consensus.cc:729] T ef6c97c8da364cae96c34d33b20154fe P 521bb93035324cecbebed2f6f2a630f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 521bb93035324cecbebed2f6f2a630f8, State: Initialized, Role: FOLLOWER
I20240502 21:28:59.375065 3648 ts_tablet_manager.cc:1398] T f8de1fd0869348e6b0aa131b958ce822 P 153026dd7f8943cc93fedf6f48450fb4: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20240502 21:28:59.375274 3647 consensus_queue.cc:260] T ef6c97c8da364cae96c34d33b20154fe P 521bb93035324cecbebed2f6f2a630f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } }
I20240502 21:28:59.376194 3647 ts_tablet_manager.cc:1398] T ef6c97c8da364cae96c34d33b20154fe P 521bb93035324cecbebed2f6f2a630f8: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20240502 21:28:59.376556 3646 ts_tablet_manager.cc:1367] T ef6c97c8da364cae96c34d33b20154fe P e632c553ff4341328acf1cbaca678392: Time spent bootstrapping tablet: real 0.025s user 0.006s sys 0.000s
I20240502 21:28:59.377419 3648 tablet_bootstrap.cc:492] T 4555e51179344acda81670fc9019d968 P 153026dd7f8943cc93fedf6f48450fb4: Bootstrap starting.
I20240502 21:28:59.377946 3645 tablet_bootstrap.cc:492] T ef6c97c8da364cae96c34d33b20154fe P 16b3a337bc5e462f83ba895ad8660deb: No bootstrap required, opened a new log
I20240502 21:28:59.378182 3645 ts_tablet_manager.cc:1367] T ef6c97c8da364cae96c34d33b20154fe P 16b3a337bc5e462f83ba895ad8660deb: Time spent bootstrapping tablet: real 0.023s user 0.004s sys 0.004s
I20240502 21:28:59.378921 3648 tablet_bootstrap.cc:654] T 4555e51179344acda81670fc9019d968 P 153026dd7f8943cc93fedf6f48450fb4: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:59.379314 3645 raft_consensus.cc:348] T ef6c97c8da364cae96c34d33b20154fe P 16b3a337bc5e462f83ba895ad8660deb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } }
I20240502 21:28:59.379647 3645 raft_consensus.cc:374] T ef6c97c8da364cae96c34d33b20154fe P 16b3a337bc5e462f83ba895ad8660deb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:59.379693 3646 raft_consensus.cc:348] T ef6c97c8da364cae96c34d33b20154fe P e632c553ff4341328acf1cbaca678392 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } }
I20240502 21:28:59.379963 3646 raft_consensus.cc:374] T ef6c97c8da364cae96c34d33b20154fe P e632c553ff4341328acf1cbaca678392 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:59.380072 3646 raft_consensus.cc:729] T ef6c97c8da364cae96c34d33b20154fe P e632c553ff4341328acf1cbaca678392 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e632c553ff4341328acf1cbaca678392, State: Initialized, Role: FOLLOWER
I20240502 21:28:59.380311 3647 tablet_bootstrap.cc:492] T f8de1fd0869348e6b0aa131b958ce822 P 521bb93035324cecbebed2f6f2a630f8: Bootstrap starting.
I20240502 21:28:59.380101 3645 raft_consensus.cc:729] T ef6c97c8da364cae96c34d33b20154fe P 16b3a337bc5e462f83ba895ad8660deb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16b3a337bc5e462f83ba895ad8660deb, State: Initialized, Role: FOLLOWER
I20240502 21:28:59.380295 3646 consensus_queue.cc:260] T ef6c97c8da364cae96c34d33b20154fe P e632c553ff4341328acf1cbaca678392 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } }
I20240502 21:28:59.380722 3645 consensus_queue.cc:260] T ef6c97c8da364cae96c34d33b20154fe P 16b3a337bc5e462f83ba895ad8660deb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } }
I20240502 21:28:59.381099 3646 ts_tablet_manager.cc:1398] T ef6c97c8da364cae96c34d33b20154fe P e632c553ff4341328acf1cbaca678392: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:28:59.382139 3646 tablet_bootstrap.cc:492] T 4555e51179344acda81670fc9019d968 P e632c553ff4341328acf1cbaca678392: Bootstrap starting.
I20240502 21:28:59.382720 3647 tablet_bootstrap.cc:654] T f8de1fd0869348e6b0aa131b958ce822 P 521bb93035324cecbebed2f6f2a630f8: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:59.383292 3651 tablet_bootstrap.cc:492] T f8de1fd0869348e6b0aa131b958ce822 P 6ab8df3c796f4b6cb3e29d5045ad3ea0: No bootstrap required, opened a new log
I20240502 21:28:59.383495 3651 ts_tablet_manager.cc:1367] T f8de1fd0869348e6b0aa131b958ce822 P 6ab8df3c796f4b6cb3e29d5045ad3ea0: Time spent bootstrapping tablet: real 0.034s user 0.006s sys 0.000s
I20240502 21:28:59.384549 3651 raft_consensus.cc:348] T f8de1fd0869348e6b0aa131b958ce822 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:28:59.384851 3649 tablet_bootstrap.cc:492] T 4555e51179344acda81670fc9019d968 P a62a935ff43d494ca35d0c417943133d: No bootstrap required, opened a new log
I20240502 21:28:59.384886 3651 raft_consensus.cc:374] T f8de1fd0869348e6b0aa131b958ce822 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:59.385208 3651 raft_consensus.cc:729] T f8de1fd0869348e6b0aa131b958ce822 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ab8df3c796f4b6cb3e29d5045ad3ea0, State: Initialized, Role: FOLLOWER
I20240502 21:28:59.385357 3649 ts_tablet_manager.cc:1367] T 4555e51179344acda81670fc9019d968 P a62a935ff43d494ca35d0c417943133d: Time spent bootstrapping tablet: real 0.022s user 0.002s sys 0.005s
I20240502 21:28:59.385476 3651 consensus_queue.cc:260] T f8de1fd0869348e6b0aa131b958ce822 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:28:59.386162 3651 ts_tablet_manager.cc:1398] T f8de1fd0869348e6b0aa131b958ce822 P 6ab8df3c796f4b6cb3e29d5045ad3ea0: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 21:28:59.386499 3651 tablet_bootstrap.cc:492] T 4555e51179344acda81670fc9019d968 P 6ab8df3c796f4b6cb3e29d5045ad3ea0: Bootstrap starting.
I20240502 21:28:59.386396 3649 raft_consensus.cc:348] T 4555e51179344acda81670fc9019d968 P a62a935ff43d494ca35d0c417943133d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } }
I20240502 21:28:59.387001 3649 raft_consensus.cc:374] T 4555e51179344acda81670fc9019d968 P a62a935ff43d494ca35d0c417943133d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:59.387152 3649 raft_consensus.cc:729] T 4555e51179344acda81670fc9019d968 P a62a935ff43d494ca35d0c417943133d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a62a935ff43d494ca35d0c417943133d, State: Initialized, Role: FOLLOWER
I20240502 21:28:59.387485 3649 consensus_queue.cc:260] T 4555e51179344acda81670fc9019d968 P a62a935ff43d494ca35d0c417943133d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } }
I20240502 21:28:59.389195 3650 tablet_bootstrap.cc:492] T f8de1fd0869348e6b0aa131b958ce822 P 7d1e5c1b047443cbb28d7b0a9744babf: No bootstrap required, opened a new log
I20240502 21:28:59.389422 3650 ts_tablet_manager.cc:1367] T f8de1fd0869348e6b0aa131b958ce822 P 7d1e5c1b047443cbb28d7b0a9744babf: Time spent bootstrapping tablet: real 0.026s user 0.007s sys 0.000s
I20240502 21:28:59.390156 3647 tablet_bootstrap.cc:492] T f8de1fd0869348e6b0aa131b958ce822 P 521bb93035324cecbebed2f6f2a630f8: No bootstrap required, opened a new log
I20240502 21:28:59.390487 3647 ts_tablet_manager.cc:1367] T f8de1fd0869348e6b0aa131b958ce822 P 521bb93035324cecbebed2f6f2a630f8: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.003s
I20240502 21:28:59.390380 3650 raft_consensus.cc:348] T f8de1fd0869348e6b0aa131b958ce822 P 7d1e5c1b047443cbb28d7b0a9744babf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:28:59.391192 3650 raft_consensus.cc:374] T f8de1fd0869348e6b0aa131b958ce822 P 7d1e5c1b047443cbb28d7b0a9744babf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:59.391368 3650 raft_consensus.cc:729] T f8de1fd0869348e6b0aa131b958ce822 P 7d1e5c1b047443cbb28d7b0a9744babf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d1e5c1b047443cbb28d7b0a9744babf, State: Initialized, Role: FOLLOWER
I20240502 21:28:59.391654 3650 consensus_queue.cc:260] T f8de1fd0869348e6b0aa131b958ce822 P 7d1e5c1b047443cbb28d7b0a9744babf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:28:59.391675 3647 raft_consensus.cc:348] T f8de1fd0869348e6b0aa131b958ce822 P 521bb93035324cecbebed2f6f2a630f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:28:59.391983 3647 raft_consensus.cc:374] T f8de1fd0869348e6b0aa131b958ce822 P 521bb93035324cecbebed2f6f2a630f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:59.392072 3647 raft_consensus.cc:729] T f8de1fd0869348e6b0aa131b958ce822 P 521bb93035324cecbebed2f6f2a630f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 521bb93035324cecbebed2f6f2a630f8, State: Initialized, Role: FOLLOWER
I20240502 21:28:59.392319 3646 tablet_bootstrap.cc:654] T 4555e51179344acda81670fc9019d968 P e632c553ff4341328acf1cbaca678392: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:59.393079 3645 ts_tablet_manager.cc:1398] T ef6c97c8da364cae96c34d33b20154fe P 16b3a337bc5e462f83ba895ad8660deb: Time spent starting tablet: real 0.015s user 0.002s sys 0.003s
I20240502 21:28:59.393177 3651 tablet_bootstrap.cc:654] T 4555e51179344acda81670fc9019d968 P 6ab8df3c796f4b6cb3e29d5045ad3ea0: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:59.393523 3645 tablet_bootstrap.cc:492] T f8de1fd0869348e6b0aa131b958ce822 P 16b3a337bc5e462f83ba895ad8660deb: Bootstrap starting.
I20240502 21:28:59.395210 3645 tablet_bootstrap.cc:654] T f8de1fd0869348e6b0aa131b958ce822 P 16b3a337bc5e462f83ba895ad8660deb: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:59.395960 3648 tablet_bootstrap.cc:492] T 4555e51179344acda81670fc9019d968 P 153026dd7f8943cc93fedf6f48450fb4: No bootstrap required, opened a new log
I20240502 21:28:59.396167 3648 ts_tablet_manager.cc:1367] T 4555e51179344acda81670fc9019d968 P 153026dd7f8943cc93fedf6f48450fb4: Time spent bootstrapping tablet: real 0.019s user 0.002s sys 0.001s
I20240502 21:28:59.396385 3647 consensus_queue.cc:260] T f8de1fd0869348e6b0aa131b958ce822 P 521bb93035324cecbebed2f6f2a630f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:28:59.398102 3647 ts_tablet_manager.cc:1398] T f8de1fd0869348e6b0aa131b958ce822 P 521bb93035324cecbebed2f6f2a630f8: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240502 21:28:59.398524 3647 tablet_bootstrap.cc:492] T 4555e51179344acda81670fc9019d968 P 521bb93035324cecbebed2f6f2a630f8: Bootstrap starting.
I20240502 21:28:59.397667 3648 raft_consensus.cc:348] T 4555e51179344acda81670fc9019d968 P 153026dd7f8943cc93fedf6f48450fb4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } }
I20240502 21:28:59.398732 3648 raft_consensus.cc:374] T 4555e51179344acda81670fc9019d968 P 153026dd7f8943cc93fedf6f48450fb4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:59.398859 3648 raft_consensus.cc:729] T 4555e51179344acda81670fc9019d968 P 153026dd7f8943cc93fedf6f48450fb4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 153026dd7f8943cc93fedf6f48450fb4, State: Initialized, Role: FOLLOWER
I20240502 21:28:59.399166 3648 consensus_queue.cc:260] T 4555e51179344acda81670fc9019d968 P 153026dd7f8943cc93fedf6f48450fb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } }
I20240502 21:28:59.400597 3650 ts_tablet_manager.cc:1398] T f8de1fd0869348e6b0aa131b958ce822 P 7d1e5c1b047443cbb28d7b0a9744babf: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20240502 21:28:59.402050 3650 tablet_bootstrap.cc:492] T ef6c97c8da364cae96c34d33b20154fe P 7d1e5c1b047443cbb28d7b0a9744babf: Bootstrap starting.
I20240502 21:28:59.397037 3649 ts_tablet_manager.cc:1398] T 4555e51179344acda81670fc9019d968 P a62a935ff43d494ca35d0c417943133d: Time spent starting tablet: real 0.012s user 0.000s sys 0.002s
I20240502 21:28:59.402530 3649 tablet_bootstrap.cc:492] T ef6c97c8da364cae96c34d33b20154fe P a62a935ff43d494ca35d0c417943133d: Bootstrap starting.
I20240502 21:28:59.403754 3648 ts_tablet_manager.cc:1398] T 4555e51179344acda81670fc9019d968 P 153026dd7f8943cc93fedf6f48450fb4: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20240502 21:28:59.403774 3647 tablet_bootstrap.cc:654] T 4555e51179344acda81670fc9019d968 P 521bb93035324cecbebed2f6f2a630f8: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:59.404745 3648 tablet_bootstrap.cc:492] T ef6c97c8da364cae96c34d33b20154fe P 153026dd7f8943cc93fedf6f48450fb4: Bootstrap starting.
I20240502 21:28:59.406420 3648 tablet_bootstrap.cc:654] T ef6c97c8da364cae96c34d33b20154fe P 153026dd7f8943cc93fedf6f48450fb4: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:59.407694 3650 tablet_bootstrap.cc:654] T ef6c97c8da364cae96c34d33b20154fe P 7d1e5c1b047443cbb28d7b0a9744babf: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:59.408516 3651 tablet_bootstrap.cc:492] T 4555e51179344acda81670fc9019d968 P 6ab8df3c796f4b6cb3e29d5045ad3ea0: No bootstrap required, opened a new log
I20240502 21:28:59.408739 3651 ts_tablet_manager.cc:1367] T 4555e51179344acda81670fc9019d968 P 6ab8df3c796f4b6cb3e29d5045ad3ea0: Time spent bootstrapping tablet: real 0.022s user 0.000s sys 0.006s
I20240502 21:28:59.409682 3651 raft_consensus.cc:348] T 4555e51179344acda81670fc9019d968 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } }
I20240502 21:28:59.410030 3651 raft_consensus.cc:374] T 4555e51179344acda81670fc9019d968 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:59.410133 3651 raft_consensus.cc:729] T 4555e51179344acda81670fc9019d968 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ab8df3c796f4b6cb3e29d5045ad3ea0, State: Initialized, Role: FOLLOWER
I20240502 21:28:59.410444 3649 tablet_bootstrap.cc:654] T ef6c97c8da364cae96c34d33b20154fe P a62a935ff43d494ca35d0c417943133d: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:59.410455 3651 consensus_queue.cc:260] T 4555e51179344acda81670fc9019d968 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } }
I20240502 21:28:59.411165 3651 ts_tablet_manager.cc:1398] T 4555e51179344acda81670fc9019d968 P 6ab8df3c796f4b6cb3e29d5045ad3ea0: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 21:28:59.414626 3647 tablet_bootstrap.cc:492] T 4555e51179344acda81670fc9019d968 P 521bb93035324cecbebed2f6f2a630f8: No bootstrap required, opened a new log
I20240502 21:28:59.415725 3651 tablet_bootstrap.cc:492] T ef6c97c8da364cae96c34d33b20154fe P 6ab8df3c796f4b6cb3e29d5045ad3ea0: Bootstrap starting.
I20240502 21:28:59.415848 3646 tablet_bootstrap.cc:492] T 4555e51179344acda81670fc9019d968 P e632c553ff4341328acf1cbaca678392: No bootstrap required, opened a new log
I20240502 21:28:59.416085 3646 ts_tablet_manager.cc:1367] T 4555e51179344acda81670fc9019d968 P e632c553ff4341328acf1cbaca678392: Time spent bootstrapping tablet: real 0.034s user 0.010s sys 0.000s
I20240502 21:28:59.416404 3650 tablet_bootstrap.cc:492] T ef6c97c8da364cae96c34d33b20154fe P 7d1e5c1b047443cbb28d7b0a9744babf: No bootstrap required, opened a new log
I20240502 21:28:59.417476 3651 tablet_bootstrap.cc:654] T ef6c97c8da364cae96c34d33b20154fe P 6ab8df3c796f4b6cb3e29d5045ad3ea0: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:59.417357 3646 raft_consensus.cc:348] T 4555e51179344acda81670fc9019d968 P e632c553ff4341328acf1cbaca678392 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } }
I20240502 21:28:59.417773 3646 raft_consensus.cc:374] T 4555e51179344acda81670fc9019d968 P e632c553ff4341328acf1cbaca678392 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:59.417894 3646 raft_consensus.cc:729] T 4555e51179344acda81670fc9019d968 P e632c553ff4341328acf1cbaca678392 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e632c553ff4341328acf1cbaca678392, State: Initialized, Role: FOLLOWER
I20240502 21:28:59.418680 3647 ts_tablet_manager.cc:1367] T 4555e51179344acda81670fc9019d968 P 521bb93035324cecbebed2f6f2a630f8: Time spent bootstrapping tablet: real 0.020s user 0.005s sys 0.000s
I20240502 21:28:59.418939 3649 tablet_bootstrap.cc:492] T ef6c97c8da364cae96c34d33b20154fe P a62a935ff43d494ca35d0c417943133d: No bootstrap required, opened a new log
I20240502 21:28:59.419188 3649 ts_tablet_manager.cc:1367] T ef6c97c8da364cae96c34d33b20154fe P a62a935ff43d494ca35d0c417943133d: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.002s
I20240502 21:28:59.419492 3647 raft_consensus.cc:348] T 4555e51179344acda81670fc9019d968 P 521bb93035324cecbebed2f6f2a630f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } }
I20240502 21:28:59.419788 3647 raft_consensus.cc:374] T 4555e51179344acda81670fc9019d968 P 521bb93035324cecbebed2f6f2a630f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:59.419884 3647 raft_consensus.cc:729] T 4555e51179344acda81670fc9019d968 P 521bb93035324cecbebed2f6f2a630f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 521bb93035324cecbebed2f6f2a630f8, State: Initialized, Role: FOLLOWER
I20240502 21:28:59.420689 3645 tablet_bootstrap.cc:492] T f8de1fd0869348e6b0aa131b958ce822 P 16b3a337bc5e462f83ba895ad8660deb: No bootstrap required, opened a new log
I20240502 21:28:59.420872 3645 ts_tablet_manager.cc:1367] T f8de1fd0869348e6b0aa131b958ce822 P 16b3a337bc5e462f83ba895ad8660deb: Time spent bootstrapping tablet: real 0.027s user 0.001s sys 0.005s
I20240502 21:28:59.418371 3646 consensus_queue.cc:260] T 4555e51179344acda81670fc9019d968 P e632c553ff4341328acf1cbaca678392 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } }
I20240502 21:28:59.421617 3646 ts_tablet_manager.cc:1398] T 4555e51179344acda81670fc9019d968 P e632c553ff4341328acf1cbaca678392: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240502 21:28:59.421710 3645 raft_consensus.cc:348] T f8de1fd0869348e6b0aa131b958ce822 P 16b3a337bc5e462f83ba895ad8660deb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:28:59.419953 3649 raft_consensus.cc:348] T ef6c97c8da364cae96c34d33b20154fe P a62a935ff43d494ca35d0c417943133d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } }
I20240502 21:28:59.422020 3645 raft_consensus.cc:374] T f8de1fd0869348e6b0aa131b958ce822 P 16b3a337bc5e462f83ba895ad8660deb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:59.422086 3649 raft_consensus.cc:374] T ef6c97c8da364cae96c34d33b20154fe P a62a935ff43d494ca35d0c417943133d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:59.422108 3645 raft_consensus.cc:729] T f8de1fd0869348e6b0aa131b958ce822 P 16b3a337bc5e462f83ba895ad8660deb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16b3a337bc5e462f83ba895ad8660deb, State: Initialized, Role: FOLLOWER
I20240502 21:28:59.422168 3649 raft_consensus.cc:729] T ef6c97c8da364cae96c34d33b20154fe P a62a935ff43d494ca35d0c417943133d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a62a935ff43d494ca35d0c417943133d, State: Initialized, Role: FOLLOWER
I20240502 21:28:59.422359 3645 consensus_queue.cc:260] T f8de1fd0869348e6b0aa131b958ce822 P 16b3a337bc5e462f83ba895ad8660deb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:28:59.422413 3649 consensus_queue.cc:260] T ef6c97c8da364cae96c34d33b20154fe P a62a935ff43d494ca35d0c417943133d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } }
I20240502 21:28:59.425884 3645 ts_tablet_manager.cc:1398] T f8de1fd0869348e6b0aa131b958ce822 P 16b3a337bc5e462f83ba895ad8660deb: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20240502 21:28:59.426259 3645 tablet_bootstrap.cc:492] T 4555e51179344acda81670fc9019d968 P 16b3a337bc5e462f83ba895ad8660deb: Bootstrap starting.
I20240502 21:28:59.426460 3649 ts_tablet_manager.cc:1398] T ef6c97c8da364cae96c34d33b20154fe P a62a935ff43d494ca35d0c417943133d: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20240502 21:28:59.428743 3649 tablet_bootstrap.cc:492] T f8de1fd0869348e6b0aa131b958ce822 P a62a935ff43d494ca35d0c417943133d: Bootstrap starting.
I20240502 21:28:59.428743 3650 ts_tablet_manager.cc:1367] T ef6c97c8da364cae96c34d33b20154fe P 7d1e5c1b047443cbb28d7b0a9744babf: Time spent bootstrapping tablet: real 0.027s user 0.004s sys 0.001s
I20240502 21:28:59.429487 3650 raft_consensus.cc:348] T ef6c97c8da364cae96c34d33b20154fe P 7d1e5c1b047443cbb28d7b0a9744babf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } }
I20240502 21:28:59.429847 3650 raft_consensus.cc:374] T ef6c97c8da364cae96c34d33b20154fe P 7d1e5c1b047443cbb28d7b0a9744babf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:59.429937 3650 raft_consensus.cc:729] T ef6c97c8da364cae96c34d33b20154fe P 7d1e5c1b047443cbb28d7b0a9744babf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d1e5c1b047443cbb28d7b0a9744babf, State: Initialized, Role: FOLLOWER
I20240502 21:28:59.430202 3650 consensus_queue.cc:260] T ef6c97c8da364cae96c34d33b20154fe P 7d1e5c1b047443cbb28d7b0a9744babf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } }
I20240502 21:28:59.430522 3646 tablet_bootstrap.cc:492] T f8de1fd0869348e6b0aa131b958ce822 P e632c553ff4341328acf1cbaca678392: Bootstrap starting.
I20240502 21:28:59.431564 3645 tablet_bootstrap.cc:654] T 4555e51179344acda81670fc9019d968 P 16b3a337bc5e462f83ba895ad8660deb: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:59.431564 3649 tablet_bootstrap.cc:654] T f8de1fd0869348e6b0aa131b958ce822 P a62a935ff43d494ca35d0c417943133d: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:59.431715 3648 tablet_bootstrap.cc:492] T ef6c97c8da364cae96c34d33b20154fe P 153026dd7f8943cc93fedf6f48450fb4: No bootstrap required, opened a new log
I20240502 21:28:59.431913 3648 ts_tablet_manager.cc:1367] T ef6c97c8da364cae96c34d33b20154fe P 153026dd7f8943cc93fedf6f48450fb4: Time spent bootstrapping tablet: real 0.027s user 0.012s sys 0.008s
I20240502 21:28:59.431862 3647 consensus_queue.cc:260] T 4555e51179344acda81670fc9019d968 P 521bb93035324cecbebed2f6f2a630f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } }
I20240502 21:28:59.432124 3646 tablet_bootstrap.cc:654] T f8de1fd0869348e6b0aa131b958ce822 P e632c553ff4341328acf1cbaca678392: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:59.432998 3647 ts_tablet_manager.cc:1398] T 4555e51179344acda81670fc9019d968 P 521bb93035324cecbebed2f6f2a630f8: Time spent starting tablet: real 0.014s user 0.002s sys 0.000s
I20240502 21:28:59.433480 3648 raft_consensus.cc:348] T ef6c97c8da364cae96c34d33b20154fe P 153026dd7f8943cc93fedf6f48450fb4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } }
I20240502 21:28:59.433801 3648 raft_consensus.cc:374] T ef6c97c8da364cae96c34d33b20154fe P 153026dd7f8943cc93fedf6f48450fb4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:59.433948 3648 raft_consensus.cc:729] T ef6c97c8da364cae96c34d33b20154fe P 153026dd7f8943cc93fedf6f48450fb4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 153026dd7f8943cc93fedf6f48450fb4, State: Initialized, Role: FOLLOWER
I20240502 21:28:59.434252 3648 consensus_queue.cc:260] T ef6c97c8da364cae96c34d33b20154fe P 153026dd7f8943cc93fedf6f48450fb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } }
I20240502 21:28:59.434921 3650 ts_tablet_manager.cc:1398] T ef6c97c8da364cae96c34d33b20154fe P 7d1e5c1b047443cbb28d7b0a9744babf: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20240502 21:28:59.436094 3648 ts_tablet_manager.cc:1398] T ef6c97c8da364cae96c34d33b20154fe P 153026dd7f8943cc93fedf6f48450fb4: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 21:28:59.436785 3650 tablet_bootstrap.cc:492] T 4555e51179344acda81670fc9019d968 P 7d1e5c1b047443cbb28d7b0a9744babf: Bootstrap starting.
I20240502 21:28:59.437886 3253 raft_consensus.cc:482] T ef6c97c8da364cae96c34d33b20154fe P a62a935ff43d494ca35d0c417943133d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:28:59.438050 3253 raft_consensus.cc:504] T ef6c97c8da364cae96c34d33b20154fe P a62a935ff43d494ca35d0c417943133d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } }
I20240502 21:28:59.439249 3650 tablet_bootstrap.cc:654] T 4555e51179344acda81670fc9019d968 P 7d1e5c1b047443cbb28d7b0a9744babf: Neither blocks nor log segments found. Creating new log.
I20240502 21:28:59.439464 3253 leader_election.cc:288] T ef6c97c8da364cae96c34d33b20154fe P a62a935ff43d494ca35d0c417943133d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7d1e5c1b047443cbb28d7b0a9744babf (127.15.255.129:37897), 16b3a337bc5e462f83ba895ad8660deb (127.15.255.131:36033), 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132:44947), e632c553ff4341328acf1cbaca678392 (127.15.255.133:37439), 521bb93035324cecbebed2f6f2a630f8 (127.15.255.134:40821), 6ab8df3c796f4b6cb3e29d5045ad3ea0 (127.15.255.135:34701)
I20240502 21:28:59.439628 1685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef6c97c8da364cae96c34d33b20154fe" candidate_uuid: "a62a935ff43d494ca35d0c417943133d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16b3a337bc5e462f83ba895ad8660deb" is_pre_election: true
I20240502 21:28:59.439864 1685 raft_consensus.cc:2454] T ef6c97c8da364cae96c34d33b20154fe P 16b3a337bc5e462f83ba895ad8660deb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a62a935ff43d494ca35d0c417943133d in term 0.
I20240502 21:28:59.440238 1954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef6c97c8da364cae96c34d33b20154fe" candidate_uuid: "a62a935ff43d494ca35d0c417943133d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e632c553ff4341328acf1cbaca678392" is_pre_election: true
I20240502 21:28:59.440321 2089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef6c97c8da364cae96c34d33b20154fe" candidate_uuid: "a62a935ff43d494ca35d0c417943133d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "521bb93035324cecbebed2f6f2a630f8" is_pre_election: true
I20240502 21:28:59.440456 1954 raft_consensus.cc:2454] T ef6c97c8da364cae96c34d33b20154fe P e632c553ff4341328acf1cbaca678392 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a62a935ff43d494ca35d0c417943133d in term 0.
I20240502 21:28:59.440469 1414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef6c97c8da364cae96c34d33b20154fe" candidate_uuid: "a62a935ff43d494ca35d0c417943133d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" is_pre_election: true
I20240502 21:28:59.440590 2089 raft_consensus.cc:2454] T ef6c97c8da364cae96c34d33b20154fe P 521bb93035324cecbebed2f6f2a630f8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a62a935ff43d494ca35d0c417943133d in term 0.
I20240502 21:28:59.440647 1414 raft_consensus.cc:2454] T ef6c97c8da364cae96c34d33b20154fe P 7d1e5c1b047443cbb28d7b0a9744babf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a62a935ff43d494ca35d0c417943133d in term 0.
I20240502 21:28:59.440838 2226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef6c97c8da364cae96c34d33b20154fe" candidate_uuid: "a62a935ff43d494ca35d0c417943133d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" is_pre_election: true
W20240502 21:28:59.441427 1481 leader_election.cc:341] T ef6c97c8da364cae96c34d33b20154fe P a62a935ff43d494ca35d0c417943133d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6ab8df3c796f4b6cb3e29d5045ad3ea0 (127.15.255.135:34701): Illegal state: must be running to vote when last-logged opid is not known
I20240502 21:28:59.442198 1480 leader_election.cc:302] T ef6c97c8da364cae96c34d33b20154fe P a62a935ff43d494ca35d0c417943133d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 5 responses out of 7 voters: 4 yes votes; 1 no votes. yes voters: 521bb93035324cecbebed2f6f2a630f8, 7d1e5c1b047443cbb28d7b0a9744babf, a62a935ff43d494ca35d0c417943133d, e632c553ff4341328acf1cbaca678392; no voters: 6ab8df3c796f4b6cb3e29d5045ad3ea0
I20240502 21:28:59.442566 3253 raft_consensus.cc:2781] T ef6c97c8da364cae96c34d33b20154fe P a62a935ff43d494ca35d0c417943133d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:28:59.442673 3253 raft_consensus.cc:482] T ef6c97c8da364cae96c34d33b20154fe P a62a935ff43d494ca35d0c417943133d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:28:59.442756 3253 raft_consensus.cc:3037] T ef6c97c8da364cae96c34d33b20154fe P a62a935ff43d494ca35d0c417943133d [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:59.442890 1818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef6c97c8da364cae96c34d33b20154fe" candidate_uuid: "a62a935ff43d494ca35d0c417943133d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "153026dd7f8943cc93fedf6f48450fb4" is_pre_election: true
I20240502 21:28:59.443116 1818 raft_consensus.cc:2454] T ef6c97c8da364cae96c34d33b20154fe P 153026dd7f8943cc93fedf6f48450fb4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a62a935ff43d494ca35d0c417943133d in term 0.
I20240502 21:28:59.445233 3645 tablet_bootstrap.cc:492] T 4555e51179344acda81670fc9019d968 P 16b3a337bc5e462f83ba895ad8660deb: No bootstrap required, opened a new log
I20240502 21:28:59.445381 3645 ts_tablet_manager.cc:1367] T 4555e51179344acda81670fc9019d968 P 16b3a337bc5e462f83ba895ad8660deb: Time spent bootstrapping tablet: real 0.019s user 0.000s sys 0.004s
I20240502 21:28:59.445433 3253 raft_consensus.cc:504] T ef6c97c8da364cae96c34d33b20154fe P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } }
I20240502 21:28:59.446156 3645 raft_consensus.cc:348] T 4555e51179344acda81670fc9019d968 P 16b3a337bc5e462f83ba895ad8660deb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } }
I20240502 21:28:59.446449 3645 raft_consensus.cc:374] T 4555e51179344acda81670fc9019d968 P 16b3a337bc5e462f83ba895ad8660deb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:59.446548 3645 raft_consensus.cc:729] T 4555e51179344acda81670fc9019d968 P 16b3a337bc5e462f83ba895ad8660deb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16b3a337bc5e462f83ba895ad8660deb, State: Initialized, Role: FOLLOWER
I20240502 21:28:59.446660 3649 tablet_bootstrap.cc:492] T f8de1fd0869348e6b0aa131b958ce822 P a62a935ff43d494ca35d0c417943133d: No bootstrap required, opened a new log
I20240502 21:28:59.446676 1414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef6c97c8da364cae96c34d33b20154fe" candidate_uuid: "a62a935ff43d494ca35d0c417943133d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d1e5c1b047443cbb28d7b0a9744babf"
I20240502 21:28:59.446797 3649 ts_tablet_manager.cc:1367] T f8de1fd0869348e6b0aa131b958ce822 P a62a935ff43d494ca35d0c417943133d: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.004s
I20240502 21:28:59.446854 1414 raft_consensus.cc:3037] T ef6c97c8da364cae96c34d33b20154fe P 7d1e5c1b047443cbb28d7b0a9744babf [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:59.446826 3645 consensus_queue.cc:260] T 4555e51179344acda81670fc9019d968 P 16b3a337bc5e462f83ba895ad8660deb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } }
I20240502 21:28:59.447562 3649 raft_consensus.cc:348] T f8de1fd0869348e6b0aa131b958ce822 P a62a935ff43d494ca35d0c417943133d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:28:59.448264 3645 ts_tablet_manager.cc:1398] T 4555e51179344acda81670fc9019d968 P 16b3a337bc5e462f83ba895ad8660deb: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240502 21:28:59.449635 1414 raft_consensus.cc:2454] T ef6c97c8da364cae96c34d33b20154fe P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a62a935ff43d494ca35d0c417943133d in term 1.
I20240502 21:28:59.450333 3650 tablet_bootstrap.cc:492] T 4555e51179344acda81670fc9019d968 P 7d1e5c1b047443cbb28d7b0a9744babf: No bootstrap required, opened a new log
I20240502 21:28:59.450578 3650 ts_tablet_manager.cc:1367] T 4555e51179344acda81670fc9019d968 P 7d1e5c1b047443cbb28d7b0a9744babf: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.001s
I20240502 21:28:59.447841 3649 raft_consensus.cc:374] T f8de1fd0869348e6b0aa131b958ce822 P a62a935ff43d494ca35d0c417943133d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:59.451148 3649 raft_consensus.cc:729] T f8de1fd0869348e6b0aa131b958ce822 P a62a935ff43d494ca35d0c417943133d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a62a935ff43d494ca35d0c417943133d, State: Initialized, Role: FOLLOWER
I20240502 21:28:59.451421 3649 consensus_queue.cc:260] T f8de1fd0869348e6b0aa131b958ce822 P a62a935ff43d494ca35d0c417943133d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:28:59.451548 3650 raft_consensus.cc:348] T 4555e51179344acda81670fc9019d968 P 7d1e5c1b047443cbb28d7b0a9744babf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } }
I20240502 21:28:59.451995 3650 raft_consensus.cc:374] T 4555e51179344acda81670fc9019d968 P 7d1e5c1b047443cbb28d7b0a9744babf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:59.452147 3650 raft_consensus.cc:729] T 4555e51179344acda81670fc9019d968 P 7d1e5c1b047443cbb28d7b0a9744babf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d1e5c1b047443cbb28d7b0a9744babf, State: Initialized, Role: FOLLOWER
I20240502 21:28:59.452502 3651 tablet_bootstrap.cc:492] T ef6c97c8da364cae96c34d33b20154fe P 6ab8df3c796f4b6cb3e29d5045ad3ea0: No bootstrap required, opened a new log
I20240502 21:28:59.452710 3651 ts_tablet_manager.cc:1367] T ef6c97c8da364cae96c34d33b20154fe P 6ab8df3c796f4b6cb3e29d5045ad3ea0: Time spent bootstrapping tablet: real 0.037s user 0.001s sys 0.006s
I20240502 21:28:59.452502 3650 consensus_queue.cc:260] T 4555e51179344acda81670fc9019d968 P 7d1e5c1b047443cbb28d7b0a9744babf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } }
I20240502 21:28:59.453022 1685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef6c97c8da364cae96c34d33b20154fe" candidate_uuid: "a62a935ff43d494ca35d0c417943133d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16b3a337bc5e462f83ba895ad8660deb"
I20240502 21:28:59.453215 1685 raft_consensus.cc:3037] T ef6c97c8da364cae96c34d33b20154fe P 16b3a337bc5e462f83ba895ad8660deb [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:59.453567 3651 raft_consensus.cc:348] T ef6c97c8da364cae96c34d33b20154fe P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } }
I20240502 21:28:59.454021 3651 raft_consensus.cc:374] T ef6c97c8da364cae96c34d33b20154fe P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:59.454130 3651 raft_consensus.cc:729] T ef6c97c8da364cae96c34d33b20154fe P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ab8df3c796f4b6cb3e29d5045ad3ea0, State: Initialized, Role: FOLLOWER
I20240502 21:28:59.454131 3650 ts_tablet_manager.cc:1398] T 4555e51179344acda81670fc9019d968 P 7d1e5c1b047443cbb28d7b0a9744babf: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 21:28:59.454406 3651 consensus_queue.cc:260] T ef6c97c8da364cae96c34d33b20154fe P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } }
I20240502 21:28:59.454972 3651 ts_tablet_manager.cc:1398] T ef6c97c8da364cae96c34d33b20154fe P 6ab8df3c796f4b6cb3e29d5045ad3ea0: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 21:28:59.456002 1685 raft_consensus.cc:2454] T ef6c97c8da364cae96c34d33b20154fe P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a62a935ff43d494ca35d0c417943133d in term 1.
I20240502 21:28:59.457070 1818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef6c97c8da364cae96c34d33b20154fe" candidate_uuid: "a62a935ff43d494ca35d0c417943133d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "153026dd7f8943cc93fedf6f48450fb4"
I20240502 21:28:59.457109 1954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef6c97c8da364cae96c34d33b20154fe" candidate_uuid: "a62a935ff43d494ca35d0c417943133d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e632c553ff4341328acf1cbaca678392"
I20240502 21:28:59.457295 1818 raft_consensus.cc:3037] T ef6c97c8da364cae96c34d33b20154fe P 153026dd7f8943cc93fedf6f48450fb4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:59.457306 1954 raft_consensus.cc:3037] T ef6c97c8da364cae96c34d33b20154fe P e632c553ff4341328acf1cbaca678392 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:59.457805 3253 leader_election.cc:288] T ef6c97c8da364cae96c34d33b20154fe P a62a935ff43d494ca35d0c417943133d [CANDIDATE]: Term 1 election: Requested vote from peers 7d1e5c1b047443cbb28d7b0a9744babf (127.15.255.129:37897), 16b3a337bc5e462f83ba895ad8660deb (127.15.255.131:36033), 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132:44947), e632c553ff4341328acf1cbaca678392 (127.15.255.133:37439), 521bb93035324cecbebed2f6f2a630f8 (127.15.255.134:40821), 6ab8df3c796f4b6cb3e29d5045ad3ea0 (127.15.255.135:34701)
I20240502 21:28:59.458442 3649 ts_tablet_manager.cc:1398] T f8de1fd0869348e6b0aa131b958ce822 P a62a935ff43d494ca35d0c417943133d: Time spent starting tablet: real 0.012s user 0.001s sys 0.003s
I20240502 21:28:59.459692 1818 raft_consensus.cc:2454] T ef6c97c8da364cae96c34d33b20154fe P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a62a935ff43d494ca35d0c417943133d in term 1.
I20240502 21:28:59.459820 1954 raft_consensus.cc:2454] T ef6c97c8da364cae96c34d33b20154fe P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a62a935ff43d494ca35d0c417943133d in term 1.
I20240502 21:28:59.460247 1480 leader_election.cc:302] T ef6c97c8da364cae96c34d33b20154fe P a62a935ff43d494ca35d0c417943133d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 16b3a337bc5e462f83ba895ad8660deb, 7d1e5c1b047443cbb28d7b0a9744babf, a62a935ff43d494ca35d0c417943133d, e632c553ff4341328acf1cbaca678392; no voters:
I20240502 21:28:59.460314 2089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef6c97c8da364cae96c34d33b20154fe" candidate_uuid: "a62a935ff43d494ca35d0c417943133d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "521bb93035324cecbebed2f6f2a630f8"
I20240502 21:28:59.460391 2226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef6c97c8da364cae96c34d33b20154fe" candidate_uuid: "a62a935ff43d494ca35d0c417943133d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0"
I20240502 21:28:59.460572 2089 raft_consensus.cc:3037] T ef6c97c8da364cae96c34d33b20154fe P 521bb93035324cecbebed2f6f2a630f8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:59.460605 2226 raft_consensus.cc:3037] T ef6c97c8da364cae96c34d33b20154fe P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:59.462842 2089 raft_consensus.cc:2454] T ef6c97c8da364cae96c34d33b20154fe P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a62a935ff43d494ca35d0c417943133d in term 1.
I20240502 21:28:59.462873 2226 raft_consensus.cc:2454] T ef6c97c8da364cae96c34d33b20154fe P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a62a935ff43d494ca35d0c417943133d in term 1.
I20240502 21:28:59.463153 3674 raft_consensus.cc:2781] T ef6c97c8da364cae96c34d33b20154fe P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:28:59.463383 3674 raft_consensus.cc:686] T ef6c97c8da364cae96c34d33b20154fe P a62a935ff43d494ca35d0c417943133d [term 1 LEADER]: Becoming Leader. State: Replica: a62a935ff43d494ca35d0c417943133d, State: Running, Role: LEADER
I20240502 21:28:59.463857 3674 consensus_queue.cc:237] T ef6c97c8da364cae96c34d33b20154fe P a62a935ff43d494ca35d0c417943133d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } }
I20240502 21:28:59.468040 3637 raft_consensus.cc:482] T 4555e51179344acda81670fc9019d968 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:28:59.468227 1270 catalog_manager.cc:5484] T ef6c97c8da364cae96c34d33b20154fe P a62a935ff43d494ca35d0c417943133d reported cstate change: term changed from 0 to 1, leader changed from <none> to a62a935ff43d494ca35d0c417943133d (127.15.255.130). New cstate: current_term: 1 leader_uuid: "a62a935ff43d494ca35d0c417943133d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } health_report { overall_health: UNKNOWN } } }
I20240502 21:28:59.468226 3637 raft_consensus.cc:504] T 4555e51179344acda81670fc9019d968 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } }
I20240502 21:28:59.469920 3637 leader_election.cc:288] T 4555e51179344acda81670fc9019d968 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7d1e5c1b047443cbb28d7b0a9744babf (127.15.255.129:37897), a62a935ff43d494ca35d0c417943133d (127.15.255.130:37991), 16b3a337bc5e462f83ba895ad8660deb (127.15.255.131:36033), 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132:44947), e632c553ff4341328acf1cbaca678392 (127.15.255.133:37439), 521bb93035324cecbebed2f6f2a630f8 (127.15.255.134:40821)
I20240502 21:28:59.470170 1685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4555e51179344acda81670fc9019d968" candidate_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16b3a337bc5e462f83ba895ad8660deb" is_pre_election: true
I20240502 21:28:59.470405 1685 raft_consensus.cc:2454] T 4555e51179344acda81670fc9019d968 P 16b3a337bc5e462f83ba895ad8660deb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ab8df3c796f4b6cb3e29d5045ad3ea0 in term 0.
I20240502 21:28:59.470677 2089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4555e51179344acda81670fc9019d968" candidate_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "521bb93035324cecbebed2f6f2a630f8" is_pre_election: true
I20240502 21:28:59.470865 2089 raft_consensus.cc:2454] T 4555e51179344acda81670fc9019d968 P 521bb93035324cecbebed2f6f2a630f8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ab8df3c796f4b6cb3e29d5045ad3ea0 in term 0.
I20240502 21:28:59.471544 1818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4555e51179344acda81670fc9019d968" candidate_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "153026dd7f8943cc93fedf6f48450fb4" is_pre_election: true
I20240502 21:28:59.471979 1818 raft_consensus.cc:2454] T 4555e51179344acda81670fc9019d968 P 153026dd7f8943cc93fedf6f48450fb4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ab8df3c796f4b6cb3e29d5045ad3ea0 in term 0.
I20240502 21:28:59.471973 1550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4555e51179344acda81670fc9019d968" candidate_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a62a935ff43d494ca35d0c417943133d" is_pre_election: true
I20240502 21:28:59.472193 1550 raft_consensus.cc:2454] T 4555e51179344acda81670fc9019d968 P a62a935ff43d494ca35d0c417943133d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ab8df3c796f4b6cb3e29d5045ad3ea0 in term 0.
I20240502 21:28:59.472285 1414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4555e51179344acda81670fc9019d968" candidate_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" is_pre_election: true
I20240502 21:28:59.472442 2160 leader_election.cc:302] T 4555e51179344acda81670fc9019d968 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 153026dd7f8943cc93fedf6f48450fb4, 16b3a337bc5e462f83ba895ad8660deb, 521bb93035324cecbebed2f6f2a630f8, 6ab8df3c796f4b6cb3e29d5045ad3ea0; no voters:
I20240502 21:28:59.472520 1414 raft_consensus.cc:2454] T 4555e51179344acda81670fc9019d968 P 7d1e5c1b047443cbb28d7b0a9744babf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ab8df3c796f4b6cb3e29d5045ad3ea0 in term 0.
I20240502 21:28:59.472767 3637 raft_consensus.cc:2781] T 4555e51179344acda81670fc9019d968 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:28:59.472892 3637 raft_consensus.cc:482] T 4555e51179344acda81670fc9019d968 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:28:59.472999 3637 raft_consensus.cc:3037] T 4555e51179344acda81670fc9019d968 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:59.472312 1954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4555e51179344acda81670fc9019d968" candidate_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e632c553ff4341328acf1cbaca678392" is_pre_election: true
I20240502 21:28:59.473284 1954 raft_consensus.cc:2454] T 4555e51179344acda81670fc9019d968 P e632c553ff4341328acf1cbaca678392 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6ab8df3c796f4b6cb3e29d5045ad3ea0 in term 0.
I20240502 21:28:59.475327 3637 raft_consensus.cc:504] T 4555e51179344acda81670fc9019d968 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } }
I20240502 21:28:59.476732 1550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4555e51179344acda81670fc9019d968" candidate_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a62a935ff43d494ca35d0c417943133d"
I20240502 21:28:59.476922 1550 raft_consensus.cc:3037] T 4555e51179344acda81670fc9019d968 P a62a935ff43d494ca35d0c417943133d [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:59.476953 1685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4555e51179344acda81670fc9019d968" candidate_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16b3a337bc5e462f83ba895ad8660deb"
I20240502 21:28:59.477047 1820 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4555e51179344acda81670fc9019d968" candidate_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "153026dd7f8943cc93fedf6f48450fb4"
I20240502 21:28:59.477156 1685 raft_consensus.cc:3037] T 4555e51179344acda81670fc9019d968 P 16b3a337bc5e462f83ba895ad8660deb [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:59.477275 1820 raft_consensus.cc:3037] T 4555e51179344acda81670fc9019d968 P 153026dd7f8943cc93fedf6f48450fb4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:59.479179 3646 tablet_bootstrap.cc:492] T f8de1fd0869348e6b0aa131b958ce822 P e632c553ff4341328acf1cbaca678392: No bootstrap required, opened a new log
I20240502 21:28:59.479327 3646 ts_tablet_manager.cc:1367] T f8de1fd0869348e6b0aa131b958ce822 P e632c553ff4341328acf1cbaca678392: Time spent bootstrapping tablet: real 0.057s user 0.005s sys 0.004s
I20240502 21:28:59.479337 1550 raft_consensus.cc:2454] T 4555e51179344acda81670fc9019d968 P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ab8df3c796f4b6cb3e29d5045ad3ea0 in term 1.
I20240502 21:28:59.479882 1820 raft_consensus.cc:2454] T 4555e51179344acda81670fc9019d968 P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ab8df3c796f4b6cb3e29d5045ad3ea0 in term 1.
I20240502 21:28:59.479904 1685 raft_consensus.cc:2454] T 4555e51179344acda81670fc9019d968 P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ab8df3c796f4b6cb3e29d5045ad3ea0 in term 1.
I20240502 21:28:59.480012 1414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4555e51179344acda81670fc9019d968" candidate_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d1e5c1b047443cbb28d7b0a9744babf"
I20240502 21:28:59.480192 1414 raft_consensus.cc:3037] T 4555e51179344acda81670fc9019d968 P 7d1e5c1b047443cbb28d7b0a9744babf [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:59.480363 2161 leader_election.cc:302] T 4555e51179344acda81670fc9019d968 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 153026dd7f8943cc93fedf6f48450fb4, 16b3a337bc5e462f83ba895ad8660deb, 6ab8df3c796f4b6cb3e29d5045ad3ea0, a62a935ff43d494ca35d0c417943133d; no voters:
I20240502 21:28:59.480365 3646 raft_consensus.cc:348] T f8de1fd0869348e6b0aa131b958ce822 P e632c553ff4341328acf1cbaca678392 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:28:59.480626 3646 raft_consensus.cc:374] T f8de1fd0869348e6b0aa131b958ce822 P e632c553ff4341328acf1cbaca678392 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 21:28:59.480732 3487 raft_consensus.cc:2781] T 4555e51179344acda81670fc9019d968 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:28:59.480804 3646 raft_consensus.cc:729] T f8de1fd0869348e6b0aa131b958ce822 P e632c553ff4341328acf1cbaca678392 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e632c553ff4341328acf1cbaca678392, State: Initialized, Role: FOLLOWER
I20240502 21:28:59.481055 3646 consensus_queue.cc:260] T f8de1fd0869348e6b0aa131b958ce822 P e632c553ff4341328acf1cbaca678392 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:28:59.482139 3487 raft_consensus.cc:686] T 4555e51179344acda81670fc9019d968 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 LEADER]: Becoming Leader. State: Replica: 6ab8df3c796f4b6cb3e29d5045ad3ea0, State: Running, Role: LEADER
I20240502 21:28:59.482514 3487 consensus_queue.cc:237] T 4555e51179344acda81670fc9019d968 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } }
I20240502 21:28:59.482658 1414 raft_consensus.cc:2454] T 4555e51179344acda81670fc9019d968 P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ab8df3c796f4b6cb3e29d5045ad3ea0 in term 1.
I20240502 21:28:59.483819 3646 ts_tablet_manager.cc:1398] T f8de1fd0869348e6b0aa131b958ce822 P e632c553ff4341328acf1cbaca678392: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 21:28:59.485085 1954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4555e51179344acda81670fc9019d968" candidate_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e632c553ff4341328acf1cbaca678392"
I20240502 21:28:59.485344 1954 raft_consensus.cc:3037] T 4555e51179344acda81670fc9019d968 P e632c553ff4341328acf1cbaca678392 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:59.485360 2089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4555e51179344acda81670fc9019d968" candidate_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "521bb93035324cecbebed2f6f2a630f8"
I20240502 21:28:59.485512 2089 raft_consensus.cc:3037] T 4555e51179344acda81670fc9019d968 P 521bb93035324cecbebed2f6f2a630f8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:59.487969 1954 raft_consensus.cc:2454] T 4555e51179344acda81670fc9019d968 P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ab8df3c796f4b6cb3e29d5045ad3ea0 in term 1.
I20240502 21:28:59.488039 2089 raft_consensus.cc:2454] T 4555e51179344acda81670fc9019d968 P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6ab8df3c796f4b6cb3e29d5045ad3ea0 in term 1.
I20240502 21:28:59.488740 3637 leader_election.cc:288] T 4555e51179344acda81670fc9019d968 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [CANDIDATE]: Term 1 election: Requested vote from peers 7d1e5c1b047443cbb28d7b0a9744babf (127.15.255.129:37897), a62a935ff43d494ca35d0c417943133d (127.15.255.130:37991), 16b3a337bc5e462f83ba895ad8660deb (127.15.255.131:36033), 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132:44947), e632c553ff4341328acf1cbaca678392 (127.15.255.133:37439), 521bb93035324cecbebed2f6f2a630f8 (127.15.255.134:40821)
I20240502 21:28:59.494411 1270 catalog_manager.cc:5484] T 4555e51179344acda81670fc9019d968 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6ab8df3c796f4b6cb3e29d5045ad3ea0 (127.15.255.135). New cstate: current_term: 1 leader_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } health_report { overall_health: HEALTHY } } }
I20240502 21:28:59.588711 3467 raft_consensus.cc:482] T f8de1fd0869348e6b0aa131b958ce822 P 521bb93035324cecbebed2f6f2a630f8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 21:28:59.588938 3467 raft_consensus.cc:504] T f8de1fd0869348e6b0aa131b958ce822 P 521bb93035324cecbebed2f6f2a630f8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:28:59.590307 1954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8de1fd0869348e6b0aa131b958ce822" candidate_uuid: "521bb93035324cecbebed2f6f2a630f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e632c553ff4341328acf1cbaca678392" is_pre_election: true
I20240502 21:28:59.590523 1954 raft_consensus.cc:2454] T f8de1fd0869348e6b0aa131b958ce822 P e632c553ff4341328acf1cbaca678392 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 521bb93035324cecbebed2f6f2a630f8 in term 0.
I20240502 21:28:59.590458 1820 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8de1fd0869348e6b0aa131b958ce822" candidate_uuid: "521bb93035324cecbebed2f6f2a630f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "153026dd7f8943cc93fedf6f48450fb4" is_pre_election: true
I20240502 21:28:59.590705 1820 raft_consensus.cc:2454] T f8de1fd0869348e6b0aa131b958ce822 P 153026dd7f8943cc93fedf6f48450fb4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 521bb93035324cecbebed2f6f2a630f8 in term 0.
I20240502 21:28:59.590993 3467 leader_election.cc:288] T f8de1fd0869348e6b0aa131b958ce822 P 521bb93035324cecbebed2f6f2a630f8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e632c553ff4341328acf1cbaca678392 (127.15.255.133:37439), 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132:44947), 6ab8df3c796f4b6cb3e29d5045ad3ea0 (127.15.255.135:34701), 7d1e5c1b047443cbb28d7b0a9744babf (127.15.255.129:37897), a62a935ff43d494ca35d0c417943133d (127.15.255.130:37991), 16b3a337bc5e462f83ba895ad8660deb (127.15.255.131:36033)
I20240502 21:28:59.591464 1685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8de1fd0869348e6b0aa131b958ce822" candidate_uuid: "521bb93035324cecbebed2f6f2a630f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16b3a337bc5e462f83ba895ad8660deb" is_pre_election: true
I20240502 21:28:59.591571 1550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8de1fd0869348e6b0aa131b958ce822" candidate_uuid: "521bb93035324cecbebed2f6f2a630f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a62a935ff43d494ca35d0c417943133d" is_pre_election: true
I20240502 21:28:59.591698 1685 raft_consensus.cc:2454] T f8de1fd0869348e6b0aa131b958ce822 P 16b3a337bc5e462f83ba895ad8660deb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 521bb93035324cecbebed2f6f2a630f8 in term 0.
I20240502 21:28:59.591780 1550 raft_consensus.cc:2454] T f8de1fd0869348e6b0aa131b958ce822 P a62a935ff43d494ca35d0c417943133d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 521bb93035324cecbebed2f6f2a630f8 in term 0.
I20240502 21:28:59.591876 2226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8de1fd0869348e6b0aa131b958ce822" candidate_uuid: "521bb93035324cecbebed2f6f2a630f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" is_pre_election: true
I20240502 21:28:59.592121 2226 raft_consensus.cc:2454] T f8de1fd0869348e6b0aa131b958ce822 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 521bb93035324cecbebed2f6f2a630f8 in term 0.
I20240502 21:28:59.592263 2023 leader_election.cc:302] T f8de1fd0869348e6b0aa131b958ce822 P 521bb93035324cecbebed2f6f2a630f8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 153026dd7f8943cc93fedf6f48450fb4, 521bb93035324cecbebed2f6f2a630f8, a62a935ff43d494ca35d0c417943133d, e632c553ff4341328acf1cbaca678392; no voters:
I20240502 21:28:59.592711 3467 raft_consensus.cc:2781] T f8de1fd0869348e6b0aa131b958ce822 P 521bb93035324cecbebed2f6f2a630f8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 21:28:59.592758 1414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8de1fd0869348e6b0aa131b958ce822" candidate_uuid: "521bb93035324cecbebed2f6f2a630f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" is_pre_election: true
I20240502 21:28:59.592852 3467 raft_consensus.cc:482] T f8de1fd0869348e6b0aa131b958ce822 P 521bb93035324cecbebed2f6f2a630f8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 21:28:59.592931 3467 raft_consensus.cc:3037] T f8de1fd0869348e6b0aa131b958ce822 P 521bb93035324cecbebed2f6f2a630f8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:59.592981 1414 raft_consensus.cc:2454] T f8de1fd0869348e6b0aa131b958ce822 P 7d1e5c1b047443cbb28d7b0a9744babf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 521bb93035324cecbebed2f6f2a630f8 in term 0.
I20240502 21:28:59.595134 3467 raft_consensus.cc:504] T f8de1fd0869348e6b0aa131b958ce822 P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:28:59.596208 2226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8de1fd0869348e6b0aa131b958ce822" candidate_uuid: "521bb93035324cecbebed2f6f2a630f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0"
I20240502 21:28:59.596385 2226 raft_consensus.cc:3037] T f8de1fd0869348e6b0aa131b958ce822 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:59.596356 1820 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8de1fd0869348e6b0aa131b958ce822" candidate_uuid: "521bb93035324cecbebed2f6f2a630f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "153026dd7f8943cc93fedf6f48450fb4"
I20240502 21:28:59.596577 1820 raft_consensus.cc:3037] T f8de1fd0869348e6b0aa131b958ce822 P 153026dd7f8943cc93fedf6f48450fb4 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:59.597257 1550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8de1fd0869348e6b0aa131b958ce822" candidate_uuid: "521bb93035324cecbebed2f6f2a630f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a62a935ff43d494ca35d0c417943133d"
I20240502 21:28:59.597425 1550 raft_consensus.cc:3037] T f8de1fd0869348e6b0aa131b958ce822 P a62a935ff43d494ca35d0c417943133d [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:59.599077 2226 raft_consensus.cc:2454] T f8de1fd0869348e6b0aa131b958ce822 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 521bb93035324cecbebed2f6f2a630f8 in term 1.
I20240502 21:28:59.599216 1820 raft_consensus.cc:2454] T f8de1fd0869348e6b0aa131b958ce822 P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 521bb93035324cecbebed2f6f2a630f8 in term 1.
I20240502 21:28:59.599946 1550 raft_consensus.cc:2454] T f8de1fd0869348e6b0aa131b958ce822 P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 521bb93035324cecbebed2f6f2a630f8 in term 1.
I20240502 21:28:59.600132 1954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8de1fd0869348e6b0aa131b958ce822" candidate_uuid: "521bb93035324cecbebed2f6f2a630f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e632c553ff4341328acf1cbaca678392"
I20240502 21:28:59.600320 1954 raft_consensus.cc:3037] T f8de1fd0869348e6b0aa131b958ce822 P e632c553ff4341328acf1cbaca678392 [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:59.600379 2023 leader_election.cc:302] T f8de1fd0869348e6b0aa131b958ce822 P 521bb93035324cecbebed2f6f2a630f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 7 voters: 4 yes votes; 0 no votes. yes voters: 153026dd7f8943cc93fedf6f48450fb4, 521bb93035324cecbebed2f6f2a630f8, 6ab8df3c796f4b6cb3e29d5045ad3ea0, a62a935ff43d494ca35d0c417943133d; no voters:
I20240502 21:28:59.600677 3638 raft_consensus.cc:2781] T f8de1fd0869348e6b0aa131b958ce822 P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 21:28:59.600788 3638 raft_consensus.cc:686] T f8de1fd0869348e6b0aa131b958ce822 P 521bb93035324cecbebed2f6f2a630f8 [term 1 LEADER]: Becoming Leader. State: Replica: 521bb93035324cecbebed2f6f2a630f8, State: Running, Role: LEADER
I20240502 21:28:59.601222 1414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8de1fd0869348e6b0aa131b958ce822" candidate_uuid: "521bb93035324cecbebed2f6f2a630f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d1e5c1b047443cbb28d7b0a9744babf"
I20240502 21:28:59.601151 3638 consensus_queue.cc:237] T f8de1fd0869348e6b0aa131b958ce822 P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:28:59.601426 1414 raft_consensus.cc:3037] T f8de1fd0869348e6b0aa131b958ce822 P 7d1e5c1b047443cbb28d7b0a9744babf [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:59.603440 1954 raft_consensus.cc:2454] T f8de1fd0869348e6b0aa131b958ce822 P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 521bb93035324cecbebed2f6f2a630f8 in term 1.
I20240502 21:28:59.604460 1414 raft_consensus.cc:2454] T f8de1fd0869348e6b0aa131b958ce822 P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 521bb93035324cecbebed2f6f2a630f8 in term 1.
I20240502 21:28:59.605782 3467 leader_election.cc:288] T f8de1fd0869348e6b0aa131b958ce822 P 521bb93035324cecbebed2f6f2a630f8 [CANDIDATE]: Term 1 election: Requested vote from peers e632c553ff4341328acf1cbaca678392 (127.15.255.133:37439), 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132:44947), 6ab8df3c796f4b6cb3e29d5045ad3ea0 (127.15.255.135:34701), 7d1e5c1b047443cbb28d7b0a9744babf (127.15.255.129:37897), a62a935ff43d494ca35d0c417943133d (127.15.255.130:37991), 16b3a337bc5e462f83ba895ad8660deb (127.15.255.131:36033)
I20240502 21:28:59.606472 1685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8de1fd0869348e6b0aa131b958ce822" candidate_uuid: "521bb93035324cecbebed2f6f2a630f8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16b3a337bc5e462f83ba895ad8660deb"
I20240502 21:28:59.606689 1685 raft_consensus.cc:3037] T f8de1fd0869348e6b0aa131b958ce822 P 16b3a337bc5e462f83ba895ad8660deb [term 0 FOLLOWER]: Advancing to term 1
I20240502 21:28:59.606411 1270 catalog_manager.cc:5484] T f8de1fd0869348e6b0aa131b958ce822 P 521bb93035324cecbebed2f6f2a630f8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 521bb93035324cecbebed2f6f2a630f8 (127.15.255.134). New cstate: current_term: 1 leader_uuid: "521bb93035324cecbebed2f6f2a630f8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } health_report { overall_health: UNKNOWN } } }
I20240502 21:28:59.609187 1685 raft_consensus.cc:2454] T f8de1fd0869348e6b0aa131b958ce822 P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 521bb93035324cecbebed2f6f2a630f8 in term 1.
I20240502 21:28:59.893280 3674 consensus_queue.cc:1035] T ef6c97c8da364cae96c34d33b20154fe P a62a935ff43d494ca35d0c417943133d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:28:59.911799 3487 consensus_queue.cc:1035] T 4555e51179344acda81670fc9019d968 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:28:59.922457 3674 consensus_queue.cc:1035] T ef6c97c8da364cae96c34d33b20154fe P a62a935ff43d494ca35d0c417943133d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 21:28:59.983261 3679 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:28:59.983633 3679 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240502 21:28:59.984706 3674 consensus_queue.cc:1035] T ef6c97c8da364cae96c34d33b20154fe P a62a935ff43d494ca35d0c417943133d [LEADER]: Connected to new peer: Peer: permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 21:28:59.994252 3679 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 21:28:59.994536 3674 consensus_queue.cc:1035] T ef6c97c8da364cae96c34d33b20154fe P a62a935ff43d494ca35d0c417943133d [LEADER]: Connected to new peer: Peer: permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:29:00.002750 3674 consensus_queue.cc:1035] T ef6c97c8da364cae96c34d33b20154fe P a62a935ff43d494ca35d0c417943133d [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:29:00.009397 3545 consensus_queue.cc:1035] T f8de1fd0869348e6b0aa131b958ce822 P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:29:00.018550 3697 consensus_queue.cc:1035] T ef6c97c8da364cae96c34d33b20154fe P a62a935ff43d494ca35d0c417943133d [LEADER]: Connected to new peer: Peer: permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240502 21:29:00.019402 3487 consensus_queue.cc:1035] T 4555e51179344acda81670fc9019d968 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:29:00.020041 3487 consensus_queue.cc:1035] T 4555e51179344acda81670fc9019d968 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:29:00.021698 3487 consensus_queue.cc:1035] T 4555e51179344acda81670fc9019d968 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:29:00.027395 3487 consensus_queue.cc:1035] T 4555e51179344acda81670fc9019d968 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:29:00.073263 3545 consensus_queue.cc:1035] T f8de1fd0869348e6b0aa131b958ce822 P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:29:00.079648 3713 consensus_queue.cc:1035] T 4555e51179344acda81670fc9019d968 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:29:00.122959 3545 consensus_queue.cc:1035] T f8de1fd0869348e6b0aa131b958ce822 P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_6'
I20240502 21:29:00.131863 3638 consensus_queue.cc:1035] T f8de1fd0869348e6b0aa131b958ce822 P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:29:00.132151 3545 consensus_queue.cc:1035] T f8de1fd0869348e6b0aa131b958ce822 P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 21:29:00.147675 3467 consensus_queue.cc:1035] T f8de1fd0869348e6b0aa131b958ce822 P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 21:29:00.436311 3733 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.15.255.134:40821, user_credentials={real_user=slave}} blocked reactor thread for 54617.5us
INSERT report
rows total: 2000
time total: 484.035 ms
time per row: 0.242018 ms
I20240502 21:29:00.750413 16382 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvNciQZ/build/asan/bin/kudu
/tmp/dist-test-taskvNciQZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.255.136:41459
--local_ip_for_outbound_sockets=127.15.255.136
--tserver_master_addrs=127.15.255.190:43173
--webserver_port=38591
--webserver_interface=127.15.255.136
--builtin_ntp_servers=127.15.255.148:41767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240502 21:29:01.094085 3772 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:29:01.094345 3772 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:29:01.094468 3772 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:29:01.102015 3772 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 21:29:01.102138 3772 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:29:01.102280 3772 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.255.136
I20240502 21:29:01.112195 3772 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.255.148:41767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.255.136:41459
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.15.255.136
--webserver_port=38591
--tserver_master_addrs=127.15.255.190:43173
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.255.136
--log_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:29:01.113366 3772 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:29:01.114919 3772 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:29:01.126809 3780 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:29:01.127563 3777 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:29:01.127604 3778 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:29:01.128085 3772 server_base.cc:1034] running on GCE node
I20240502 21:29:01.128644 3772 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:29:01.130347 3772 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:29:01.131536 3772 hybrid_clock.cc:648] HybridClock initialized: now 1714685341131489 us; error 49 us; skew 500 ppm
I20240502 21:29:01.131806 3772 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:29:01.134267 3772 webserver.cc:460] Webserver started at http://127.15.255.136:38591/ using document root <none> and password file <none>
I20240502 21:29:01.135002 3772 fs_manager.cc:359] Metadata directory not provided
I20240502 21:29:01.135128 3772 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:29:01.141121 3772 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20240502 21:29:01.144260 3786 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:29:01.145812 3772 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240502 21:29:01.145992 3772 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "d8d7297186694f77bb98de9174e4279c"
format_stamp: "Formatted at 2024-05-02 21:28:43 on dist-test-slave-jdfx"
I20240502 21:29:01.146914 3772 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:29:01.167868 3772 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:29:01.168659 3772 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:29:01.168893 3772 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:29:01.169778 3772 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:29:01.171272 3772 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:29:01.171341 3772 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:29:01.171430 3772 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:29:01.171471 3772 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 21:29:01.201023 3772 rpc_server.cc:307] RPC server started. Bound to: 127.15.255.136:41459
I20240502 21:29:01.201100 3901 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.255.136:41459 every 8 connection(s)
I20240502 21:29:01.202454 3772 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240502 21:29:01.211843 16382 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvNciQZ/build/asan/bin/kudu as pid 3772
I20240502 21:29:01.212409 16382 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvNciQZ/build/asan/bin/kudu
/tmp/dist-test-taskvNciQZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-8/wal
--fs_data_dirs=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-8/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-8/logs
--server_dump_info_path=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-8/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.255.137:43717
--local_ip_for_outbound_sockets=127.15.255.137
--tserver_master_addrs=127.15.255.190:43173
--webserver_port=46781
--webserver_interface=127.15.255.137
--builtin_ntp_servers=127.15.255.148:41767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 21:29:01.220201 3902 heartbeater.cc:347] Connected to a master server at 127.15.255.190:43173
I20240502 21:29:01.220602 3902 heartbeater.cc:464] Registering TS with master...
I20240502 21:29:01.221707 3902 heartbeater.cc:511] Master 127.15.255.190:43173 requested a full tablet report, sending...
I20240502 21:29:01.226994 1263 ts_manager.cc:194] Re-registered known tserver with Master: d8d7297186694f77bb98de9174e4279c (127.15.255.136:41459)
I20240502 21:29:01.228291 1263 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.255.136:44855
W20240502 21:29:01.575662 3906 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:29:01.575978 3906 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:29:01.576112 3906 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:29:01.584378 3906 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 21:29:01.584553 3906 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:29:01.584722 3906 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.255.137
I20240502 21:29:01.594657 3906 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.255.148:41767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-8/data
--fs_wal_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-8/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.255.137:43717
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-8/data/info.pb
--webserver_interface=127.15.255.137
--webserver_port=46781
--tserver_master_addrs=127.15.255.190:43173
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.255.137
--log_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-8/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:29:01.595978 3906 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:29:01.597766 3906 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:29:01.609540 3914 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:29:01.609891 3911 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:29:01.610278 3912 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:29:01.611021 3906 server_base.cc:1034] running on GCE node
I20240502 21:29:01.611588 3906 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:29:01.612562 3906 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:29:01.613754 3906 hybrid_clock.cc:648] HybridClock initialized: now 1714685341613712 us; error 38 us; skew 500 ppm
I20240502 21:29:01.614066 3906 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:29:01.616206 3906 webserver.cc:460] Webserver started at http://127.15.255.137:46781/ using document root <none> and password file <none>
I20240502 21:29:01.616947 3906 fs_manager.cc:359] Metadata directory not provided
I20240502 21:29:01.617079 3906 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:29:01.622759 3906 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.001s
I20240502 21:29:01.625587 3920 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:29:01.626936 3906 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20240502 21:29:01.627094 3906 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-8/data,/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-8/wal
uuid: "16a97c1394f346f0b987f97cce7371b6"
format_stamp: "Formatted at 2024-05-02 21:28:44 on dist-test-slave-jdfx"
I20240502 21:29:01.627914 3906 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-8/wal
metadata directory: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-8/wal
1 data directories: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-8/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:29:01.659945 3906 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:29:01.660828 3906 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:29:01.661096 3906 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:29:01.662132 3906 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:29:01.663779 3906 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:29:01.663870 3906 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:29:01.663995 3906 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:29:01.664050 3906 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 21:29:01.694995 3906 rpc_server.cc:307] RPC server started. Bound to: 127.15.255.137:43717
I20240502 21:29:01.695116 4033 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.255.137:43717 every 8 connection(s)
I20240502 21:29:01.696720 3906 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-8/data/info.pb
I20240502 21:29:01.702068 16382 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvNciQZ/build/asan/bin/kudu as pid 3906
I20240502 21:29:01.702661 16382 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskvNciQZ/build/asan/bin/kudu
/tmp/dist-test-taskvNciQZ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-9/wal
--fs_data_dirs=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-9/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-9/logs
--server_dump_info_path=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-9/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.15.255.138:33375
--local_ip_for_outbound_sockets=127.15.255.138
--tserver_master_addrs=127.15.255.190:43173
--webserver_port=40023
--webserver_interface=127.15.255.138
--builtin_ntp_servers=127.15.255.148:41767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 21:29:01.709352 4034 heartbeater.cc:347] Connected to a master server at 127.15.255.190:43173
I20240502 21:29:01.709667 4034 heartbeater.cc:464] Registering TS with master...
I20240502 21:29:01.710629 4034 heartbeater.cc:511] Master 127.15.255.190:43173 requested a full tablet report, sending...
I20240502 21:29:01.712074 1263 ts_manager.cc:194] Re-registered known tserver with Master: 16a97c1394f346f0b987f97cce7371b6 (127.15.255.137:43717)
I20240502 21:29:01.712905 1263 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.255.137:44731
W20240502 21:29:02.076201 4037 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 21:29:02.076527 4037 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 21:29:02.076663 4037 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 21:29:02.084360 4037 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 21:29:02.084509 4037 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 21:29:02.084637 4037 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.15.255.138
I20240502 21:29:02.094429 4037 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.15.255.148:41767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-9/data
--fs_wal_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-9/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.15.255.138:33375
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-9/data/info.pb
--webserver_interface=127.15.255.138
--webserver_port=40023
--tserver_master_addrs=127.15.255.190:43173
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.15.255.138
--log_dir=/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-9/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision fdb42489d76098eeca371ac57ae02ea176438c64
build type FASTDEBUG
built by None at 02 May 2024 20:51:32 UTC on 1a15035a6830
build id 95
ASAN enabled
I20240502 21:29:02.095782 4037 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 21:29:02.097748 4037 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 21:29:02.109439 4043 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:29:02.109969 4044 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 21:29:02.110731 4047 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 21:29:02.111399 4037 server_base.cc:1034] running on GCE node
I20240502 21:29:02.111999 4037 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 21:29:02.112919 4037 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 21:29:02.114126 4037 hybrid_clock.cc:648] HybridClock initialized: now 1714685342114076 us; error 46 us; skew 500 ppm
I20240502 21:29:02.114434 4037 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 21:29:02.116546 4037 webserver.cc:460] Webserver started at http://127.15.255.138:40023/ using document root <none> and password file <none>
I20240502 21:29:02.117256 4037 fs_manager.cc:359] Metadata directory not provided
I20240502 21:29:02.117379 4037 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 21:29:02.122582 4037 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.003s
I20240502 21:29:02.125351 4053 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 21:29:02.126613 4037 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20240502 21:29:02.126761 4037 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-9/data,/tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-9/wal
uuid: "1fab5794cf1740288abd82f2cebb9b1a"
format_stamp: "Formatted at 2024-05-02 21:28:44 on dist-test-slave-jdfx"
I20240502 21:29:02.127523 4037 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-9/wal
metadata directory: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-9/wal
1 data directories: /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-9/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 21:29:02.165954 4037 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 21:29:02.167032 4037 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 21:29:02.167316 4037 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 21:29:02.168239 4037 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 21:29:02.169971 4037 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 21:29:02.170049 4037 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 21:29:02.170171 4037 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 21:29:02.170238 4037 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 21:29:02.213511 4037 rpc_server.cc:307] RPC server started. Bound to: 127.15.255.138:33375
I20240502 21:29:02.213567 4165 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.15.255.138:33375 every 8 connection(s)
I20240502 21:29:02.215185 4037 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-9/data/info.pb
I20240502 21:29:02.219663 16382 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskvNciQZ/build/asan/bin/kudu as pid 4037
I20240502 21:29:02.231160 3902 heartbeater.cc:503] Master 127.15.255.190:43173 was elected leader, sending a full tablet report...
I20240502 21:29:02.235747 4166 heartbeater.cc:347] Connected to a master server at 127.15.255.190:43173
I20240502 21:29:02.236073 4166 heartbeater.cc:464] Registering TS with master...
I20240502 21:29:02.237143 4166 heartbeater.cc:511] Master 127.15.255.190:43173 requested a full tablet report, sending...
I20240502 21:29:02.238672 1263 ts_manager.cc:194] Re-registered known tserver with Master: 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138:33375)
I20240502 21:29:02.239605 1263 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.15.255.138:52849
I20240502 21:29:02.715626 4034 heartbeater.cc:503] Master 127.15.255.190:43173 was elected leader, sending a full tablet report...
I20240502 21:29:02.734972 3968 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:02.749989 1798 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:29:02.774322 1664 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:02.794232 4101 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:02.808925 2068 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 21:29:02.837805 2205 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:29:02.839236 1528 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:29:02.878110 3836 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:02.888844 1931 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:29:02.898967 1394 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:29:03.131284 1798 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:29:03.140278 3968 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:03.141721 1664 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:03.199079 4101 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:03.206094 2068 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 21:29:03.239394 2205 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:29:03.242300 4166 heartbeater.cc:503] Master 127.15.255.190:43173 was elected leader, sending a full tablet report...
I20240502 21:29:03.248263 3836 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:03.267966 1931 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:29:03.274596 1394 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:29:03.285305 1528 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:29:03.526849 1798 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:29:03.540546 4101 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:03.579068 2068 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 21:29:03.581238 2205 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:29:03.584502 1664 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:03.584731 3968 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:03.592707 1394 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:29:03.612150 1528 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:29:03.637125 3836 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:03.648218 1931 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:29:03.792603 1414 consensus_queue.cc:237] T 72355dd734ea4aeba031171e8c39696d P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } attrs { replace: true } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:29:03.795778 1954 raft_consensus.cc:1261] T 72355dd734ea4aeba031171e8c39696d P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:29:03.795795 1549 raft_consensus.cc:1261] T 72355dd734ea4aeba031171e8c39696d P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:29:03.796401 3602 consensus_queue.cc:1035] T 72355dd734ea4aeba031171e8c39696d P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:03.796975 1685 raft_consensus.cc:1261] T 72355dd734ea4aeba031171e8c39696d P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:29:03.797492 2226 raft_consensus.cc:1261] T 72355dd734ea4aeba031171e8c39696d P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:29:03.798194 3602 consensus_queue.cc:1035] T 72355dd734ea4aeba031171e8c39696d P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:03.798599 4270 consensus_queue.cc:1035] T 72355dd734ea4aeba031171e8c39696d P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:03.799219 4170 consensus_queue.cc:1035] T 72355dd734ea4aeba031171e8c39696d P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:03.805928 1819 raft_consensus.cc:1261] T 72355dd734ea4aeba031171e8c39696d P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:29:03.806178 2089 raft_consensus.cc:1261] T 72355dd734ea4aeba031171e8c39696d P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:29:03.806632 4270 consensus_queue.cc:1035] T 72355dd734ea4aeba031171e8c39696d P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:03.806890 3602 consensus_queue.cc:1035] T 72355dd734ea4aeba031171e8c39696d P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:03.819955 3602 raft_consensus.cc:2932] T 72355dd734ea4aeba031171e8c39696d P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } attrs { replace: true } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } }
I20240502 21:29:03.827152 1549 raft_consensus.cc:2932] T 72355dd734ea4aeba031171e8c39696d P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } attrs { replace: true } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } }
I20240502 21:29:03.827675 1685 raft_consensus.cc:2932] T 72355dd734ea4aeba031171e8c39696d P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } attrs { replace: true } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } }
I20240502 21:29:03.844069 1268 catalog_manager.cc:5484] T 72355dd734ea4aeba031171e8c39696d P 7d1e5c1b047443cbb28d7b0a9744babf reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } health_report { overall_health: HEALTHY } } }
I20240502 21:29:03.847234 1818 raft_consensus.cc:2932] T 72355dd734ea4aeba031171e8c39696d P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } attrs { replace: true } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } }
I20240502 21:29:03.852617 2089 raft_consensus.cc:2932] T 72355dd734ea4aeba031171e8c39696d P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } attrs { replace: true } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } }
I20240502 21:29:03.876725 2225 raft_consensus.cc:2932] T 72355dd734ea4aeba031171e8c39696d P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } attrs { replace: true } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } }
I20240502 21:29:03.884733 1955 raft_consensus.cc:2932] T 72355dd734ea4aeba031171e8c39696d P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } attrs { replace: true } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } }
I20240502 21:29:03.893867 1414 consensus_queue.cc:237] T 72355dd734ea4aeba031171e8c39696d P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } attrs { replace: true } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: true } }
I20240502 21:29:03.897229 2089 raft_consensus.cc:1261] T 72355dd734ea4aeba031171e8c39696d P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:29:03.897927 3602 consensus_queue.cc:1035] T 72355dd734ea4aeba031171e8c39696d P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:29:03.901096 1955 raft_consensus.cc:1261] T 72355dd734ea4aeba031171e8c39696d P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:29:03.902194 4320 consensus_queue.cc:1035] T 72355dd734ea4aeba031171e8c39696d P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:29:03.903494 2225 raft_consensus.cc:1261] T 72355dd734ea4aeba031171e8c39696d P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:29:03.904263 4170 consensus_queue.cc:1035] T 72355dd734ea4aeba031171e8c39696d P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:29:03.904853 1685 raft_consensus.cc:1261] T 72355dd734ea4aeba031171e8c39696d P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:29:03.905619 4320 consensus_queue.cc:1035] T 72355dd734ea4aeba031171e8c39696d P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:29:03.905891 1549 raft_consensus.cc:1261] T 72355dd734ea4aeba031171e8c39696d P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:29:03.907877 3602 consensus_queue.cc:1035] T 72355dd734ea4aeba031171e8c39696d P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:29:03.909605 3602 raft_consensus.cc:2932] T 72355dd734ea4aeba031171e8c39696d P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER d8d7297186694f77bb98de9174e4279c (127.15.255.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } attrs { replace: true } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: true } } }
I20240502 21:29:03.912320 1685 raft_consensus.cc:2932] T 72355dd734ea4aeba031171e8c39696d P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER d8d7297186694f77bb98de9174e4279c (127.15.255.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } attrs { replace: true } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: true } } }
I20240502 21:29:03.913241 1955 raft_consensus.cc:2932] T 72355dd734ea4aeba031171e8c39696d P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER d8d7297186694f77bb98de9174e4279c (127.15.255.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } attrs { replace: true } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: true } } }
I20240502 21:29:03.917346 1819 raft_consensus.cc:1261] T 72355dd734ea4aeba031171e8c39696d P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:29:03.917897 4270 consensus_queue.cc:1035] T 72355dd734ea4aeba031171e8c39696d P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240502 21:29:03.918967 1349 consensus_peers.cc:485] T 72355dd734ea4aeba031171e8c39696d P 7d1e5c1b047443cbb28d7b0a9744babf -> Peer d8d7297186694f77bb98de9174e4279c (127.15.255.136:41459): Couldn't send request to peer d8d7297186694f77bb98de9174e4279c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 72355dd734ea4aeba031171e8c39696d. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:29:03.920321 1263 catalog_manager.cc:5484] T 72355dd734ea4aeba031171e8c39696d P e632c553ff4341328acf1cbaca678392 reported cstate change: config changed from index 4 to 5, NON_VOTER d8d7297186694f77bb98de9174e4279c (127.15.255.136) added. New cstate: current_term: 1 leader_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } attrs { replace: true } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: true } } }
I20240502 21:29:03.920833 1819 raft_consensus.cc:2932] T 72355dd734ea4aeba031171e8c39696d P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER d8d7297186694f77bb98de9174e4279c (127.15.255.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } attrs { replace: true } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: true } } }
I20240502 21:29:03.923702 1549 raft_consensus.cc:2932] T 72355dd734ea4aeba031171e8c39696d P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER d8d7297186694f77bb98de9174e4279c (127.15.255.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } attrs { replace: true } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: true } } }
I20240502 21:29:03.926134 1818 consensus_queue.cc:237] T 13af1b96121c492c9373f9acbb55725a P 153026dd7f8943cc93fedf6f48450fb4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } }
I20240502 21:29:03.937654 1255 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 72355dd734ea4aeba031171e8c39696d with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240502 21:29:03.938989 2089 raft_consensus.cc:2932] T 72355dd734ea4aeba031171e8c39696d P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER d8d7297186694f77bb98de9174e4279c (127.15.255.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } attrs { replace: true } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: true } } }
I20240502 21:29:03.939272 2226 raft_consensus.cc:2932] T 72355dd734ea4aeba031171e8c39696d P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER d8d7297186694f77bb98de9174e4279c (127.15.255.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } attrs { replace: true } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: true } } }
I20240502 21:29:03.953644 2089 raft_consensus.cc:1261] T 13af1b96121c492c9373f9acbb55725a P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Refusing update from remote peer 153026dd7f8943cc93fedf6f48450fb4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:29:03.953660 1955 raft_consensus.cc:1261] T 13af1b96121c492c9373f9acbb55725a P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Refusing update from remote peer 153026dd7f8943cc93fedf6f48450fb4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:29:03.954520 3357 consensus_queue.cc:1035] T 13af1b96121c492c9373f9acbb55725a P 153026dd7f8943cc93fedf6f48450fb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:03.958446 3357 consensus_queue.cc:1035] T 13af1b96121c492c9373f9acbb55725a P 153026dd7f8943cc93fedf6f48450fb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20240502 21:29:03.967092 2226 raft_consensus.cc:1261] T 13af1b96121c492c9373f9acbb55725a P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Refusing update from remote peer 153026dd7f8943cc93fedf6f48450fb4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:29:03.967667 3357 consensus_queue.cc:1035] T 13af1b96121c492c9373f9acbb55725a P 153026dd7f8943cc93fedf6f48450fb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:03.977658 1549 raft_consensus.cc:1261] T 13af1b96121c492c9373f9acbb55725a P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Refusing update from remote peer 153026dd7f8943cc93fedf6f48450fb4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:29:03.977702 1685 raft_consensus.cc:1261] T 13af1b96121c492c9373f9acbb55725a P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Refusing update from remote peer 153026dd7f8943cc93fedf6f48450fb4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:29:03.978271 3357 consensus_queue.cc:1035] T 13af1b96121c492c9373f9acbb55725a P 153026dd7f8943cc93fedf6f48450fb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:03.979116 4341 consensus_queue.cc:1035] T 13af1b96121c492c9373f9acbb55725a P 153026dd7f8943cc93fedf6f48450fb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:03.981799 3357 raft_consensus.cc:2932] T 13af1b96121c492c9373f9acbb55725a P 153026dd7f8943cc93fedf6f48450fb4 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } }
I20240502 21:29:03.987298 1414 raft_consensus.cc:1261] T 13af1b96121c492c9373f9acbb55725a P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Refusing update from remote peer 153026dd7f8943cc93fedf6f48450fb4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:29:03.986863 1685 raft_consensus.cc:2932] T 13af1b96121c492c9373f9acbb55725a P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } }
I20240502 21:29:03.991555 4348 consensus_queue.cc:1035] T 13af1b96121c492c9373f9acbb55725a P 153026dd7f8943cc93fedf6f48450fb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:03.991891 1268 catalog_manager.cc:5484] T 13af1b96121c492c9373f9acbb55725a P 16b3a337bc5e462f83ba895ad8660deb reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "153026dd7f8943cc93fedf6f48450fb4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } }
I20240502 21:29:03.992856 1549 raft_consensus.cc:2932] T 13af1b96121c492c9373f9acbb55725a P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } }
I20240502 21:29:03.994755 2226 raft_consensus.cc:2932] T 13af1b96121c492c9373f9acbb55725a P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } }
I20240502 21:29:03.995409 2089 raft_consensus.cc:2932] T 13af1b96121c492c9373f9acbb55725a P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } }
I20240502 21:29:03.998988 1955 raft_consensus.cc:2932] T 13af1b96121c492c9373f9acbb55725a P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } }
I20240502 21:29:04.002413 1414 raft_consensus.cc:2932] T 13af1b96121c492c9373f9acbb55725a P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } }
I20240502 21:29:04.018635 1818 consensus_queue.cc:237] T 13af1b96121c492c9373f9acbb55725a P 153026dd7f8943cc93fedf6f48450fb4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: true } }
I20240502 21:29:04.022326 1685 raft_consensus.cc:1261] T 13af1b96121c492c9373f9acbb55725a P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Refusing update from remote peer 153026dd7f8943cc93fedf6f48450fb4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:29:04.022665 2226 raft_consensus.cc:1261] T 13af1b96121c492c9373f9acbb55725a P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Refusing update from remote peer 153026dd7f8943cc93fedf6f48450fb4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:29:04.023008 4350 consensus_queue.cc:1035] T 13af1b96121c492c9373f9acbb55725a P 153026dd7f8943cc93fedf6f48450fb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:29:04.023213 1549 raft_consensus.cc:1261] T 13af1b96121c492c9373f9acbb55725a P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Refusing update from remote peer 153026dd7f8943cc93fedf6f48450fb4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:29:04.023473 1955 raft_consensus.cc:1261] T 13af1b96121c492c9373f9acbb55725a P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Refusing update from remote peer 153026dd7f8943cc93fedf6f48450fb4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:29:04.024708 1414 raft_consensus.cc:1261] T 13af1b96121c492c9373f9acbb55725a P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Refusing update from remote peer 153026dd7f8943cc93fedf6f48450fb4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:29:04.025635 4348 consensus_queue.cc:1035] T 13af1b96121c492c9373f9acbb55725a P 153026dd7f8943cc93fedf6f48450fb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:29:04.025887 4349 consensus_queue.cc:1035] T 13af1b96121c492c9373f9acbb55725a P 153026dd7f8943cc93fedf6f48450fb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:29:04.026443 2089 raft_consensus.cc:1261] T 13af1b96121c492c9373f9acbb55725a P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Refusing update from remote peer 153026dd7f8943cc93fedf6f48450fb4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:29:04.026541 4349 consensus_queue.cc:1035] T 13af1b96121c492c9373f9acbb55725a P 153026dd7f8943cc93fedf6f48450fb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:29:04.027230 4351 consensus_queue.cc:1035] T 13af1b96121c492c9373f9acbb55725a P 153026dd7f8943cc93fedf6f48450fb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:29:04.027701 4351 consensus_queue.cc:1035] T 13af1b96121c492c9373f9acbb55725a P 153026dd7f8943cc93fedf6f48450fb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:29:04.030766 4351 raft_consensus.cc:2932] T 13af1b96121c492c9373f9acbb55725a P 153026dd7f8943cc93fedf6f48450fb4 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER d8d7297186694f77bb98de9174e4279c (127.15.255.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: true } } }
I20240502 21:29:04.031558 2087 raft_consensus.cc:2932] T 13af1b96121c492c9373f9acbb55725a P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER d8d7297186694f77bb98de9174e4279c (127.15.255.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: true } } }
I20240502 21:29:04.031782 1955 raft_consensus.cc:2932] T 13af1b96121c492c9373f9acbb55725a P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER d8d7297186694f77bb98de9174e4279c (127.15.255.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: true } } }
I20240502 21:29:04.034020 1549 raft_consensus.cc:2932] T 13af1b96121c492c9373f9acbb55725a P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER d8d7297186694f77bb98de9174e4279c (127.15.255.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: true } } }
I20240502 21:29:04.035477 1256 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 13af1b96121c492c9373f9acbb55725a with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240502 21:29:04.036727 2226 raft_consensus.cc:2932] T 13af1b96121c492c9373f9acbb55725a P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER d8d7297186694f77bb98de9174e4279c (127.15.255.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: true } } }
I20240502 21:29:04.036787 1685 raft_consensus.cc:2932] T 13af1b96121c492c9373f9acbb55725a P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER d8d7297186694f77bb98de9174e4279c (127.15.255.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: true } } }
I20240502 21:29:04.037602 1263 catalog_manager.cc:5484] T 13af1b96121c492c9373f9acbb55725a P e632c553ff4341328acf1cbaca678392 reported cstate change: config changed from index 4 to 5, NON_VOTER d8d7297186694f77bb98de9174e4279c (127.15.255.136) added. New cstate: current_term: 1 leader_uuid: "153026dd7f8943cc93fedf6f48450fb4" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: true } } }
I20240502 21:29:04.039999 1414 raft_consensus.cc:2932] T 13af1b96121c492c9373f9acbb55725a P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER d8d7297186694f77bb98de9174e4279c (127.15.255.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: true } } }
W20240502 21:29:04.058449 1753 consensus_peers.cc:485] T 13af1b96121c492c9373f9acbb55725a P 153026dd7f8943cc93fedf6f48450fb4 -> Peer d8d7297186694f77bb98de9174e4279c (127.15.255.136:41459): Couldn't send request to peer d8d7297186694f77bb98de9174e4279c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 13af1b96121c492c9373f9acbb55725a. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:29:04.066339 1414 consensus_queue.cc:237] T f8bee9bdcbff404280f000bdaeadf465 P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:29:04.070091 1549 raft_consensus.cc:1261] T f8bee9bdcbff404280f000bdaeadf465 P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:29:04.070606 1685 raft_consensus.cc:1261] T f8bee9bdcbff404280f000bdaeadf465 P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:29:04.070719 2226 raft_consensus.cc:1261] T f8bee9bdcbff404280f000bdaeadf465 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:29:04.070948 1955 raft_consensus.cc:1261] T f8bee9bdcbff404280f000bdaeadf465 P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:29:04.071161 3602 consensus_queue.cc:1035] T f8bee9bdcbff404280f000bdaeadf465 P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:04.071322 2087 raft_consensus.cc:1261] T f8bee9bdcbff404280f000bdaeadf465 P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:29:04.071439 4270 consensus_queue.cc:1035] T f8bee9bdcbff404280f000bdaeadf465 P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:04.071841 4320 consensus_queue.cc:1035] T f8bee9bdcbff404280f000bdaeadf465 P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:04.072217 1818 raft_consensus.cc:1261] T f8bee9bdcbff404280f000bdaeadf465 P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:29:04.072342 4320 consensus_queue.cc:1035] T f8bee9bdcbff404280f000bdaeadf465 P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:04.072768 4170 consensus_queue.cc:1035] T f8bee9bdcbff404280f000bdaeadf465 P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:04.073081 3602 consensus_queue.cc:1035] T f8bee9bdcbff404280f000bdaeadf465 P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:04.078248 4170 raft_consensus.cc:2932] T f8bee9bdcbff404280f000bdaeadf465 P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } }
I20240502 21:29:04.081871 1549 raft_consensus.cc:2932] T f8bee9bdcbff404280f000bdaeadf465 P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } }
I20240502 21:29:04.082158 2226 raft_consensus.cc:2932] T f8bee9bdcbff404280f000bdaeadf465 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } }
I20240502 21:29:04.083001 2087 raft_consensus.cc:2932] T f8bee9bdcbff404280f000bdaeadf465 P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } }
I20240502 21:29:04.086561 1265 catalog_manager.cc:5484] T f8bee9bdcbff404280f000bdaeadf465 P 7d1e5c1b047443cbb28d7b0a9744babf reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } health_report { overall_health: HEALTHY } } }
I20240502 21:29:04.087522 1818 raft_consensus.cc:2932] T f8bee9bdcbff404280f000bdaeadf465 P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } }
I20240502 21:29:04.094059 1414 consensus_queue.cc:237] T f8bee9bdcbff404280f000bdaeadf465 P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: true } }
I20240502 21:29:04.094858 1685 raft_consensus.cc:2932] T f8bee9bdcbff404280f000bdaeadf465 P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } }
I20240502 21:29:04.097754 1819 raft_consensus.cc:1261] T f8bee9bdcbff404280f000bdaeadf465 P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:29:04.098426 4320 consensus_queue.cc:1035] T f8bee9bdcbff404280f000bdaeadf465 P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:29:04.099330 2087 raft_consensus.cc:1261] T f8bee9bdcbff404280f000bdaeadf465 P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:29:04.099884 4170 consensus_queue.cc:1035] T f8bee9bdcbff404280f000bdaeadf465 P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240502 21:29:04.100241 1349 consensus_peers.cc:485] T f8bee9bdcbff404280f000bdaeadf465 P 7d1e5c1b047443cbb28d7b0a9744babf -> Peer d8d7297186694f77bb98de9174e4279c (127.15.255.136:41459): Couldn't send request to peer d8d7297186694f77bb98de9174e4279c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f8bee9bdcbff404280f000bdaeadf465. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:29:04.100922 1955 raft_consensus.cc:1261] T f8bee9bdcbff404280f000bdaeadf465 P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:29:04.101426 4170 consensus_queue.cc:1035] T f8bee9bdcbff404280f000bdaeadf465 P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:04.102758 1954 raft_consensus.cc:2932] T f8bee9bdcbff404280f000bdaeadf465 P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } }
I20240502 21:29:04.113015 4361 raft_consensus.cc:2932] T f8bee9bdcbff404280f000bdaeadf465 P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER d8d7297186694f77bb98de9174e4279c (127.15.255.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: true } } }
I20240502 21:29:04.114122 1954 raft_consensus.cc:2932] T f8bee9bdcbff404280f000bdaeadf465 P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER d8d7297186694f77bb98de9174e4279c (127.15.255.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: true } } }
I20240502 21:29:04.117748 1685 raft_consensus.cc:1261] T f8bee9bdcbff404280f000bdaeadf465 P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:29:04.118157 1255 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet f8bee9bdcbff404280f000bdaeadf465 with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240502 21:29:04.118906 4361 consensus_queue.cc:1035] T f8bee9bdcbff404280f000bdaeadf465 P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:29:04.118860 1819 raft_consensus.cc:2932] T f8bee9bdcbff404280f000bdaeadf465 P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER d8d7297186694f77bb98de9174e4279c (127.15.255.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: true } } }
I20240502 21:29:04.119269 2087 raft_consensus.cc:2932] T f8bee9bdcbff404280f000bdaeadf465 P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER d8d7297186694f77bb98de9174e4279c (127.15.255.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: true } } }
I20240502 21:29:04.122824 2226 raft_consensus.cc:1261] T f8bee9bdcbff404280f000bdaeadf465 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:29:04.123838 1549 raft_consensus.cc:1261] T f8bee9bdcbff404280f000bdaeadf465 P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:29:04.125233 4368 consensus_queue.cc:1035] T f8bee9bdcbff404280f000bdaeadf465 P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:29:04.125075 1265 catalog_manager.cc:5484] T f8bee9bdcbff404280f000bdaeadf465 P 153026dd7f8943cc93fedf6f48450fb4 reported cstate change: config changed from index 4 to 5, NON_VOTER d8d7297186694f77bb98de9174e4279c (127.15.255.136) added. New cstate: current_term: 1 leader_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: true } } }
I20240502 21:29:04.125706 4368 consensus_queue.cc:1035] T f8bee9bdcbff404280f000bdaeadf465 P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:29:04.125252 1685 raft_consensus.cc:2932] T f8bee9bdcbff404280f000bdaeadf465 P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER d8d7297186694f77bb98de9174e4279c (127.15.255.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: true } } }
I20240502 21:29:04.128588 1549 raft_consensus.cc:2932] T f8bee9bdcbff404280f000bdaeadf465 P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER d8d7297186694f77bb98de9174e4279c (127.15.255.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: true } } }
I20240502 21:29:04.137656 2226 raft_consensus.cc:2932] T f8bee9bdcbff404280f000bdaeadf465 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER d8d7297186694f77bb98de9174e4279c (127.15.255.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: true } } }
I20240502 21:29:04.158453 2087 consensus_queue.cc:237] T 1aa57620996e42b6adf941e9d606f7e2 P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } attrs { replace: true } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } }
I20240502 21:29:04.161725 2226 raft_consensus.cc:1261] T 1aa57620996e42b6adf941e9d606f7e2 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:29:04.161748 1685 raft_consensus.cc:1261] T 1aa57620996e42b6adf941e9d606f7e2 P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:29:04.161732 1819 raft_consensus.cc:1261] T 1aa57620996e42b6adf941e9d606f7e2 P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:29:04.162364 3545 consensus_queue.cc:1035] T 1aa57620996e42b6adf941e9d606f7e2 P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:04.162951 3545 consensus_queue.cc:1035] T 1aa57620996e42b6adf941e9d606f7e2 P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:04.163668 4332 consensus_queue.cc:1035] T 1aa57620996e42b6adf941e9d606f7e2 P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:04.166935 1954 raft_consensus.cc:1261] T 1aa57620996e42b6adf941e9d606f7e2 P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:29:04.167560 4332 consensus_queue.cc:1035] T 1aa57620996e42b6adf941e9d606f7e2 P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:04.168846 1549 raft_consensus.cc:1261] T 1aa57620996e42b6adf941e9d606f7e2 P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:29:04.169704 1414 raft_consensus.cc:1261] T 1aa57620996e42b6adf941e9d606f7e2 P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:29:04.170387 3765 consensus_queue.cc:1035] T 1aa57620996e42b6adf941e9d606f7e2 P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:04.170928 3765 raft_consensus.cc:2932] T 1aa57620996e42b6adf941e9d606f7e2 P 521bb93035324cecbebed2f6f2a630f8 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } attrs { replace: true } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } }
I20240502 21:29:04.173000 1819 raft_consensus.cc:2932] T 1aa57620996e42b6adf941e9d606f7e2 P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } attrs { replace: true } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } }
I20240502 21:29:04.175034 3545 consensus_queue.cc:1035] T 1aa57620996e42b6adf941e9d606f7e2 P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:04.175446 1549 raft_consensus.cc:2932] T 1aa57620996e42b6adf941e9d606f7e2 P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } attrs { replace: true } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } }
I20240502 21:29:04.180436 2225 raft_consensus.cc:2932] T 1aa57620996e42b6adf941e9d606f7e2 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } attrs { replace: true } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } }
I20240502 21:29:04.186954 1268 catalog_manager.cc:5484] T 1aa57620996e42b6adf941e9d606f7e2 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "521bb93035324cecbebed2f6f2a630f8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } attrs { replace: true } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } }
I20240502 21:29:04.191955 1954 raft_consensus.cc:2932] T 1aa57620996e42b6adf941e9d606f7e2 P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } attrs { replace: true } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } }
I20240502 21:29:04.197650 1685 raft_consensus.cc:2932] T 1aa57620996e42b6adf941e9d606f7e2 P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } attrs { replace: true } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } }
I20240502 21:29:04.200809 1414 raft_consensus.cc:2932] T 1aa57620996e42b6adf941e9d606f7e2 P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } attrs { replace: true } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } }
I20240502 21:29:04.211777 2087 consensus_queue.cc:237] T 1aa57620996e42b6adf941e9d606f7e2 P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } attrs { replace: true } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: true } }
I20240502 21:29:04.221304 1685 raft_consensus.cc:1261] T 1aa57620996e42b6adf941e9d606f7e2 P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:29:04.221897 4383 consensus_queue.cc:1035] T 1aa57620996e42b6adf941e9d606f7e2 P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:29:04.222837 1414 raft_consensus.cc:1261] T 1aa57620996e42b6adf941e9d606f7e2 P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:29:04.223529 3765 consensus_queue.cc:1035] T 1aa57620996e42b6adf941e9d606f7e2 P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:29:04.223838 2225 raft_consensus.cc:1261] T 1aa57620996e42b6adf941e9d606f7e2 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:29:04.224251 1549 raft_consensus.cc:1261] T 1aa57620996e42b6adf941e9d606f7e2 P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:29:04.227268 3545 consensus_queue.cc:1035] T 1aa57620996e42b6adf941e9d606f7e2 P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:29:04.227754 3545 consensus_queue.cc:1035] T 1aa57620996e42b6adf941e9d606f7e2 P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:29:04.230824 4377 raft_consensus.cc:2932] T 1aa57620996e42b6adf941e9d606f7e2 P 521bb93035324cecbebed2f6f2a630f8 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER d8d7297186694f77bb98de9174e4279c (127.15.255.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } attrs { replace: true } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: true } } }
I20240502 21:29:04.231573 2225 raft_consensus.cc:2932] T 1aa57620996e42b6adf941e9d606f7e2 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER d8d7297186694f77bb98de9174e4279c (127.15.255.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } attrs { replace: true } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: true } } }
I20240502 21:29:04.232467 1819 raft_consensus.cc:1261] T 1aa57620996e42b6adf941e9d606f7e2 P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:29:04.234872 1254 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 1aa57620996e42b6adf941e9d606f7e2 with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240502 21:29:04.235110 1549 raft_consensus.cc:2932] T 1aa57620996e42b6adf941e9d606f7e2 P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER d8d7297186694f77bb98de9174e4279c (127.15.255.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } attrs { replace: true } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: true } } }
I20240502 21:29:04.235592 1685 raft_consensus.cc:2932] T 1aa57620996e42b6adf941e9d606f7e2 P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER d8d7297186694f77bb98de9174e4279c (127.15.255.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } attrs { replace: true } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: true } } }
I20240502 21:29:04.237257 1414 raft_consensus.cc:2932] T 1aa57620996e42b6adf941e9d606f7e2 P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER d8d7297186694f77bb98de9174e4279c (127.15.255.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } attrs { replace: true } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: true } } }
I20240502 21:29:04.240389 1954 raft_consensus.cc:1261] T 1aa57620996e42b6adf941e9d606f7e2 P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:29:04.240936 4377 consensus_queue.cc:1035] T 1aa57620996e42b6adf941e9d606f7e2 P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:29:04.241291 1263 catalog_manager.cc:5484] T 1aa57620996e42b6adf941e9d606f7e2 P 16b3a337bc5e462f83ba895ad8660deb reported cstate change: config changed from index 4 to 5, NON_VOTER d8d7297186694f77bb98de9174e4279c (127.15.255.136) added. New cstate: current_term: 1 leader_uuid: "521bb93035324cecbebed2f6f2a630f8" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } attrs { replace: true } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: true } } }
I20240502 21:29:04.242381 1955 raft_consensus.cc:2932] T 1aa57620996e42b6adf941e9d606f7e2 P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER d8d7297186694f77bb98de9174e4279c (127.15.255.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } attrs { replace: true } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: true } } }
I20240502 21:29:04.250488 4332 consensus_queue.cc:1035] T 1aa57620996e42b6adf941e9d606f7e2 P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:29:04.252151 1819 raft_consensus.cc:2932] T 1aa57620996e42b6adf941e9d606f7e2 P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER d8d7297186694f77bb98de9174e4279c (127.15.255.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } attrs { replace: true } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: true } } }
W20240502 21:29:04.258008 2024 consensus_peers.cc:485] T 1aa57620996e42b6adf941e9d606f7e2 P 521bb93035324cecbebed2f6f2a630f8 -> Peer d8d7297186694f77bb98de9174e4279c (127.15.255.136:41459): Couldn't send request to peer d8d7297186694f77bb98de9174e4279c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1aa57620996e42b6adf941e9d606f7e2. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:29:04.266584 2225 consensus_queue.cc:237] T bcddb146822e4573837a5697d74134f7 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:29:04.269423 1684 raft_consensus.cc:1261] T bcddb146822e4573837a5697d74134f7 P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Refusing update from remote peer 6ab8df3c796f4b6cb3e29d5045ad3ea0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:29:04.269747 1549 raft_consensus.cc:1261] T bcddb146822e4573837a5697d74134f7 P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Refusing update from remote peer 6ab8df3c796f4b6cb3e29d5045ad3ea0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:29:04.270051 1954 raft_consensus.cc:1261] T bcddb146822e4573837a5697d74134f7 P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Refusing update from remote peer 6ab8df3c796f4b6cb3e29d5045ad3ea0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:29:04.269997 4356 consensus_queue.cc:1035] T bcddb146822e4573837a5697d74134f7 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:04.270359 3705 consensus_queue.cc:1035] T bcddb146822e4573837a5697d74134f7 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:04.270617 1819 raft_consensus.cc:1261] T bcddb146822e4573837a5697d74134f7 P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Refusing update from remote peer 6ab8df3c796f4b6cb3e29d5045ad3ea0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:29:04.270615 4356 consensus_queue.cc:1035] T bcddb146822e4573837a5697d74134f7 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:04.271593 3705 consensus_queue.cc:1035] T bcddb146822e4573837a5697d74134f7 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:04.271636 1414 raft_consensus.cc:1261] T bcddb146822e4573837a5697d74134f7 P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Refusing update from remote peer 6ab8df3c796f4b6cb3e29d5045ad3ea0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:29:04.282845 4394 consensus_queue.cc:1035] T bcddb146822e4573837a5697d74134f7 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:04.283841 2087 raft_consensus.cc:1261] T bcddb146822e4573837a5697d74134f7 P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Refusing update from remote peer 6ab8df3c796f4b6cb3e29d5045ad3ea0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:29:04.290760 3705 consensus_queue.cc:1035] T bcddb146822e4573837a5697d74134f7 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:04.301316 4356 raft_consensus.cc:2932] T bcddb146822e4573837a5697d74134f7 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } }
I20240502 21:29:04.307662 1414 raft_consensus.cc:2932] T bcddb146822e4573837a5697d74134f7 P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } }
I20240502 21:29:04.307919 1549 raft_consensus.cc:2932] T bcddb146822e4573837a5697d74134f7 P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } }
I20240502 21:29:04.309209 1684 raft_consensus.cc:2932] T bcddb146822e4573837a5697d74134f7 P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } }
I20240502 21:29:04.310312 1819 raft_consensus.cc:2932] T bcddb146822e4573837a5697d74134f7 P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } }
I20240502 21:29:04.313189 1270 catalog_manager.cc:5484] T bcddb146822e4573837a5697d74134f7 P 7d1e5c1b047443cbb28d7b0a9744babf reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } }
I20240502 21:29:04.315302 2087 raft_consensus.cc:2932] T bcddb146822e4573837a5697d74134f7 P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } }
I20240502 21:29:04.316779 1954 raft_consensus.cc:2932] T bcddb146822e4573837a5697d74134f7 P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } }
I20240502 21:29:04.325070 2225 consensus_queue.cc:237] T bcddb146822e4573837a5697d74134f7 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: NON_VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: true } }
I20240502 21:29:04.334138 1819 raft_consensus.cc:1261] T bcddb146822e4573837a5697d74134f7 P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Refusing update from remote peer 6ab8df3c796f4b6cb3e29d5045ad3ea0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:29:04.339066 2087 raft_consensus.cc:1261] T bcddb146822e4573837a5697d74134f7 P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Refusing update from remote peer 6ab8df3c796f4b6cb3e29d5045ad3ea0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:29:04.339656 1414 raft_consensus.cc:1261] T bcddb146822e4573837a5697d74134f7 P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Refusing update from remote peer 6ab8df3c796f4b6cb3e29d5045ad3ea0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:29:04.340102 4356 consensus_queue.cc:1035] T bcddb146822e4573837a5697d74134f7 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:29:04.341276 3705 consensus_queue.cc:1035] T bcddb146822e4573837a5697d74134f7 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:29:04.341795 3705 consensus_queue.cc:1035] T bcddb146822e4573837a5697d74134f7 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:29:04.341919 1955 raft_consensus.cc:1261] T bcddb146822e4573837a5697d74134f7 P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Refusing update from remote peer 6ab8df3c796f4b6cb3e29d5045ad3ea0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:29:04.342206 1684 raft_consensus.cc:1261] T bcddb146822e4573837a5697d74134f7 P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Refusing update from remote peer 6ab8df3c796f4b6cb3e29d5045ad3ea0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:29:04.343019 4393 consensus_queue.cc:1035] T bcddb146822e4573837a5697d74134f7 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:29:04.343561 4393 consensus_queue.cc:1035] T bcddb146822e4573837a5697d74134f7 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:29:04.345265 1549 raft_consensus.cc:1261] T bcddb146822e4573837a5697d74134f7 P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Refusing update from remote peer 6ab8df3c796f4b6cb3e29d5045ad3ea0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:29:04.345906 4393 consensus_queue.cc:1035] T bcddb146822e4573837a5697d74134f7 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:29:04.347796 4393 raft_consensus.cc:2932] T bcddb146822e4573837a5697d74134f7 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 16a97c1394f346f0b987f97cce7371b6 (127.15.255.137) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: NON_VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: true } } }
I20240502 21:29:04.352609 1955 raft_consensus.cc:2932] T bcddb146822e4573837a5697d74134f7 P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 16a97c1394f346f0b987f97cce7371b6 (127.15.255.137) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: NON_VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: true } } }
I20240502 21:29:04.352677 2087 raft_consensus.cc:2932] T bcddb146822e4573837a5697d74134f7 P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 16a97c1394f346f0b987f97cce7371b6 (127.15.255.137) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: NON_VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: true } } }
I20240502 21:29:04.358430 1270 catalog_manager.cc:5484] T bcddb146822e4573837a5697d74134f7 P e632c553ff4341328acf1cbaca678392 reported cstate change: config changed from index 4 to 5, NON_VOTER 16a97c1394f346f0b987f97cce7371b6 (127.15.255.137) added. New cstate: current_term: 1 leader_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: NON_VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: true } } }
I20240502 21:29:04.361142 1255 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet bcddb146822e4573837a5697d74134f7 with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240502 21:29:04.362250 1414 raft_consensus.cc:2932] T bcddb146822e4573837a5697d74134f7 P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 16a97c1394f346f0b987f97cce7371b6 (127.15.255.137) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: NON_VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: true } } }
W20240502 21:29:04.364881 2158 consensus_peers.cc:485] T bcddb146822e4573837a5697d74134f7 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 -> Peer 16a97c1394f346f0b987f97cce7371b6 (127.15.255.137:43717): Couldn't send request to peer 16a97c1394f346f0b987f97cce7371b6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: bcddb146822e4573837a5697d74134f7. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:29:04.367532 1819 raft_consensus.cc:2932] T bcddb146822e4573837a5697d74134f7 P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 16a97c1394f346f0b987f97cce7371b6 (127.15.255.137) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: NON_VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: true } } }
I20240502 21:29:04.368351 1684 raft_consensus.cc:2932] T bcddb146822e4573837a5697d74134f7 P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 16a97c1394f346f0b987f97cce7371b6 (127.15.255.137) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: NON_VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: true } } }
I20240502 21:29:04.375658 1549 raft_consensus.cc:2932] T bcddb146822e4573837a5697d74134f7 P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 16a97c1394f346f0b987f97cce7371b6 (127.15.255.137) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: NON_VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: true } } }
I20240502 21:29:04.391292 2087 consensus_queue.cc:237] T 8f86d1517df74ead9e397ca9bed44c3f P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } }
I20240502 21:29:04.393643 2225 raft_consensus.cc:1261] T 8f86d1517df74ead9e397ca9bed44c3f P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:29:04.393702 1819 raft_consensus.cc:1261] T 8f86d1517df74ead9e397ca9bed44c3f P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:29:04.393725 1955 raft_consensus.cc:1261] T 8f86d1517df74ead9e397ca9bed44c3f P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:29:04.394177 4377 consensus_queue.cc:1035] T 8f86d1517df74ead9e397ca9bed44c3f P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:04.394475 3545 consensus_queue.cc:1035] T 8f86d1517df74ead9e397ca9bed44c3f P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:04.394965 3765 consensus_queue.cc:1035] T 8f86d1517df74ead9e397ca9bed44c3f P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:04.396248 1684 raft_consensus.cc:1261] T 8f86d1517df74ead9e397ca9bed44c3f P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:29:04.396718 1549 raft_consensus.cc:1261] T 8f86d1517df74ead9e397ca9bed44c3f P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:29:04.396859 4383 consensus_queue.cc:1035] T 8f86d1517df74ead9e397ca9bed44c3f P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:04.397722 4377 consensus_queue.cc:1035] T 8f86d1517df74ead9e397ca9bed44c3f P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:04.403216 1414 raft_consensus.cc:1261] T 8f86d1517df74ead9e397ca9bed44c3f P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:29:04.403782 4383 consensus_queue.cc:1035] T 8f86d1517df74ead9e397ca9bed44c3f P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:04.418527 4410 raft_consensus.cc:2932] T 8f86d1517df74ead9e397ca9bed44c3f P 521bb93035324cecbebed2f6f2a630f8 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } }
I20240502 21:29:04.419550 2225 raft_consensus.cc:2932] T 8f86d1517df74ead9e397ca9bed44c3f P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } }
I20240502 21:29:04.419749 1550 raft_consensus.cc:2932] T 8f86d1517df74ead9e397ca9bed44c3f P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } }
I20240502 21:29:04.420082 1819 raft_consensus.cc:2932] T 8f86d1517df74ead9e397ca9bed44c3f P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } }
I20240502 21:29:04.425586 1270 catalog_manager.cc:5484] T 8f86d1517df74ead9e397ca9bed44c3f P 6ab8df3c796f4b6cb3e29d5045ad3ea0 reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "521bb93035324cecbebed2f6f2a630f8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } }
I20240502 21:29:04.426486 1954 raft_consensus.cc:2932] T 8f86d1517df74ead9e397ca9bed44c3f P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } }
I20240502 21:29:04.427160 1414 raft_consensus.cc:2932] T 8f86d1517df74ead9e397ca9bed44c3f P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } }
I20240502 21:29:04.439240 2087 consensus_queue.cc:237] T 8f86d1517df74ead9e397ca9bed44c3f P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: NON_VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: true } }
I20240502 21:29:04.444736 1414 raft_consensus.cc:1261] T 8f86d1517df74ead9e397ca9bed44c3f P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:29:04.445011 1684 raft_consensus.cc:1261] T 8f86d1517df74ead9e397ca9bed44c3f P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:29:04.445549 3765 consensus_queue.cc:1035] T 8f86d1517df74ead9e397ca9bed44c3f P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:04.446785 1685 raft_consensus.cc:2932] T 8f86d1517df74ead9e397ca9bed44c3f P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } }
I20240502 21:29:04.447678 3765 consensus_queue.cc:1035] T 8f86d1517df74ead9e397ca9bed44c3f P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:29:04.452390 1819 raft_consensus.cc:1261] T 8f86d1517df74ead9e397ca9bed44c3f P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:29:04.452942 4332 consensus_queue.cc:1035] T 8f86d1517df74ead9e397ca9bed44c3f P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:29:04.461323 3765 raft_consensus.cc:2932] T 8f86d1517df74ead9e397ca9bed44c3f P 521bb93035324cecbebed2f6f2a630f8 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: NON_VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: true } } }
I20240502 21:29:04.462224 2226 raft_consensus.cc:1261] T 8f86d1517df74ead9e397ca9bed44c3f P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:29:04.462795 3545 consensus_queue.cc:1035] T 8f86d1517df74ead9e397ca9bed44c3f P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:29:04.464491 2226 raft_consensus.cc:2932] T 8f86d1517df74ead9e397ca9bed44c3f P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: NON_VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: true } } }
I20240502 21:29:04.465998 1254 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 8f86d1517df74ead9e397ca9bed44c3f with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240502 21:29:04.467195 1549 raft_consensus.cc:1261] T 8f86d1517df74ead9e397ca9bed44c3f P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:29:04.466742 1819 raft_consensus.cc:2932] T 8f86d1517df74ead9e397ca9bed44c3f P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: NON_VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: true } } }
I20240502 21:29:04.468611 1415 raft_consensus.cc:2932] T 8f86d1517df74ead9e397ca9bed44c3f P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: NON_VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: true } } }
I20240502 21:29:04.473910 3545 consensus_queue.cc:1035] T 8f86d1517df74ead9e397ca9bed44c3f P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:29:04.474387 1270 catalog_manager.cc:5484] T 8f86d1517df74ead9e397ca9bed44c3f P 521bb93035324cecbebed2f6f2a630f8 reported cstate change: config changed from index 4 to 5, NON_VOTER 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138) added. New cstate: current_term: 1 leader_uuid: "521bb93035324cecbebed2f6f2a630f8" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: NON_VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 21:29:04.464663 1685 raft_consensus.cc:2932] T 8f86d1517df74ead9e397ca9bed44c3f P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: NON_VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: true } } }
I20240502 21:29:04.477795 1549 raft_consensus.cc:2932] T 8f86d1517df74ead9e397ca9bed44c3f P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: NON_VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: true } } }
I20240502 21:29:04.487717 1954 raft_consensus.cc:1261] T 8f86d1517df74ead9e397ca9bed44c3f P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:29:04.488214 3765 consensus_queue.cc:1035] T 8f86d1517df74ead9e397ca9bed44c3f P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:29:04.489717 1954 raft_consensus.cc:2932] T 8f86d1517df74ead9e397ca9bed44c3f P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: NON_VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: true } } }
I20240502 21:29:04.518800 4422 ts_tablet_manager.cc:902] T 13af1b96121c492c9373f9acbb55725a P d8d7297186694f77bb98de9174e4279c: Initiating tablet copy from peer 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132:44947)
I20240502 21:29:04.520994 4422 tablet_copy_client.cc:308] T 13af1b96121c492c9373f9acbb55725a P d8d7297186694f77bb98de9174e4279c: tablet copy: Beginning tablet copy session from remote peer at address 127.15.255.132:44947
I20240502 21:29:04.524514 1549 consensus_queue.cc:237] T 3c88daaac3734d918955a774829532c6 P a62a935ff43d494ca35d0c417943133d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } attrs { replace: true } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:29:04.530221 4425 ts_tablet_manager.cc:902] T 72355dd734ea4aeba031171e8c39696d P d8d7297186694f77bb98de9174e4279c: Initiating tablet copy from peer 7d1e5c1b047443cbb28d7b0a9744babf (127.15.255.129:37897)
W20240502 21:29:04.532445 2025 consensus_peers.cc:485] T 8f86d1517df74ead9e397ca9bed44c3f P 521bb93035324cecbebed2f6f2a630f8 -> Peer 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138:33375): Couldn't send request to peer 1fab5794cf1740288abd82f2cebb9b1a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8f86d1517df74ead9e397ca9bed44c3f. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:29:04.537684 1685 raft_consensus.cc:1261] T 3c88daaac3734d918955a774829532c6 P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Refusing update from remote peer a62a935ff43d494ca35d0c417943133d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:29:04.538251 3697 consensus_queue.cc:1035] T 3c88daaac3734d918955a774829532c6 P a62a935ff43d494ca35d0c417943133d [LEADER]: Connected to new peer: Peer: permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:04.538367 2087 raft_consensus.cc:1261] T 3c88daaac3734d918955a774829532c6 P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Refusing update from remote peer a62a935ff43d494ca35d0c417943133d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:29:04.539157 1414 raft_consensus.cc:1261] T 3c88daaac3734d918955a774829532c6 P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Refusing update from remote peer a62a935ff43d494ca35d0c417943133d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:29:04.539281 3697 consensus_queue.cc:1035] T 3c88daaac3734d918955a774829532c6 P a62a935ff43d494ca35d0c417943133d [LEADER]: Connected to new peer: Peer: permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:04.540789 3697 consensus_queue.cc:1035] T 3c88daaac3734d918955a774829532c6 P a62a935ff43d494ca35d0c417943133d [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:04.556591 1954 raft_consensus.cc:1261] T 3c88daaac3734d918955a774829532c6 P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Refusing update from remote peer a62a935ff43d494ca35d0c417943133d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:29:04.556811 1818 raft_consensus.cc:1261] T 3c88daaac3734d918955a774829532c6 P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Refusing update from remote peer a62a935ff43d494ca35d0c417943133d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:29:04.557230 4437 consensus_queue.cc:1035] T 3c88daaac3734d918955a774829532c6 P a62a935ff43d494ca35d0c417943133d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:04.557801 4437 consensus_queue.cc:1035] T 3c88daaac3734d918955a774829532c6 P a62a935ff43d494ca35d0c417943133d [LEADER]: Connected to new peer: Peer: permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:04.559649 2226 raft_consensus.cc:1261] T 3c88daaac3734d918955a774829532c6 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Refusing update from remote peer a62a935ff43d494ca35d0c417943133d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:29:04.560963 3697 consensus_queue.cc:1035] T 3c88daaac3734d918955a774829532c6 P a62a935ff43d494ca35d0c417943133d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:04.571820 1954 raft_consensus.cc:2932] T 3c88daaac3734d918955a774829532c6 P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } attrs { replace: true } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } }
I20240502 21:29:04.577570 1270 catalog_manager.cc:5484] T 3c88daaac3734d918955a774829532c6 P e632c553ff4341328acf1cbaca678392 reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "a62a935ff43d494ca35d0c417943133d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } attrs { replace: true } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } }
I20240502 21:29:04.575871 4443 raft_consensus.cc:2932] T 3c88daaac3734d918955a774829532c6 P a62a935ff43d494ca35d0c417943133d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } attrs { replace: true } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } }
I20240502 21:29:04.582872 2225 raft_consensus.cc:2932] T 3c88daaac3734d918955a774829532c6 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } attrs { replace: true } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } }
I20240502 21:29:04.595634 1685 raft_consensus.cc:2932] T 3c88daaac3734d918955a774829532c6 P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } attrs { replace: true } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } }
I20240502 21:29:04.597906 1818 raft_consensus.cc:2932] T 3c88daaac3734d918955a774829532c6 P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } attrs { replace: true } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } }
I20240502 21:29:04.606710 1414 raft_consensus.cc:2932] T 3c88daaac3734d918955a774829532c6 P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } attrs { replace: true } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } }
I20240502 21:29:04.611835 1840 tablet_copy_service.cc:140] P 153026dd7f8943cc93fedf6f48450fb4: Received BeginTabletCopySession request for tablet 13af1b96121c492c9373f9acbb55725a from peer d8d7297186694f77bb98de9174e4279c ({username='slave'} at 127.15.255.136:41357)
I20240502 21:29:04.612113 1840 tablet_copy_service.cc:161] P 153026dd7f8943cc93fedf6f48450fb4: Beginning new tablet copy session on tablet 13af1b96121c492c9373f9acbb55725a from peer d8d7297186694f77bb98de9174e4279c at {username='slave'} at 127.15.255.136:41357: session id = d8d7297186694f77bb98de9174e4279c-13af1b96121c492c9373f9acbb55725a
I20240502 21:29:04.616297 1840 tablet_copy_source_session.cc:205] T 13af1b96121c492c9373f9acbb55725a P 153026dd7f8943cc93fedf6f48450fb4: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:29:04.616736 2087 raft_consensus.cc:2932] T 3c88daaac3734d918955a774829532c6 P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } attrs { replace: true } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } }
I20240502 21:29:04.623299 4425 tablet_copy_client.cc:308] T 72355dd734ea4aeba031171e8c39696d P d8d7297186694f77bb98de9174e4279c: tablet copy: Beginning tablet copy session from remote peer at address 127.15.255.129:37897
I20240502 21:29:04.626366 4422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13af1b96121c492c9373f9acbb55725a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:29:04.628806 1550 consensus_queue.cc:237] T 3c88daaac3734d918955a774829532c6 P a62a935ff43d494ca35d0c417943133d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } attrs { replace: true } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: true } }
I20240502 21:29:04.629846 1435 tablet_copy_service.cc:140] P 7d1e5c1b047443cbb28d7b0a9744babf: Received BeginTabletCopySession request for tablet 72355dd734ea4aeba031171e8c39696d from peer d8d7297186694f77bb98de9174e4279c ({username='slave'} at 127.15.255.136:40527)
I20240502 21:29:04.630079 1435 tablet_copy_service.cc:161] P 7d1e5c1b047443cbb28d7b0a9744babf: Beginning new tablet copy session on tablet 72355dd734ea4aeba031171e8c39696d from peer d8d7297186694f77bb98de9174e4279c at {username='slave'} at 127.15.255.136:40527: session id = d8d7297186694f77bb98de9174e4279c-72355dd734ea4aeba031171e8c39696d
I20240502 21:29:04.632587 1954 raft_consensus.cc:1261] T 3c88daaac3734d918955a774829532c6 P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Refusing update from remote peer a62a935ff43d494ca35d0c417943133d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:29:04.633323 4442 consensus_queue.cc:1035] T 3c88daaac3734d918955a774829532c6 P a62a935ff43d494ca35d0c417943133d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:29:04.634124 1435 tablet_copy_source_session.cc:205] T 72355dd734ea4aeba031171e8c39696d P 7d1e5c1b047443cbb28d7b0a9744babf: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:29:04.634734 2226 raft_consensus.cc:1261] T 3c88daaac3734d918955a774829532c6 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Refusing update from remote peer a62a935ff43d494ca35d0c417943133d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:29:04.635907 4425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72355dd734ea4aeba031171e8c39696d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:29:04.635946 3697 consensus_queue.cc:1035] T 3c88daaac3734d918955a774829532c6 P a62a935ff43d494ca35d0c417943133d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:29:04.636566 1685 raft_consensus.cc:1261] T 3c88daaac3734d918955a774829532c6 P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Refusing update from remote peer a62a935ff43d494ca35d0c417943133d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:29:04.637290 2087 raft_consensus.cc:1261] T 3c88daaac3734d918955a774829532c6 P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Refusing update from remote peer a62a935ff43d494ca35d0c417943133d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:29:04.639484 4437 consensus_queue.cc:1035] T 3c88daaac3734d918955a774829532c6 P a62a935ff43d494ca35d0c417943133d [LEADER]: Connected to new peer: Peer: permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:29:04.640617 1414 raft_consensus.cc:1261] T 3c88daaac3734d918955a774829532c6 P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Refusing update from remote peer a62a935ff43d494ca35d0c417943133d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:29:04.641137 4437 consensus_queue.cc:1035] T 3c88daaac3734d918955a774829532c6 P a62a935ff43d494ca35d0c417943133d [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:29:04.642740 3697 consensus_queue.cc:1035] T 3c88daaac3734d918955a774829532c6 P a62a935ff43d494ca35d0c417943133d [LEADER]: Connected to new peer: Peer: permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:29:04.644037 4437 raft_consensus.cc:2932] T 3c88daaac3734d918955a774829532c6 P a62a935ff43d494ca35d0c417943133d [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER d8d7297186694f77bb98de9174e4279c (127.15.255.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } attrs { replace: true } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: true } } }
I20240502 21:29:04.650070 1255 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 3c88daaac3734d918955a774829532c6 with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240502 21:29:04.649746 1685 raft_consensus.cc:2932] T 3c88daaac3734d918955a774829532c6 P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER d8d7297186694f77bb98de9174e4279c (127.15.255.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } attrs { replace: true } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: true } } }
I20240502 21:29:04.651608 2087 raft_consensus.cc:2932] T 3c88daaac3734d918955a774829532c6 P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER d8d7297186694f77bb98de9174e4279c (127.15.255.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } attrs { replace: true } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: true } } }
I20240502 21:29:04.656147 4455 ts_tablet_manager.cc:902] T 1aa57620996e42b6adf941e9d606f7e2 P d8d7297186694f77bb98de9174e4279c: Initiating tablet copy from peer 521bb93035324cecbebed2f6f2a630f8 (127.15.255.134:40821)
I20240502 21:29:04.657632 1414 raft_consensus.cc:2932] T 3c88daaac3734d918955a774829532c6 P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER d8d7297186694f77bb98de9174e4279c (127.15.255.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } attrs { replace: true } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: true } } }
I20240502 21:29:04.657814 1954 raft_consensus.cc:2932] T 3c88daaac3734d918955a774829532c6 P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER d8d7297186694f77bb98de9174e4279c (127.15.255.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } attrs { replace: true } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: true } } }
I20240502 21:29:04.661614 2226 raft_consensus.cc:2932] T 3c88daaac3734d918955a774829532c6 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER d8d7297186694f77bb98de9174e4279c (127.15.255.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } attrs { replace: true } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: true } } }
I20240502 21:29:04.662099 1270 catalog_manager.cc:5484] T 3c88daaac3734d918955a774829532c6 P a62a935ff43d494ca35d0c417943133d reported cstate change: config changed from index 4 to 5, NON_VOTER d8d7297186694f77bb98de9174e4279c (127.15.255.136) added. New cstate: current_term: 1 leader_uuid: "a62a935ff43d494ca35d0c417943133d" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 21:29:04.674103 4425 tablet_copy_client.cc:783] T 72355dd734ea4aeba031171e8c39696d P d8d7297186694f77bb98de9174e4279c: tablet copy: Starting download of 0 data blocks...
I20240502 21:29:04.674425 4425 tablet_copy_client.cc:647] T 72355dd734ea4aeba031171e8c39696d P d8d7297186694f77bb98de9174e4279c: tablet copy: Starting download of 1 WAL segments...
I20240502 21:29:04.677284 4425 tablet_copy_client.cc:523] T 72355dd734ea4aeba031171e8c39696d P d8d7297186694f77bb98de9174e4279c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:29:04.688731 4425 tablet_bootstrap.cc:492] T 72355dd734ea4aeba031171e8c39696d P d8d7297186694f77bb98de9174e4279c: Bootstrap starting.
I20240502 21:29:04.693496 4422 tablet_copy_client.cc:783] T 13af1b96121c492c9373f9acbb55725a P d8d7297186694f77bb98de9174e4279c: tablet copy: Starting download of 0 data blocks...
I20240502 21:29:04.695250 4422 tablet_copy_client.cc:647] T 13af1b96121c492c9373f9acbb55725a P d8d7297186694f77bb98de9174e4279c: tablet copy: Starting download of 1 WAL segments...
I20240502 21:29:04.697216 1818 raft_consensus.cc:1261] T 3c88daaac3734d918955a774829532c6 P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Refusing update from remote peer a62a935ff43d494ca35d0c417943133d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:29:04.697660 4422 tablet_copy_client.cc:523] T 13af1b96121c492c9373f9acbb55725a P d8d7297186694f77bb98de9174e4279c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:29:04.699862 4443 consensus_queue.cc:1035] T 3c88daaac3734d918955a774829532c6 P a62a935ff43d494ca35d0c417943133d [LEADER]: Connected to new peer: Peer: permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:29:04.701587 4455 tablet_copy_client.cc:308] T 1aa57620996e42b6adf941e9d606f7e2 P d8d7297186694f77bb98de9174e4279c: tablet copy: Beginning tablet copy session from remote peer at address 127.15.255.134:40821
I20240502 21:29:04.701531 1818 raft_consensus.cc:2932] T 3c88daaac3734d918955a774829532c6 P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER d8d7297186694f77bb98de9174e4279c (127.15.255.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } attrs { replace: true } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: true } } }
I20240502 21:29:04.703380 4422 tablet_bootstrap.cc:492] T 13af1b96121c492c9373f9acbb55725a P d8d7297186694f77bb98de9174e4279c: Bootstrap starting.
I20240502 21:29:04.715523 2113 tablet_copy_service.cc:140] P 521bb93035324cecbebed2f6f2a630f8: Received BeginTabletCopySession request for tablet 1aa57620996e42b6adf941e9d606f7e2 from peer d8d7297186694f77bb98de9174e4279c ({username='slave'} at 127.15.255.136:49789)
I20240502 21:29:04.715767 2113 tablet_copy_service.cc:161] P 521bb93035324cecbebed2f6f2a630f8: Beginning new tablet copy session on tablet 1aa57620996e42b6adf941e9d606f7e2 from peer d8d7297186694f77bb98de9174e4279c at {username='slave'} at 127.15.255.136:49789: session id = d8d7297186694f77bb98de9174e4279c-1aa57620996e42b6adf941e9d606f7e2
I20240502 21:29:04.718154 4422 log.cc:824] T 13af1b96121c492c9373f9acbb55725a P d8d7297186694f77bb98de9174e4279c: Log is configured to *not* fsync() on all Append() calls
I20240502 21:29:04.719719 2113 tablet_copy_source_session.cc:205] T 1aa57620996e42b6adf941e9d606f7e2 P 521bb93035324cecbebed2f6f2a630f8: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:29:04.730309 4455 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1aa57620996e42b6adf941e9d606f7e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:29:04.738106 4455 tablet_copy_client.cc:783] T 1aa57620996e42b6adf941e9d606f7e2 P d8d7297186694f77bb98de9174e4279c: tablet copy: Starting download of 0 data blocks...
I20240502 21:29:04.738469 4455 tablet_copy_client.cc:647] T 1aa57620996e42b6adf941e9d606f7e2 P d8d7297186694f77bb98de9174e4279c: tablet copy: Starting download of 1 WAL segments...
I20240502 21:29:04.744750 4455 tablet_copy_client.cc:523] T 1aa57620996e42b6adf941e9d606f7e2 P d8d7297186694f77bb98de9174e4279c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:29:04.749317 4455 tablet_bootstrap.cc:492] T 1aa57620996e42b6adf941e9d606f7e2 P d8d7297186694f77bb98de9174e4279c: Bootstrap starting.
I20240502 21:29:04.765628 4453 ts_tablet_manager.cc:902] T f8bee9bdcbff404280f000bdaeadf465 P d8d7297186694f77bb98de9174e4279c: Initiating tablet copy from peer 7d1e5c1b047443cbb28d7b0a9744babf (127.15.255.129:37897)
W20240502 21:29:04.772751 1482 consensus_peers.cc:485] T 3c88daaac3734d918955a774829532c6 P a62a935ff43d494ca35d0c417943133d -> Peer d8d7297186694f77bb98de9174e4279c (127.15.255.136:41459): Couldn't send request to peer d8d7297186694f77bb98de9174e4279c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3c88daaac3734d918955a774829532c6. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:29:04.775487 1954 consensus_queue.cc:237] T 4dad8773f5b54efa9ff527252bac24aa P e632c553ff4341328acf1cbaca678392 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:29:04.781241 1685 raft_consensus.cc:1261] T 4dad8773f5b54efa9ff527252bac24aa P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Refusing update from remote peer e632c553ff4341328acf1cbaca678392: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:29:04.781778 1550 raft_consensus.cc:1261] T 4dad8773f5b54efa9ff527252bac24aa P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Refusing update from remote peer e632c553ff4341328acf1cbaca678392: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:29:04.781877 4173 consensus_queue.cc:1035] T 4dad8773f5b54efa9ff527252bac24aa P e632c553ff4341328acf1cbaca678392 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:04.782495 4173 consensus_queue.cc:1035] T 4dad8773f5b54efa9ff527252bac24aa P e632c553ff4341328acf1cbaca678392 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:04.782629 2087 raft_consensus.cc:1261] T 4dad8773f5b54efa9ff527252bac24aa P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Refusing update from remote peer e632c553ff4341328acf1cbaca678392: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:29:04.783279 4313 consensus_queue.cc:1035] T 4dad8773f5b54efa9ff527252bac24aa P e632c553ff4341328acf1cbaca678392 [LEADER]: Connected to new peer: Peer: permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:04.783355 1414 raft_consensus.cc:1261] T 4dad8773f5b54efa9ff527252bac24aa P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Refusing update from remote peer e632c553ff4341328acf1cbaca678392: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:29:04.784622 2226 raft_consensus.cc:1261] T 4dad8773f5b54efa9ff527252bac24aa P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Refusing update from remote peer e632c553ff4341328acf1cbaca678392: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:29:04.786270 4313 consensus_queue.cc:1035] T 4dad8773f5b54efa9ff527252bac24aa P e632c553ff4341328acf1cbaca678392 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:04.788125 4313 consensus_queue.cc:1035] T 4dad8773f5b54efa9ff527252bac24aa P e632c553ff4341328acf1cbaca678392 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:04.789444 1818 raft_consensus.cc:1261] T 4dad8773f5b54efa9ff527252bac24aa P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Refusing update from remote peer e632c553ff4341328acf1cbaca678392: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:29:04.797039 4173 consensus_queue.cc:1035] T 4dad8773f5b54efa9ff527252bac24aa P e632c553ff4341328acf1cbaca678392 [LEADER]: Connected to new peer: Peer: permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:04.799954 4469 ts_tablet_manager.cc:902] T bcddb146822e4573837a5697d74134f7 P 16a97c1394f346f0b987f97cce7371b6: Initiating tablet copy from peer 6ab8df3c796f4b6cb3e29d5045ad3ea0 (127.15.255.135:34701)
I20240502 21:29:04.799764 4313 raft_consensus.cc:2932] T 4dad8773f5b54efa9ff527252bac24aa P e632c553ff4341328acf1cbaca678392 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } }
I20240502 21:29:04.802807 1684 raft_consensus.cc:2932] T 4dad8773f5b54efa9ff527252bac24aa P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } }
I20240502 21:29:04.803015 1550 raft_consensus.cc:2932] T 4dad8773f5b54efa9ff527252bac24aa P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } }
I20240502 21:29:04.808771 1265 catalog_manager.cc:5484] T 4dad8773f5b54efa9ff527252bac24aa P 16b3a337bc5e462f83ba895ad8660deb reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "e632c553ff4341328acf1cbaca678392" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } }
I20240502 21:29:04.828049 1414 raft_consensus.cc:2932] T 4dad8773f5b54efa9ff527252bac24aa P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } }
I20240502 21:29:04.833514 2087 raft_consensus.cc:2932] T 4dad8773f5b54efa9ff527252bac24aa P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } }
I20240502 21:29:04.844918 1954 consensus_queue.cc:237] T 4dad8773f5b54efa9ff527252bac24aa P e632c553ff4341328acf1cbaca678392 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: NON_VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: true } }
I20240502 21:29:04.851151 1685 raft_consensus.cc:1261] T 4dad8773f5b54efa9ff527252bac24aa P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Refusing update from remote peer e632c553ff4341328acf1cbaca678392: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:29:04.851810 4313 consensus_queue.cc:1035] T 4dad8773f5b54efa9ff527252bac24aa P e632c553ff4341328acf1cbaca678392 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:29:04.854804 1550 raft_consensus.cc:1261] T 4dad8773f5b54efa9ff527252bac24aa P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Refusing update from remote peer e632c553ff4341328acf1cbaca678392: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:29:04.855304 1414 raft_consensus.cc:1261] T 4dad8773f5b54efa9ff527252bac24aa P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Refusing update from remote peer e632c553ff4341328acf1cbaca678392: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:29:04.856034 4173 consensus_queue.cc:1035] T 4dad8773f5b54efa9ff527252bac24aa P e632c553ff4341328acf1cbaca678392 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:29:04.856560 4173 consensus_queue.cc:1035] T 4dad8773f5b54efa9ff527252bac24aa P e632c553ff4341328acf1cbaca678392 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:29:04.862663 4313 raft_consensus.cc:2932] T 4dad8773f5b54efa9ff527252bac24aa P e632c553ff4341328acf1cbaca678392 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 16a97c1394f346f0b987f97cce7371b6 (127.15.255.137) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: NON_VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: true } } }
I20240502 21:29:04.867247 1550 raft_consensus.cc:2932] T 4dad8773f5b54efa9ff527252bac24aa P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 16a97c1394f346f0b987f97cce7371b6 (127.15.255.137) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: NON_VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: true } } }
I20240502 21:29:04.875162 1254 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 4dad8773f5b54efa9ff527252bac24aa with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240502 21:29:04.876087 1265 catalog_manager.cc:5484] T 4dad8773f5b54efa9ff527252bac24aa P a62a935ff43d494ca35d0c417943133d reported cstate change: config changed from index 4 to 5, NON_VOTER 16a97c1394f346f0b987f97cce7371b6 (127.15.255.137) added. New cstate: current_term: 1 leader_uuid: "e632c553ff4341328acf1cbaca678392" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: NON_VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: true } } }
I20240502 21:29:04.888468 2087 raft_consensus.cc:1261] T 4dad8773f5b54efa9ff527252bac24aa P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Refusing update from remote peer e632c553ff4341328acf1cbaca678392: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:29:04.894754 1819 raft_consensus.cc:1261] T 4dad8773f5b54efa9ff527252bac24aa P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Refusing update from remote peer e632c553ff4341328acf1cbaca678392: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:29:04.895622 2225 raft_consensus.cc:1261] T 4dad8773f5b54efa9ff527252bac24aa P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Refusing update from remote peer e632c553ff4341328acf1cbaca678392: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:29:04.897071 1685 raft_consensus.cc:2932] T 4dad8773f5b54efa9ff527252bac24aa P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 16a97c1394f346f0b987f97cce7371b6 (127.15.255.137) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: NON_VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: true } } }
I20240502 21:29:04.903111 4173 consensus_queue.cc:1035] T 4dad8773f5b54efa9ff527252bac24aa P e632c553ff4341328acf1cbaca678392 [LEADER]: Connected to new peer: Peer: permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:29:04.903590 4173 consensus_queue.cc:1035] T 4dad8773f5b54efa9ff527252bac24aa P e632c553ff4341328acf1cbaca678392 [LEADER]: Connected to new peer: Peer: permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:04.904006 4173 consensus_queue.cc:1035] T 4dad8773f5b54efa9ff527252bac24aa P e632c553ff4341328acf1cbaca678392 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:04.904464 4453 tablet_copy_client.cc:308] T f8bee9bdcbff404280f000bdaeadf465 P d8d7297186694f77bb98de9174e4279c: tablet copy: Beginning tablet copy session from remote peer at address 127.15.255.129:37897
I20240502 21:29:04.904994 1819 raft_consensus.cc:2932] T 4dad8773f5b54efa9ff527252bac24aa P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } }
I20240502 21:29:04.909140 1435 tablet_copy_service.cc:140] P 7d1e5c1b047443cbb28d7b0a9744babf: Received BeginTabletCopySession request for tablet f8bee9bdcbff404280f000bdaeadf465 from peer d8d7297186694f77bb98de9174e4279c ({username='slave'} at 127.15.255.136:40527)
I20240502 21:29:04.909329 1435 tablet_copy_service.cc:161] P 7d1e5c1b047443cbb28d7b0a9744babf: Beginning new tablet copy session on tablet f8bee9bdcbff404280f000bdaeadf465 from peer d8d7297186694f77bb98de9174e4279c at {username='slave'} at 127.15.255.136:40527: session id = d8d7297186694f77bb98de9174e4279c-f8bee9bdcbff404280f000bdaeadf465
I20240502 21:29:04.913826 2087 raft_consensus.cc:2932] T 4dad8773f5b54efa9ff527252bac24aa P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 16a97c1394f346f0b987f97cce7371b6 (127.15.255.137) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: NON_VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: true } } }
I20240502 21:29:04.919307 1819 raft_consensus.cc:2932] T 4dad8773f5b54efa9ff527252bac24aa P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 16a97c1394f346f0b987f97cce7371b6 (127.15.255.137) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: NON_VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: true } } }
I20240502 21:29:04.928494 1435 tablet_copy_source_session.cc:205] T f8bee9bdcbff404280f000bdaeadf465 P 7d1e5c1b047443cbb28d7b0a9744babf: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:29:04.935926 4453 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8bee9bdcbff404280f000bdaeadf465. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:29:04.943976 4453 tablet_copy_client.cc:783] T f8bee9bdcbff404280f000bdaeadf465 P d8d7297186694f77bb98de9174e4279c: tablet copy: Starting download of 0 data blocks...
W20240502 21:29:04.949013 1885 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.135:34701, user_credentials={real_user=slave}} blocked reactor thread for 46588.9us
I20240502 21:29:04.950222 1414 raft_consensus.cc:2932] T 4dad8773f5b54efa9ff527252bac24aa P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 16a97c1394f346f0b987f97cce7371b6 (127.15.255.137) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: NON_VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: true } } }
I20240502 21:29:04.950417 2225 raft_consensus.cc:2932] T 4dad8773f5b54efa9ff527252bac24aa P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } }
I20240502 21:29:04.954804 2225 raft_consensus.cc:2932] T 4dad8773f5b54efa9ff527252bac24aa P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 16a97c1394f346f0b987f97cce7371b6 (127.15.255.137) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: NON_VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: true } } }
I20240502 21:29:04.960652 4453 tablet_copy_client.cc:647] T f8bee9bdcbff404280f000bdaeadf465 P d8d7297186694f77bb98de9174e4279c: tablet copy: Starting download of 1 WAL segments...
I20240502 21:29:04.966393 4453 tablet_copy_client.cc:523] T f8bee9bdcbff404280f000bdaeadf465 P d8d7297186694f77bb98de9174e4279c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
W20240502 21:29:04.973536 1884 consensus_peers.cc:485] T 4dad8773f5b54efa9ff527252bac24aa P e632c553ff4341328acf1cbaca678392 -> Peer 16a97c1394f346f0b987f97cce7371b6 (127.15.255.137:43717): Couldn't send request to peer 16a97c1394f346f0b987f97cce7371b6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4dad8773f5b54efa9ff527252bac24aa. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:29:04.974833 4422 tablet_bootstrap.cc:492] T 13af1b96121c492c9373f9acbb55725a P d8d7297186694f77bb98de9174e4279c: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:29:04.975392 4422 tablet_bootstrap.cc:492] T 13af1b96121c492c9373f9acbb55725a P d8d7297186694f77bb98de9174e4279c: Bootstrap complete.
I20240502 21:29:04.975883 4422 ts_tablet_manager.cc:1367] T 13af1b96121c492c9373f9acbb55725a P d8d7297186694f77bb98de9174e4279c: Time spent bootstrapping tablet: real 0.273s user 0.065s sys 0.006s
I20240502 21:29:04.975975 4453 tablet_bootstrap.cc:492] T f8bee9bdcbff404280f000bdaeadf465 P d8d7297186694f77bb98de9174e4279c: Bootstrap starting.
I20240502 21:29:04.982460 4469 tablet_copy_client.cc:308] T bcddb146822e4573837a5697d74134f7 P 16a97c1394f346f0b987f97cce7371b6: tablet copy: Beginning tablet copy session from remote peer at address 127.15.255.135:34701
I20240502 21:29:04.999815 4455 tablet_bootstrap.cc:492] T 1aa57620996e42b6adf941e9d606f7e2 P d8d7297186694f77bb98de9174e4279c: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:29:05.000236 4455 tablet_bootstrap.cc:492] T 1aa57620996e42b6adf941e9d606f7e2 P d8d7297186694f77bb98de9174e4279c: Bootstrap complete.
I20240502 21:29:05.000542 4455 ts_tablet_manager.cc:1367] T 1aa57620996e42b6adf941e9d606f7e2 P d8d7297186694f77bb98de9174e4279c: Time spent bootstrapping tablet: real 0.251s user 0.045s sys 0.012s
I20240502 21:29:05.003978 4425 tablet_bootstrap.cc:492] T 72355dd734ea4aeba031171e8c39696d P d8d7297186694f77bb98de9174e4279c: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:29:05.004853 4425 tablet_bootstrap.cc:492] T 72355dd734ea4aeba031171e8c39696d P d8d7297186694f77bb98de9174e4279c: Bootstrap complete.
I20240502 21:29:05.005117 4425 ts_tablet_manager.cc:1367] T 72355dd734ea4aeba031171e8c39696d P d8d7297186694f77bb98de9174e4279c: Time spent bootstrapping tablet: real 0.317s user 0.048s sys 0.004s
I20240502 21:29:05.016523 4493 ts_tablet_manager.cc:902] T 8f86d1517df74ead9e397ca9bed44c3f P 1fab5794cf1740288abd82f2cebb9b1a: Initiating tablet copy from peer 521bb93035324cecbebed2f6f2a630f8 (127.15.255.134:40821)
I20240502 21:29:05.018522 4422 raft_consensus.cc:348] T 13af1b96121c492c9373f9acbb55725a P d8d7297186694f77bb98de9174e4279c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: true } }
I20240502 21:29:05.019088 4422 raft_consensus.cc:729] T 13af1b96121c492c9373f9acbb55725a P d8d7297186694f77bb98de9174e4279c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d8d7297186694f77bb98de9174e4279c, State: Initialized, Role: LEARNER
I20240502 21:29:05.019667 4422 consensus_queue.cc:260] T 13af1b96121c492c9373f9acbb55725a P d8d7297186694f77bb98de9174e4279c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: true } }
I20240502 21:29:05.021046 4425 raft_consensus.cc:348] T 72355dd734ea4aeba031171e8c39696d P d8d7297186694f77bb98de9174e4279c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } attrs { replace: true } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: true } }
I20240502 21:29:05.021340 4425 raft_consensus.cc:729] T 72355dd734ea4aeba031171e8c39696d P d8d7297186694f77bb98de9174e4279c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d8d7297186694f77bb98de9174e4279c, State: Initialized, Role: LEARNER
I20240502 21:29:05.022564 4425 consensus_queue.cc:260] T 72355dd734ea4aeba031171e8c39696d P d8d7297186694f77bb98de9174e4279c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } attrs { replace: true } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: true } }
I20240502 21:29:05.024590 4455 raft_consensus.cc:348] T 1aa57620996e42b6adf941e9d606f7e2 P d8d7297186694f77bb98de9174e4279c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } attrs { replace: true } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: true } }
I20240502 21:29:05.024897 4455 raft_consensus.cc:729] T 1aa57620996e42b6adf941e9d606f7e2 P d8d7297186694f77bb98de9174e4279c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d8d7297186694f77bb98de9174e4279c, State: Initialized, Role: LEARNER
I20240502 21:29:05.025120 4455 consensus_queue.cc:260] T 1aa57620996e42b6adf941e9d606f7e2 P d8d7297186694f77bb98de9174e4279c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } attrs { replace: true } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: true } }
I20240502 21:29:05.038647 2087 consensus_queue.cc:237] T e45af71e3fc14c579b32ba4dbe2adf78 P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } attrs { replace: true } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:29:05.038990 4422 ts_tablet_manager.cc:1398] T 13af1b96121c492c9373f9acbb55725a P d8d7297186694f77bb98de9174e4279c: Time spent starting tablet: real 0.063s user 0.012s sys 0.000s
I20240502 21:29:05.040874 1840 tablet_copy_service.cc:342] P 153026dd7f8943cc93fedf6f48450fb4: Request end of tablet copy session d8d7297186694f77bb98de9174e4279c-13af1b96121c492c9373f9acbb55725a received from {username='slave'} at 127.15.255.136:41357
I20240502 21:29:05.041052 1840 tablet_copy_service.cc:433] P 153026dd7f8943cc93fedf6f48450fb4: ending tablet copy session d8d7297186694f77bb98de9174e4279c-13af1b96121c492c9373f9acbb55725a on tablet 13af1b96121c492c9373f9acbb55725a with peer d8d7297186694f77bb98de9174e4279c
I20240502 21:29:05.041447 3855 raft_consensus.cc:1203] T 13af1b96121c492c9373f9acbb55725a P d8d7297186694f77bb98de9174e4279c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240502 21:29:05.041864 4455 ts_tablet_manager.cc:1398] T 1aa57620996e42b6adf941e9d606f7e2 P d8d7297186694f77bb98de9174e4279c: Time spent starting tablet: real 0.041s user 0.001s sys 0.003s
I20240502 21:29:05.044627 2113 tablet_copy_service.cc:342] P 521bb93035324cecbebed2f6f2a630f8: Request end of tablet copy session d8d7297186694f77bb98de9174e4279c-1aa57620996e42b6adf941e9d606f7e2 received from {username='slave'} at 127.15.255.136:49789
I20240502 21:29:05.044777 2113 tablet_copy_service.cc:433] P 521bb93035324cecbebed2f6f2a630f8: ending tablet copy session d8d7297186694f77bb98de9174e4279c-1aa57620996e42b6adf941e9d606f7e2 on tablet 1aa57620996e42b6adf941e9d606f7e2 with peer d8d7297186694f77bb98de9174e4279c
I20240502 21:29:05.046352 4425 ts_tablet_manager.cc:1398] T 72355dd734ea4aeba031171e8c39696d P d8d7297186694f77bb98de9174e4279c: Time spent starting tablet: real 0.041s user 0.002s sys 0.002s
I20240502 21:29:05.048049 1435 tablet_copy_service.cc:342] P 7d1e5c1b047443cbb28d7b0a9744babf: Request end of tablet copy session d8d7297186694f77bb98de9174e4279c-72355dd734ea4aeba031171e8c39696d received from {username='slave'} at 127.15.255.136:40527
I20240502 21:29:05.048225 1435 tablet_copy_service.cc:433] P 7d1e5c1b047443cbb28d7b0a9744babf: ending tablet copy session d8d7297186694f77bb98de9174e4279c-72355dd734ea4aeba031171e8c39696d on tablet 72355dd734ea4aeba031171e8c39696d with peer d8d7297186694f77bb98de9174e4279c
I20240502 21:29:05.057206 1819 raft_consensus.cc:1261] T e45af71e3fc14c579b32ba4dbe2adf78 P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:29:05.057439 1414 raft_consensus.cc:1261] T e45af71e3fc14c579b32ba4dbe2adf78 P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:29:05.057994 1550 raft_consensus.cc:1261] T e45af71e3fc14c579b32ba4dbe2adf78 P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:29:05.058032 4383 consensus_queue.cc:1035] T e45af71e3fc14c579b32ba4dbe2adf78 P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:05.058485 2225 raft_consensus.cc:1261] T e45af71e3fc14c579b32ba4dbe2adf78 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:29:05.059823 3765 consensus_queue.cc:1035] T e45af71e3fc14c579b32ba4dbe2adf78 P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:05.060329 3765 consensus_queue.cc:1035] T e45af71e3fc14c579b32ba4dbe2adf78 P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:05.060869 3765 consensus_queue.cc:1035] T e45af71e3fc14c579b32ba4dbe2adf78 P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:05.061472 1685 raft_consensus.cc:1261] T e45af71e3fc14c579b32ba4dbe2adf78 P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:29:05.062013 3545 consensus_queue.cc:1035] T e45af71e3fc14c579b32ba4dbe2adf78 P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:05.083611 4422 ts_tablet_manager.cc:902] T 3c88daaac3734d918955a774829532c6 P d8d7297186694f77bb98de9174e4279c: Initiating tablet copy from peer a62a935ff43d494ca35d0c417943133d (127.15.255.130:37991)
I20240502 21:29:05.089274 3855 raft_consensus.cc:1203] T 72355dd734ea4aeba031171e8c39696d P d8d7297186694f77bb98de9174e4279c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240502 21:29:05.097115 3765 raft_consensus.cc:2932] T e45af71e3fc14c579b32ba4dbe2adf78 P 521bb93035324cecbebed2f6f2a630f8 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } attrs { replace: true } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } }
I20240502 21:29:05.098400 2226 raft_consensus.cc:2932] T e45af71e3fc14c579b32ba4dbe2adf78 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } attrs { replace: true } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } }
I20240502 21:29:05.101359 1819 raft_consensus.cc:2932] T e45af71e3fc14c579b32ba4dbe2adf78 P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } attrs { replace: true } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } }
I20240502 21:29:05.105800 1415 raft_consensus.cc:2932] T e45af71e3fc14c579b32ba4dbe2adf78 P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } attrs { replace: true } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } }
I20240502 21:29:05.109261 1265 catalog_manager.cc:5484] T e45af71e3fc14c579b32ba4dbe2adf78 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "521bb93035324cecbebed2f6f2a630f8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } attrs { replace: true } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } }
I20240502 21:29:05.116680 1954 raft_consensus.cc:1261] T e45af71e3fc14c579b32ba4dbe2adf78 P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:29:05.116604 1684 raft_consensus.cc:2932] T e45af71e3fc14c579b32ba4dbe2adf78 P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } attrs { replace: true } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } }
I20240502 21:29:05.117439 3545 consensus_queue.cc:1035] T e45af71e3fc14c579b32ba4dbe2adf78 P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:05.120182 1550 raft_consensus.cc:2932] T e45af71e3fc14c579b32ba4dbe2adf78 P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } attrs { replace: true } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } }
I20240502 21:29:05.128829 1954 raft_consensus.cc:2932] T e45af71e3fc14c579b32ba4dbe2adf78 P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } attrs { replace: true } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } }
I20240502 21:29:05.135654 2087 consensus_queue.cc:237] T e45af71e3fc14c579b32ba4dbe2adf78 P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } attrs { replace: true } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: NON_VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: true } }
I20240502 21:29:05.137845 2249 tablet_copy_service.cc:140] P 6ab8df3c796f4b6cb3e29d5045ad3ea0: Received BeginTabletCopySession request for tablet bcddb146822e4573837a5697d74134f7 from peer 16a97c1394f346f0b987f97cce7371b6 ({username='slave'} at 127.15.255.137:35291)
I20240502 21:29:05.138100 2249 tablet_copy_service.cc:161] P 6ab8df3c796f4b6cb3e29d5045ad3ea0: Beginning new tablet copy session on tablet bcddb146822e4573837a5697d74134f7 from peer 16a97c1394f346f0b987f97cce7371b6 at {username='slave'} at 127.15.255.137:35291: session id = 16a97c1394f346f0b987f97cce7371b6-bcddb146822e4573837a5697d74134f7
I20240502 21:29:05.141672 1818 raft_consensus.cc:1261] T e45af71e3fc14c579b32ba4dbe2adf78 P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:29:05.142223 2249 tablet_copy_source_session.cc:205] T bcddb146822e4573837a5697d74134f7 P 6ab8df3c796f4b6cb3e29d5045ad3ea0: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:29:05.142669 3765 consensus_queue.cc:1035] T e45af71e3fc14c579b32ba4dbe2adf78 P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:29:05.147979 4469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bcddb146822e4573837a5697d74134f7. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 21:29:05.155741 2022 consensus_peers.cc:485] T e45af71e3fc14c579b32ba4dbe2adf78 P 521bb93035324cecbebed2f6f2a630f8 -> Peer 16a97c1394f346f0b987f97cce7371b6 (127.15.255.137:43717): Couldn't send request to peer 16a97c1394f346f0b987f97cce7371b6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e45af71e3fc14c579b32ba4dbe2adf78. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:29:05.158567 2226 raft_consensus.cc:1261] T e45af71e3fc14c579b32ba4dbe2adf78 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:29:05.159472 4493 tablet_copy_client.cc:308] T 8f86d1517df74ead9e397ca9bed44c3f P 1fab5794cf1740288abd82f2cebb9b1a: tablet copy: Beginning tablet copy session from remote peer at address 127.15.255.134:40821
I20240502 21:29:05.159554 3765 consensus_queue.cc:1035] T e45af71e3fc14c579b32ba4dbe2adf78 P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:29:05.161813 1954 raft_consensus.cc:1261] T e45af71e3fc14c579b32ba4dbe2adf78 P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:29:05.162400 3765 consensus_queue.cc:1035] T e45af71e3fc14c579b32ba4dbe2adf78 P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:29:05.164794 4383 raft_consensus.cc:2932] T e45af71e3fc14c579b32ba4dbe2adf78 P 521bb93035324cecbebed2f6f2a630f8 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 16a97c1394f346f0b987f97cce7371b6 (127.15.255.137) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } attrs { replace: true } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: NON_VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: true } } }
I20240502 21:29:05.165428 1955 raft_consensus.cc:2932] T e45af71e3fc14c579b32ba4dbe2adf78 P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 16a97c1394f346f0b987f97cce7371b6 (127.15.255.137) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } attrs { replace: true } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: NON_VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: true } } }
I20240502 21:29:05.172168 1414 raft_consensus.cc:1261] T e45af71e3fc14c579b32ba4dbe2adf78 P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:29:05.172403 1265 catalog_manager.cc:5484] T e45af71e3fc14c579b32ba4dbe2adf78 P e632c553ff4341328acf1cbaca678392 reported cstate change: config changed from index 4 to 5, NON_VOTER 16a97c1394f346f0b987f97cce7371b6 (127.15.255.137) added. New cstate: current_term: 1 leader_uuid: "521bb93035324cecbebed2f6f2a630f8" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } attrs { replace: true } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: NON_VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: true } } }
I20240502 21:29:05.172909 1254 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet e45af71e3fc14c579b32ba4dbe2adf78 with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240502 21:29:05.173192 1550 raft_consensus.cc:1261] T e45af71e3fc14c579b32ba4dbe2adf78 P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:29:05.173722 4332 consensus_queue.cc:1035] T e45af71e3fc14c579b32ba4dbe2adf78 P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:29:05.174320 4332 consensus_queue.cc:1035] T e45af71e3fc14c579b32ba4dbe2adf78 P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:29:05.175608 1414 raft_consensus.cc:2932] T e45af71e3fc14c579b32ba4dbe2adf78 P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 16a97c1394f346f0b987f97cce7371b6 (127.15.255.137) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } attrs { replace: true } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: NON_VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: true } } }
I20240502 21:29:05.182507 1550 raft_consensus.cc:2932] T e45af71e3fc14c579b32ba4dbe2adf78 P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 16a97c1394f346f0b987f97cce7371b6 (127.15.255.137) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } attrs { replace: true } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: NON_VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: true } } }
I20240502 21:29:05.185048 1684 raft_consensus.cc:1261] T e45af71e3fc14c579b32ba4dbe2adf78 P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:29:05.185976 3545 consensus_queue.cc:1035] T e45af71e3fc14c579b32ba4dbe2adf78 P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:29:05.187479 1684 raft_consensus.cc:2932] T e45af71e3fc14c579b32ba4dbe2adf78 P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 16a97c1394f346f0b987f97cce7371b6 (127.15.255.137) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } attrs { replace: true } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: NON_VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: true } } }
I20240502 21:29:05.192862 2226 raft_consensus.cc:2932] T e45af71e3fc14c579b32ba4dbe2adf78 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 16a97c1394f346f0b987f97cce7371b6 (127.15.255.137) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } attrs { replace: true } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: NON_VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: true } } }
I20240502 21:29:05.197142 1818 raft_consensus.cc:2932] T e45af71e3fc14c579b32ba4dbe2adf78 P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 16a97c1394f346f0b987f97cce7371b6 (127.15.255.137) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } attrs { replace: true } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: NON_VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: true } } }
I20240502 21:29:05.206372 3855 raft_consensus.cc:1203] T 1aa57620996e42b6adf941e9d606f7e2 P d8d7297186694f77bb98de9174e4279c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240502 21:29:05.208009 2113 tablet_copy_service.cc:140] P 521bb93035324cecbebed2f6f2a630f8: Received BeginTabletCopySession request for tablet 8f86d1517df74ead9e397ca9bed44c3f from peer 1fab5794cf1740288abd82f2cebb9b1a ({username='slave'} at 127.15.255.138:37811)
I20240502 21:29:05.208163 2113 tablet_copy_service.cc:161] P 521bb93035324cecbebed2f6f2a630f8: Beginning new tablet copy session on tablet 8f86d1517df74ead9e397ca9bed44c3f from peer 1fab5794cf1740288abd82f2cebb9b1a at {username='slave'} at 127.15.255.138:37811: session id = 1fab5794cf1740288abd82f2cebb9b1a-8f86d1517df74ead9e397ca9bed44c3f
I20240502 21:29:05.210496 2113 tablet_copy_source_session.cc:205] T 8f86d1517df74ead9e397ca9bed44c3f P 521bb93035324cecbebed2f6f2a630f8: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:29:05.219623 4493 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f86d1517df74ead9e397ca9bed44c3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:29:05.238955 4493 tablet_copy_client.cc:783] T 8f86d1517df74ead9e397ca9bed44c3f P 1fab5794cf1740288abd82f2cebb9b1a: tablet copy: Starting download of 0 data blocks...
I20240502 21:29:05.239370 4493 tablet_copy_client.cc:647] T 8f86d1517df74ead9e397ca9bed44c3f P 1fab5794cf1740288abd82f2cebb9b1a: tablet copy: Starting download of 1 WAL segments...
I20240502 21:29:05.241765 4493 tablet_copy_client.cc:523] T 8f86d1517df74ead9e397ca9bed44c3f P 1fab5794cf1740288abd82f2cebb9b1a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:29:05.246188 4493 tablet_bootstrap.cc:492] T 8f86d1517df74ead9e397ca9bed44c3f P 1fab5794cf1740288abd82f2cebb9b1a: Bootstrap starting.
I20240502 21:29:05.246856 4469 tablet_copy_client.cc:783] T bcddb146822e4573837a5697d74134f7 P 16a97c1394f346f0b987f97cce7371b6: tablet copy: Starting download of 0 data blocks...
I20240502 21:29:05.247208 4469 tablet_copy_client.cc:647] T bcddb146822e4573837a5697d74134f7 P 16a97c1394f346f0b987f97cce7371b6: tablet copy: Starting download of 1 WAL segments...
I20240502 21:29:05.249996 4469 tablet_copy_client.cc:523] T bcddb146822e4573837a5697d74134f7 P 16a97c1394f346f0b987f97cce7371b6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:29:05.251602 4422 tablet_copy_client.cc:308] T 3c88daaac3734d918955a774829532c6 P d8d7297186694f77bb98de9174e4279c: tablet copy: Beginning tablet copy session from remote peer at address 127.15.255.130:37991
I20240502 21:29:05.268843 4493 log.cc:824] T 8f86d1517df74ead9e397ca9bed44c3f P 1fab5794cf1740288abd82f2cebb9b1a: Log is configured to *not* fsync() on all Append() calls
I20240502 21:29:05.272987 1570 tablet_copy_service.cc:140] P a62a935ff43d494ca35d0c417943133d: Received BeginTabletCopySession request for tablet 3c88daaac3734d918955a774829532c6 from peer d8d7297186694f77bb98de9174e4279c ({username='slave'} at 127.15.255.136:51073)
I20240502 21:29:05.273253 1570 tablet_copy_service.cc:161] P a62a935ff43d494ca35d0c417943133d: Beginning new tablet copy session on tablet 3c88daaac3734d918955a774829532c6 from peer d8d7297186694f77bb98de9174e4279c at {username='slave'} at 127.15.255.136:51073: session id = d8d7297186694f77bb98de9174e4279c-3c88daaac3734d918955a774829532c6
I20240502 21:29:05.277361 1570 tablet_copy_source_session.cc:205] T 3c88daaac3734d918955a774829532c6 P a62a935ff43d494ca35d0c417943133d: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:29:05.279534 4422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c88daaac3734d918955a774829532c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:29:05.290596 4453 tablet_bootstrap.cc:492] T f8bee9bdcbff404280f000bdaeadf465 P d8d7297186694f77bb98de9174e4279c: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:29:05.291002 4453 tablet_bootstrap.cc:492] T f8bee9bdcbff404280f000bdaeadf465 P d8d7297186694f77bb98de9174e4279c: Bootstrap complete.
I20240502 21:29:05.291357 4453 ts_tablet_manager.cc:1367] T f8bee9bdcbff404280f000bdaeadf465 P d8d7297186694f77bb98de9174e4279c: Time spent bootstrapping tablet: real 0.316s user 0.084s sys 0.043s
I20240502 21:29:05.293545 4469 tablet_bootstrap.cc:492] T bcddb146822e4573837a5697d74134f7 P 16a97c1394f346f0b987f97cce7371b6: Bootstrap starting.
I20240502 21:29:05.295624 4422 tablet_copy_client.cc:783] T 3c88daaac3734d918955a774829532c6 P d8d7297186694f77bb98de9174e4279c: tablet copy: Starting download of 0 data blocks...
I20240502 21:29:05.296277 4422 tablet_copy_client.cc:647] T 3c88daaac3734d918955a774829532c6 P d8d7297186694f77bb98de9174e4279c: tablet copy: Starting download of 1 WAL segments...
I20240502 21:29:05.302320 4422 tablet_copy_client.cc:523] T 3c88daaac3734d918955a774829532c6 P d8d7297186694f77bb98de9174e4279c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:29:05.304975 4453 raft_consensus.cc:348] T f8bee9bdcbff404280f000bdaeadf465 P d8d7297186694f77bb98de9174e4279c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: true } }
I20240502 21:29:05.305310 4453 raft_consensus.cc:729] T f8bee9bdcbff404280f000bdaeadf465 P d8d7297186694f77bb98de9174e4279c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d8d7297186694f77bb98de9174e4279c, State: Initialized, Role: LEARNER
I20240502 21:29:05.305550 4453 consensus_queue.cc:260] T f8bee9bdcbff404280f000bdaeadf465 P d8d7297186694f77bb98de9174e4279c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: true } }
I20240502 21:29:05.309361 4422 tablet_bootstrap.cc:492] T 3c88daaac3734d918955a774829532c6 P d8d7297186694f77bb98de9174e4279c: Bootstrap starting.
I20240502 21:29:05.310461 4453 ts_tablet_manager.cc:1398] T f8bee9bdcbff404280f000bdaeadf465 P d8d7297186694f77bb98de9174e4279c: Time spent starting tablet: real 0.019s user 0.002s sys 0.000s
I20240502 21:29:05.314086 1435 tablet_copy_service.cc:342] P 7d1e5c1b047443cbb28d7b0a9744babf: Request end of tablet copy session d8d7297186694f77bb98de9174e4279c-f8bee9bdcbff404280f000bdaeadf465 received from {username='slave'} at 127.15.255.136:40527
I20240502 21:29:05.314226 1435 tablet_copy_service.cc:433] P 7d1e5c1b047443cbb28d7b0a9744babf: ending tablet copy session d8d7297186694f77bb98de9174e4279c-f8bee9bdcbff404280f000bdaeadf465 on tablet f8bee9bdcbff404280f000bdaeadf465 with peer d8d7297186694f77bb98de9174e4279c
I20240502 21:29:05.318490 4538 ts_tablet_manager.cc:902] T 4dad8773f5b54efa9ff527252bac24aa P 16a97c1394f346f0b987f97cce7371b6: Initiating tablet copy from peer e632c553ff4341328acf1cbaca678392 (127.15.255.133:37439)
W20240502 21:29:05.374917 2158 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.134:40821, user_credentials={real_user=slave}} blocked reactor thread for 45374.9us
I20240502 21:29:05.380043 4469 log.cc:824] T bcddb146822e4573837a5697d74134f7 P 16a97c1394f346f0b987f97cce7371b6: Log is configured to *not* fsync() on all Append() calls
I20240502 21:29:05.382092 4538 tablet_copy_client.cc:308] T 4dad8773f5b54efa9ff527252bac24aa P 16a97c1394f346f0b987f97cce7371b6: tablet copy: Beginning tablet copy session from remote peer at address 127.15.255.133:37439
I20240502 21:29:05.388401 4493 tablet_bootstrap.cc:492] T 8f86d1517df74ead9e397ca9bed44c3f P 1fab5794cf1740288abd82f2cebb9b1a: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:29:05.389019 4493 tablet_bootstrap.cc:492] T 8f86d1517df74ead9e397ca9bed44c3f P 1fab5794cf1740288abd82f2cebb9b1a: Bootstrap complete.
I20240502 21:29:05.389567 4493 ts_tablet_manager.cc:1367] T 8f86d1517df74ead9e397ca9bed44c3f P 1fab5794cf1740288abd82f2cebb9b1a: Time spent bootstrapping tablet: real 0.144s user 0.063s sys 0.014s
I20240502 21:29:05.411868 1975 tablet_copy_service.cc:140] P e632c553ff4341328acf1cbaca678392: Received BeginTabletCopySession request for tablet 4dad8773f5b54efa9ff527252bac24aa from peer 16a97c1394f346f0b987f97cce7371b6 ({username='slave'} at 127.15.255.137:38165)
I20240502 21:29:05.412117 1975 tablet_copy_service.cc:161] P e632c553ff4341328acf1cbaca678392: Beginning new tablet copy session on tablet 4dad8773f5b54efa9ff527252bac24aa from peer 16a97c1394f346f0b987f97cce7371b6 at {username='slave'} at 127.15.255.137:38165: session id = 16a97c1394f346f0b987f97cce7371b6-4dad8773f5b54efa9ff527252bac24aa
I20240502 21:29:05.414647 4493 raft_consensus.cc:348] T 8f86d1517df74ead9e397ca9bed44c3f P 1fab5794cf1740288abd82f2cebb9b1a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: NON_VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: true } }
I20240502 21:29:05.415203 4493 raft_consensus.cc:729] T 8f86d1517df74ead9e397ca9bed44c3f P 1fab5794cf1740288abd82f2cebb9b1a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1fab5794cf1740288abd82f2cebb9b1a, State: Initialized, Role: LEARNER
I20240502 21:29:05.415769 1975 tablet_copy_source_session.cc:205] T 4dad8773f5b54efa9ff527252bac24aa P e632c553ff4341328acf1cbaca678392: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:29:05.415779 4493 consensus_queue.cc:260] T 8f86d1517df74ead9e397ca9bed44c3f P 1fab5794cf1740288abd82f2cebb9b1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: NON_VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: true } }
I20240502 21:29:05.417703 4538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4dad8773f5b54efa9ff527252bac24aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:29:05.427121 4538 tablet_copy_client.cc:783] T 4dad8773f5b54efa9ff527252bac24aa P 16a97c1394f346f0b987f97cce7371b6: tablet copy: Starting download of 0 data blocks...
I20240502 21:29:05.427562 4538 tablet_copy_client.cc:647] T 4dad8773f5b54efa9ff527252bac24aa P 16a97c1394f346f0b987f97cce7371b6: tablet copy: Starting download of 1 WAL segments...
I20240502 21:29:05.428650 4493 ts_tablet_manager.cc:1398] T 8f86d1517df74ead9e397ca9bed44c3f P 1fab5794cf1740288abd82f2cebb9b1a: Time spent starting tablet: real 0.039s user 0.002s sys 0.011s
I20240502 21:29:05.429463 4538 tablet_copy_client.cc:523] T 4dad8773f5b54efa9ff527252bac24aa P 16a97c1394f346f0b987f97cce7371b6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:29:05.434664 2113 tablet_copy_service.cc:342] P 521bb93035324cecbebed2f6f2a630f8: Request end of tablet copy session 1fab5794cf1740288abd82f2cebb9b1a-8f86d1517df74ead9e397ca9bed44c3f received from {username='slave'} at 127.15.255.138:37811
I20240502 21:29:05.434790 2113 tablet_copy_service.cc:433] P 521bb93035324cecbebed2f6f2a630f8: ending tablet copy session 1fab5794cf1740288abd82f2cebb9b1a-8f86d1517df74ead9e397ca9bed44c3f on tablet 8f86d1517df74ead9e397ca9bed44c3f with peer 1fab5794cf1740288abd82f2cebb9b1a
I20240502 21:29:05.437494 4538 tablet_bootstrap.cc:492] T 4dad8773f5b54efa9ff527252bac24aa P 16a97c1394f346f0b987f97cce7371b6: Bootstrap starting.
I20240502 21:29:05.452231 4121 raft_consensus.cc:1203] T 8f86d1517df74ead9e397ca9bed44c3f P 1fab5794cf1740288abd82f2cebb9b1a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240502 21:29:05.482460 4422 tablet_bootstrap.cc:492] T 3c88daaac3734d918955a774829532c6 P d8d7297186694f77bb98de9174e4279c: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:29:05.483902 4496 raft_consensus.cc:1050] T 13af1b96121c492c9373f9acbb55725a P 153026dd7f8943cc93fedf6f48450fb4: attempting to promote NON_VOTER d8d7297186694f77bb98de9174e4279c to VOTER
I20240502 21:29:05.485152 4496 consensus_queue.cc:237] T 13af1b96121c492c9373f9acbb55725a P 153026dd7f8943cc93fedf6f48450fb4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } }
I20240502 21:29:05.485833 4422 tablet_bootstrap.cc:492] T 3c88daaac3734d918955a774829532c6 P d8d7297186694f77bb98de9174e4279c: Bootstrap complete.
I20240502 21:29:05.486148 4422 ts_tablet_manager.cc:1367] T 3c88daaac3734d918955a774829532c6 P d8d7297186694f77bb98de9174e4279c: Time spent bootstrapping tablet: real 0.177s user 0.069s sys 0.035s
I20240502 21:29:05.486964 4422 raft_consensus.cc:348] T 3c88daaac3734d918955a774829532c6 P d8d7297186694f77bb98de9174e4279c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } attrs { replace: true } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: true } }
I20240502 21:29:05.487308 4422 raft_consensus.cc:729] T 3c88daaac3734d918955a774829532c6 P d8d7297186694f77bb98de9174e4279c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d8d7297186694f77bb98de9174e4279c, State: Initialized, Role: LEARNER
I20240502 21:29:05.487540 4422 consensus_queue.cc:260] T 3c88daaac3734d918955a774829532c6 P d8d7297186694f77bb98de9174e4279c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } attrs { replace: true } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: true } }
I20240502 21:29:05.493309 4422 ts_tablet_manager.cc:1398] T 3c88daaac3734d918955a774829532c6 P d8d7297186694f77bb98de9174e4279c: Time spent starting tablet: real 0.007s user 0.002s sys 0.001s
I20240502 21:29:05.494735 1570 tablet_copy_service.cc:342] P a62a935ff43d494ca35d0c417943133d: Request end of tablet copy session d8d7297186694f77bb98de9174e4279c-3c88daaac3734d918955a774829532c6 received from {username='slave'} at 127.15.255.136:51073
I20240502 21:29:05.494910 1570 tablet_copy_service.cc:433] P a62a935ff43d494ca35d0c417943133d: ending tablet copy session d8d7297186694f77bb98de9174e4279c-3c88daaac3734d918955a774829532c6 on tablet 3c88daaac3734d918955a774829532c6 with peer d8d7297186694f77bb98de9174e4279c
I20240502 21:29:05.506650 4368 raft_consensus.cc:1050] T 72355dd734ea4aeba031171e8c39696d P 7d1e5c1b047443cbb28d7b0a9744babf: attempting to promote NON_VOTER d8d7297186694f77bb98de9174e4279c to VOTER
I20240502 21:29:05.507962 4368 consensus_queue.cc:237] T 72355dd734ea4aeba031171e8c39696d P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } attrs { replace: true } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } }
I20240502 21:29:05.510368 4469 tablet_bootstrap.cc:492] T bcddb146822e4573837a5697d74134f7 P 16a97c1394f346f0b987f97cce7371b6: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:29:05.511050 4469 tablet_bootstrap.cc:492] T bcddb146822e4573837a5697d74134f7 P 16a97c1394f346f0b987f97cce7371b6: Bootstrap complete.
I20240502 21:29:05.511548 4469 ts_tablet_manager.cc:1367] T bcddb146822e4573837a5697d74134f7 P 16a97c1394f346f0b987f97cce7371b6: Time spent bootstrapping tablet: real 0.218s user 0.052s sys 0.007s
I20240502 21:29:05.511612 3855 raft_consensus.cc:1261] T 13af1b96121c492c9373f9acbb55725a P d8d7297186694f77bb98de9174e4279c [term 1 LEARNER]: Refusing update from remote peer 153026dd7f8943cc93fedf6f48450fb4: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:29:05.512805 1954 raft_consensus.cc:1261] T 13af1b96121c492c9373f9acbb55725a P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Refusing update from remote peer 153026dd7f8943cc93fedf6f48450fb4: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:29:05.512807 4496 consensus_queue.cc:1035] T 13af1b96121c492c9373f9acbb55725a P 153026dd7f8943cc93fedf6f48450fb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:29:05.513418 4496 consensus_queue.cc:1035] T 13af1b96121c492c9373f9acbb55725a P 153026dd7f8943cc93fedf6f48450fb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:29:05.515285 1550 raft_consensus.cc:1261] T 13af1b96121c492c9373f9acbb55725a P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Refusing update from remote peer 153026dd7f8943cc93fedf6f48450fb4: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:29:05.515782 1414 raft_consensus.cc:1261] T 13af1b96121c492c9373f9acbb55725a P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Refusing update from remote peer 153026dd7f8943cc93fedf6f48450fb4: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:29:05.516593 2226 raft_consensus.cc:1261] T 13af1b96121c492c9373f9acbb55725a P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Refusing update from remote peer 153026dd7f8943cc93fedf6f48450fb4: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:29:05.518002 4469 raft_consensus.cc:348] T bcddb146822e4573837a5697d74134f7 P 16a97c1394f346f0b987f97cce7371b6 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: NON_VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: true } }
I20240502 21:29:05.518532 4469 raft_consensus.cc:729] T bcddb146822e4573837a5697d74134f7 P 16a97c1394f346f0b987f97cce7371b6 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 16a97c1394f346f0b987f97cce7371b6, State: Initialized, Role: LEARNER
I20240502 21:29:05.518617 2089 raft_consensus.cc:1261] T 13af1b96121c492c9373f9acbb55725a P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Refusing update from remote peer 153026dd7f8943cc93fedf6f48450fb4: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:29:05.518776 4348 consensus_queue.cc:1035] T 13af1b96121c492c9373f9acbb55725a P 153026dd7f8943cc93fedf6f48450fb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:29:05.519295 4348 consensus_queue.cc:1035] T 13af1b96121c492c9373f9acbb55725a P 153026dd7f8943cc93fedf6f48450fb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:29:05.519143 4469 consensus_queue.cc:260] T bcddb146822e4573837a5697d74134f7 P 16a97c1394f346f0b987f97cce7371b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: NON_VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: true } }
I20240502 21:29:05.519737 4538 tablet_bootstrap.cc:492] T 4dad8773f5b54efa9ff527252bac24aa P 16a97c1394f346f0b987f97cce7371b6: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:29:05.519796 4348 consensus_queue.cc:1035] T 13af1b96121c492c9373f9acbb55725a P 153026dd7f8943cc93fedf6f48450fb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:29:05.520253 4538 tablet_bootstrap.cc:492] T 4dad8773f5b54efa9ff527252bac24aa P 16a97c1394f346f0b987f97cce7371b6: Bootstrap complete.
I20240502 21:29:05.520577 4538 ts_tablet_manager.cc:1367] T 4dad8773f5b54efa9ff527252bac24aa P 16a97c1394f346f0b987f97cce7371b6: Time spent bootstrapping tablet: real 0.083s user 0.039s sys 0.018s
I20240502 21:29:05.522939 4566 consensus_queue.cc:1035] T 13af1b96121c492c9373f9acbb55725a P 153026dd7f8943cc93fedf6f48450fb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:29:05.523192 1685 raft_consensus.cc:1261] T 13af1b96121c492c9373f9acbb55725a P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Refusing update from remote peer 153026dd7f8943cc93fedf6f48450fb4: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:29:05.523124 4538 raft_consensus.cc:348] T 4dad8773f5b54efa9ff527252bac24aa P 16a97c1394f346f0b987f97cce7371b6 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: NON_VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: true } }
I20240502 21:29:05.523824 4566 consensus_queue.cc:1035] T 13af1b96121c492c9373f9acbb55725a P 153026dd7f8943cc93fedf6f48450fb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:29:05.534694 4348 raft_consensus.cc:2932] T 13af1b96121c492c9373f9acbb55725a P 153026dd7f8943cc93fedf6f48450fb4 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, d8d7297186694f77bb98de9174e4279c (127.15.255.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } }
I20240502 21:29:05.536883 2226 raft_consensus.cc:2932] T 13af1b96121c492c9373f9acbb55725a P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, d8d7297186694f77bb98de9174e4279c (127.15.255.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } }
I20240502 21:29:05.539644 1685 raft_consensus.cc:2932] T 13af1b96121c492c9373f9acbb55725a P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, d8d7297186694f77bb98de9174e4279c (127.15.255.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } }
I20240502 21:29:05.540405 1954 raft_consensus.cc:2932] T 13af1b96121c492c9373f9acbb55725a P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, d8d7297186694f77bb98de9174e4279c (127.15.255.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } }
I20240502 21:29:05.547673 1265 catalog_manager.cc:5484] T 13af1b96121c492c9373f9acbb55725a P e632c553ff4341328acf1cbaca678392 reported cstate change: config changed from index 5 to 6, d8d7297186694f77bb98de9174e4279c (127.15.255.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "153026dd7f8943cc93fedf6f48450fb4" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } }
I20240502 21:29:05.556753 4538 raft_consensus.cc:729] T 4dad8773f5b54efa9ff527252bac24aa P 16a97c1394f346f0b987f97cce7371b6 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 16a97c1394f346f0b987f97cce7371b6, State: Initialized, Role: LEARNER
I20240502 21:29:05.557859 4538 consensus_queue.cc:260] T 4dad8773f5b54efa9ff527252bac24aa P 16a97c1394f346f0b987f97cce7371b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: NON_VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: true } }
I20240502 21:29:05.558874 3855 raft_consensus.cc:2932] T 13af1b96121c492c9373f9acbb55725a P d8d7297186694f77bb98de9174e4279c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, d8d7297186694f77bb98de9174e4279c (127.15.255.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } }
I20240502 21:29:05.562865 4469 ts_tablet_manager.cc:1398] T bcddb146822e4573837a5697d74134f7 P 16a97c1394f346f0b987f97cce7371b6: Time spent starting tablet: real 0.051s user 0.010s sys 0.009s
I20240502 21:29:05.563709 2249 tablet_copy_service.cc:342] P 6ab8df3c796f4b6cb3e29d5045ad3ea0: Request end of tablet copy session 16a97c1394f346f0b987f97cce7371b6-bcddb146822e4573837a5697d74134f7 received from {username='slave'} at 127.15.255.137:35291
I20240502 21:29:05.563879 2249 tablet_copy_service.cc:433] P 6ab8df3c796f4b6cb3e29d5045ad3ea0: ending tablet copy session 16a97c1394f346f0b987f97cce7371b6-bcddb146822e4573837a5697d74134f7 on tablet bcddb146822e4573837a5697d74134f7 with peer 16a97c1394f346f0b987f97cce7371b6
I20240502 21:29:05.568799 1549 raft_consensus.cc:2932] T 13af1b96121c492c9373f9acbb55725a P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, d8d7297186694f77bb98de9174e4279c (127.15.255.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } }
I20240502 21:29:05.571547 4579 ts_tablet_manager.cc:902] T e45af71e3fc14c579b32ba4dbe2adf78 P 16a97c1394f346f0b987f97cce7371b6: Initiating tablet copy from peer 521bb93035324cecbebed2f6f2a630f8 (127.15.255.134:40821)
I20240502 21:29:05.573478 4538 ts_tablet_manager.cc:1398] T 4dad8773f5b54efa9ff527252bac24aa P 16a97c1394f346f0b987f97cce7371b6: Time spent starting tablet: real 0.053s user 0.002s sys 0.003s
I20240502 21:29:05.575592 1975 tablet_copy_service.cc:342] P e632c553ff4341328acf1cbaca678392: Request end of tablet copy session 16a97c1394f346f0b987f97cce7371b6-4dad8773f5b54efa9ff527252bac24aa received from {username='slave'} at 127.15.255.137:38165
I20240502 21:29:05.575755 1975 tablet_copy_service.cc:433] P e632c553ff4341328acf1cbaca678392: ending tablet copy session 16a97c1394f346f0b987f97cce7371b6-4dad8773f5b54efa9ff527252bac24aa on tablet 4dad8773f5b54efa9ff527252bac24aa with peer 16a97c1394f346f0b987f97cce7371b6
I20240502 21:29:05.583060 2225 raft_consensus.cc:1261] T 72355dd734ea4aeba031171e8c39696d P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:29:05.583268 2089 raft_consensus.cc:2932] T 13af1b96121c492c9373f9acbb55725a P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, d8d7297186694f77bb98de9174e4279c (127.15.255.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } }
I20240502 21:29:05.584177 4361 consensus_queue.cc:1035] T 72355dd734ea4aeba031171e8c39696d P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:29:05.586362 1685 raft_consensus.cc:1261] T 72355dd734ea4aeba031171e8c39696d P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:29:05.587039 4361 consensus_queue.cc:1035] T 72355dd734ea4aeba031171e8c39696d P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:29:05.587332 1414 raft_consensus.cc:2932] T 13af1b96121c492c9373f9acbb55725a P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, d8d7297186694f77bb98de9174e4279c (127.15.255.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } }
I20240502 21:29:05.589830 1819 consensus_queue.cc:237] T 13af1b96121c492c9373f9acbb55725a P 153026dd7f8943cc93fedf6f48450fb4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } }
I20240502 21:29:05.594101 1685 raft_consensus.cc:1261] T 13af1b96121c492c9373f9acbb55725a P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Refusing update from remote peer 153026dd7f8943cc93fedf6f48450fb4: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:29:05.594537 2089 raft_consensus.cc:1261] T 13af1b96121c492c9373f9acbb55725a P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Refusing update from remote peer 153026dd7f8943cc93fedf6f48450fb4: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:29:05.594707 4577 consensus_queue.cc:1035] T 13af1b96121c492c9373f9acbb55725a P 153026dd7f8943cc93fedf6f48450fb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:29:05.595122 4576 consensus_queue.cc:1035] T 13af1b96121c492c9373f9acbb55725a P 153026dd7f8943cc93fedf6f48450fb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:29:05.595315 1954 raft_consensus.cc:1261] T 13af1b96121c492c9373f9acbb55725a P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Refusing update from remote peer 153026dd7f8943cc93fedf6f48450fb4: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:29:05.595785 1549 raft_consensus.cc:1261] T 13af1b96121c492c9373f9acbb55725a P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Refusing update from remote peer 153026dd7f8943cc93fedf6f48450fb4: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:29:05.593619 3855 raft_consensus.cc:1261] T 72355dd734ea4aeba031171e8c39696d P d8d7297186694f77bb98de9174e4279c [term 1 LEARNER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:29:05.593874 3856 raft_consensus.cc:1261] T 13af1b96121c492c9373f9acbb55725a P d8d7297186694f77bb98de9174e4279c [term 1 FOLLOWER]: Refusing update from remote peer 153026dd7f8943cc93fedf6f48450fb4: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:29:05.596844 4576 consensus_queue.cc:1035] T 13af1b96121c492c9373f9acbb55725a P 153026dd7f8943cc93fedf6f48450fb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:29:05.598021 1818 raft_consensus.cc:1261] T 72355dd734ea4aeba031171e8c39696d P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:29:05.597956 4576 consensus_queue.cc:1035] T 13af1b96121c492c9373f9acbb55725a P 153026dd7f8943cc93fedf6f48450fb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:29:05.599817 3856 raft_consensus.cc:1203] T f8bee9bdcbff404280f000bdaeadf465 P d8d7297186694f77bb98de9174e4279c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240502 21:29:05.600216 1549 raft_consensus.cc:1261] T 72355dd734ea4aeba031171e8c39696d P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:29:05.600320 4369 consensus_queue.cc:1035] T 72355dd734ea4aeba031171e8c39696d P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:29:05.600899 4369 consensus_queue.cc:1035] T 72355dd734ea4aeba031171e8c39696d P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:29:05.601600 4369 consensus_queue.cc:1035] T 72355dd734ea4aeba031171e8c39696d P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:29:05.602675 2089 raft_consensus.cc:1261] T 72355dd734ea4aeba031171e8c39696d P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:29:05.603494 4361 consensus_queue.cc:1035] T 72355dd734ea4aeba031171e8c39696d P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:29:05.605955 1414 raft_consensus.cc:1261] T 13af1b96121c492c9373f9acbb55725a P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Refusing update from remote peer 153026dd7f8943cc93fedf6f48450fb4: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:29:05.606568 4576 consensus_queue.cc:1035] T 13af1b96121c492c9373f9acbb55725a P 153026dd7f8943cc93fedf6f48450fb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:29:05.607023 4576 consensus_queue.cc:1035] T 13af1b96121c492c9373f9acbb55725a P 153026dd7f8943cc93fedf6f48450fb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:29:05.607160 1954 raft_consensus.cc:1261] T 72355dd734ea4aeba031171e8c39696d P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:29:05.607785 4369 consensus_queue.cc:1035] T 72355dd734ea4aeba031171e8c39696d P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:29:05.607503 4576 raft_consensus.cc:2932] T 13af1b96121c492c9373f9acbb55725a P 153026dd7f8943cc93fedf6f48450fb4 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 6ab8df3c796f4b6cb3e29d5045ad3ea0 (127.15.255.135) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } }
I20240502 21:29:05.608083 1684 raft_consensus.cc:2932] T 13af1b96121c492c9373f9acbb55725a P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 6ab8df3c796f4b6cb3e29d5045ad3ea0 (127.15.255.135) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } }
I20240502 21:29:05.613408 1265 catalog_manager.cc:5484] T 13af1b96121c492c9373f9acbb55725a P 16b3a337bc5e462f83ba895ad8660deb reported cstate change: config changed from index 6 to 7, VOTER 6ab8df3c796f4b6cb3e29d5045ad3ea0 (127.15.255.135) evicted. New cstate: current_term: 1 leader_uuid: "153026dd7f8943cc93fedf6f48450fb4" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } }
I20240502 21:29:05.614135 1955 raft_consensus.cc:2932] T 13af1b96121c492c9373f9acbb55725a P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 6ab8df3c796f4b6cb3e29d5045ad3ea0 (127.15.255.135) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } }
I20240502 21:29:05.618211 2205 tablet_service.cc:1514] Processing DeleteTablet for tablet 13af1b96121c492c9373f9acbb55725a with delete_type TABLET_DATA_TOMBSTONED (TS 6ab8df3c796f4b6cb3e29d5045ad3ea0 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:37668
I20240502 21:29:05.620411 1414 raft_consensus.cc:2932] T 13af1b96121c492c9373f9acbb55725a P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 6ab8df3c796f4b6cb3e29d5045ad3ea0 (127.15.255.135) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } }
I20240502 21:29:05.631522 1256 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 13af1b96121c492c9373f9acbb55725a with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:29:05.632345 2087 raft_consensus.cc:2932] T 13af1b96121c492c9373f9acbb55725a P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 6ab8df3c796f4b6cb3e29d5045ad3ea0 (127.15.255.135) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } }
I20240502 21:29:05.636967 4598 tablet_replica.cc:330] T 13af1b96121c492c9373f9acbb55725a P 6ab8df3c796f4b6cb3e29d5045ad3ea0: stopping tablet replica
I20240502 21:29:05.637461 4598 raft_consensus.cc:2229] T 13af1b96121c492c9373f9acbb55725a P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:29:05.638079 4598 raft_consensus.cc:2258] T 13af1b96121c492c9373f9acbb55725a P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:29:05.639849 4598 ts_tablet_manager.cc:1875] T 13af1b96121c492c9373f9acbb55725a P 6ab8df3c796f4b6cb3e29d5045ad3ea0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:29:05.639513 1550 raft_consensus.cc:2932] T 13af1b96121c492c9373f9acbb55725a P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 6ab8df3c796f4b6cb3e29d5045ad3ea0 (127.15.255.135) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } }
I20240502 21:29:05.636839 4590 raft_consensus.cc:2932] T 72355dd734ea4aeba031171e8c39696d P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, d8d7297186694f77bb98de9174e4279c (127.15.255.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } attrs { replace: true } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } }
I20240502 21:29:05.640388 3855 raft_consensus.cc:2932] T 13af1b96121c492c9373f9acbb55725a P d8d7297186694f77bb98de9174e4279c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 6ab8df3c796f4b6cb3e29d5045ad3ea0 (127.15.255.135) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } }
I20240502 21:29:05.647938 1819 raft_consensus.cc:2932] T 72355dd734ea4aeba031171e8c39696d P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, d8d7297186694f77bb98de9174e4279c (127.15.255.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } attrs { replace: true } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } }
I20240502 21:29:05.648463 1550 raft_consensus.cc:2932] T 72355dd734ea4aeba031171e8c39696d P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, d8d7297186694f77bb98de9174e4279c (127.15.255.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } attrs { replace: true } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } }
I20240502 21:29:05.649528 4598 ts_tablet_manager.cc:1888] T 13af1b96121c492c9373f9acbb55725a P 6ab8df3c796f4b6cb3e29d5045ad3ea0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240502 21:29:05.649736 4598 log.cc:1196] T 13af1b96121c492c9373f9acbb55725a P 6ab8df3c796f4b6cb3e29d5045ad3ea0: Deleting WAL directory at /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-6/wal/wals/13af1b96121c492c9373f9acbb55725a
I20240502 21:29:05.650738 1255 catalog_manager.cc:4830] TS 6ab8df3c796f4b6cb3e29d5045ad3ea0 (127.15.255.135:34701): tablet 13af1b96121c492c9373f9acbb55725a (table rebalance_test_table_0 [id=82737b0de4b945dc8ca0c7d6b92af37c]) successfully deleted
I20240502 21:29:05.654450 1265 catalog_manager.cc:5484] T 72355dd734ea4aeba031171e8c39696d P a62a935ff43d494ca35d0c417943133d reported cstate change: config changed from index 5 to 6, d8d7297186694f77bb98de9174e4279c (127.15.255.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } attrs { replace: true } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } }
I20240502 21:29:05.654659 4579 tablet_copy_client.cc:308] T e45af71e3fc14c579b32ba4dbe2adf78 P 16a97c1394f346f0b987f97cce7371b6: tablet copy: Beginning tablet copy session from remote peer at address 127.15.255.134:40821
I20240502 21:29:05.657111 4383 raft_consensus.cc:1050] T 1aa57620996e42b6adf941e9d606f7e2 P 521bb93035324cecbebed2f6f2a630f8: attempting to promote NON_VOTER d8d7297186694f77bb98de9174e4279c to VOTER
I20240502 21:29:05.658390 4383 consensus_queue.cc:237] T 1aa57620996e42b6adf941e9d606f7e2 P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } attrs { replace: true } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } }
I20240502 21:29:05.661013 2089 raft_consensus.cc:2932] T 72355dd734ea4aeba031171e8c39696d P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, d8d7297186694f77bb98de9174e4279c (127.15.255.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } attrs { replace: true } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } }
I20240502 21:29:05.664270 2113 tablet_copy_service.cc:140] P 521bb93035324cecbebed2f6f2a630f8: Received BeginTabletCopySession request for tablet e45af71e3fc14c579b32ba4dbe2adf78 from peer 16a97c1394f346f0b987f97cce7371b6 ({username='slave'} at 127.15.255.137:45019)
I20240502 21:29:05.664459 2113 tablet_copy_service.cc:161] P 521bb93035324cecbebed2f6f2a630f8: Beginning new tablet copy session on tablet e45af71e3fc14c579b32ba4dbe2adf78 from peer 16a97c1394f346f0b987f97cce7371b6 at {username='slave'} at 127.15.255.137:45019: session id = 16a97c1394f346f0b987f97cce7371b6-e45af71e3fc14c579b32ba4dbe2adf78
I20240502 21:29:05.667343 2113 tablet_copy_source_session.cc:205] T e45af71e3fc14c579b32ba4dbe2adf78 P 521bb93035324cecbebed2f6f2a630f8: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:29:05.668673 4579 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e45af71e3fc14c579b32ba4dbe2adf78. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:29:05.668583 3856 raft_consensus.cc:2932] T 72355dd734ea4aeba031171e8c39696d P d8d7297186694f77bb98de9174e4279c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, d8d7297186694f77bb98de9174e4279c (127.15.255.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } attrs { replace: true } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } }
I20240502 21:29:05.668941 1685 raft_consensus.cc:2932] T 72355dd734ea4aeba031171e8c39696d P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, d8d7297186694f77bb98de9174e4279c (127.15.255.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } attrs { replace: true } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } }
I20240502 21:29:05.675297 2225 raft_consensus.cc:1261] T 1aa57620996e42b6adf941e9d606f7e2 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:29:05.675642 1818 raft_consensus.cc:1261] T 1aa57620996e42b6adf941e9d606f7e2 P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:29:05.675693 3856 raft_consensus.cc:1261] T 1aa57620996e42b6adf941e9d606f7e2 P d8d7297186694f77bb98de9174e4279c [term 1 LEARNER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:29:05.676293 1684 raft_consensus.cc:1261] T 1aa57620996e42b6adf941e9d606f7e2 P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:29:05.676270 4383 consensus_queue.cc:1035] T 1aa57620996e42b6adf941e9d606f7e2 P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:29:05.676838 1954 raft_consensus.cc:1261] T 1aa57620996e42b6adf941e9d606f7e2 P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:29:05.678444 3545 consensus_queue.cc:1035] T 1aa57620996e42b6adf941e9d606f7e2 P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:29:05.678963 4383 consensus_queue.cc:1035] T 1aa57620996e42b6adf941e9d606f7e2 P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:29:05.678604 1954 raft_consensus.cc:2932] T 72355dd734ea4aeba031171e8c39696d P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, d8d7297186694f77bb98de9174e4279c (127.15.255.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } attrs { replace: true } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } }
I20240502 21:29:05.679361 1549 raft_consensus.cc:1261] T 1aa57620996e42b6adf941e9d606f7e2 P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:29:05.682264 4504 consensus_queue.cc:1035] T 1aa57620996e42b6adf941e9d606f7e2 P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:29:05.682744 4524 consensus_queue.cc:1035] T 1aa57620996e42b6adf941e9d606f7e2 P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:29:05.683640 3765 consensus_queue.cc:1035] T 1aa57620996e42b6adf941e9d606f7e2 P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:29:05.684396 1414 raft_consensus.cc:1261] T 1aa57620996e42b6adf941e9d606f7e2 P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:29:05.685359 3765 consensus_queue.cc:1035] T 1aa57620996e42b6adf941e9d606f7e2 P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:29:05.686558 2226 raft_consensus.cc:2932] T 72355dd734ea4aeba031171e8c39696d P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, d8d7297186694f77bb98de9174e4279c (127.15.255.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } attrs { replace: true } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } }
I20240502 21:29:05.716120 4579 tablet_copy_client.cc:783] T e45af71e3fc14c579b32ba4dbe2adf78 P 16a97c1394f346f0b987f97cce7371b6: tablet copy: Starting download of 0 data blocks...
I20240502 21:29:05.716521 4579 tablet_copy_client.cc:647] T e45af71e3fc14c579b32ba4dbe2adf78 P 16a97c1394f346f0b987f97cce7371b6: tablet copy: Starting download of 1 WAL segments...
I20240502 21:29:05.719564 4579 tablet_copy_client.cc:523] T e45af71e3fc14c579b32ba4dbe2adf78 P 16a97c1394f346f0b987f97cce7371b6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:29:05.722834 1415 consensus_queue.cc:237] T 72355dd734ea4aeba031171e8c39696d P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } }
I20240502 21:29:05.725841 3856 raft_consensus.cc:1261] T 72355dd734ea4aeba031171e8c39696d P d8d7297186694f77bb98de9174e4279c [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:29:05.727146 1954 raft_consensus.cc:1261] T 72355dd734ea4aeba031171e8c39696d P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:29:05.727180 2226 raft_consensus.cc:1261] T 72355dd734ea4aeba031171e8c39696d P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:29:05.727705 4369 consensus_queue.cc:1035] T 72355dd734ea4aeba031171e8c39696d P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:29:05.728232 4592 consensus_queue.cc:1035] T 72355dd734ea4aeba031171e8c39696d P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:29:05.728768 4368 consensus_queue.cc:1035] T 72355dd734ea4aeba031171e8c39696d P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:29:05.731532 4579 tablet_bootstrap.cc:492] T e45af71e3fc14c579b32ba4dbe2adf78 P 16a97c1394f346f0b987f97cce7371b6: Bootstrap starting.
I20240502 21:29:05.731683 1819 raft_consensus.cc:1261] T 72355dd734ea4aeba031171e8c39696d P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:29:05.731210 4369 raft_consensus.cc:2932] T 72355dd734ea4aeba031171e8c39696d P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 521bb93035324cecbebed2f6f2a630f8 (127.15.255.134) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } }
I20240502 21:29:05.732306 4592 consensus_queue.cc:1035] T 72355dd734ea4aeba031171e8c39696d P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:29:05.732718 1550 raft_consensus.cc:1261] T 72355dd734ea4aeba031171e8c39696d P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:29:05.733330 4590 consensus_queue.cc:1035] T 72355dd734ea4aeba031171e8c39696d P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:29:05.733125 3856 raft_consensus.cc:2932] T 72355dd734ea4aeba031171e8c39696d P d8d7297186694f77bb98de9174e4279c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 521bb93035324cecbebed2f6f2a630f8 (127.15.255.134) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } }
I20240502 21:29:05.733803 1819 raft_consensus.cc:2932] T 72355dd734ea4aeba031171e8c39696d P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 521bb93035324cecbebed2f6f2a630f8 (127.15.255.134) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } }
I20240502 21:29:05.734853 1547 raft_consensus.cc:2932] T 72355dd734ea4aeba031171e8c39696d P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 521bb93035324cecbebed2f6f2a630f8 (127.15.255.134) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } }
I20240502 21:29:05.739773 1684 raft_consensus.cc:1261] T 72355dd734ea4aeba031171e8c39696d P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:29:05.741797 3988 raft_consensus.cc:1203] T 4dad8773f5b54efa9ff527252bac24aa P 16a97c1394f346f0b987f97cce7371b6 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240502 21:29:05.743047 4590 consensus_queue.cc:1035] T 72355dd734ea4aeba031171e8c39696d P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:29:05.744976 1684 raft_consensus.cc:2932] T 72355dd734ea4aeba031171e8c39696d P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 521bb93035324cecbebed2f6f2a630f8 (127.15.255.134) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } }
I20240502 21:29:05.746896 1270 catalog_manager.cc:5484] T 72355dd734ea4aeba031171e8c39696d P a62a935ff43d494ca35d0c417943133d reported cstate change: config changed from index 6 to 7, VOTER 521bb93035324cecbebed2f6f2a630f8 (127.15.255.134) evicted. New cstate: current_term: 1 leader_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } }
I20240502 21:29:05.751157 2068 tablet_service.cc:1514] Processing DeleteTablet for tablet 72355dd734ea4aeba031171e8c39696d with delete_type TABLET_DATA_TOMBSTONED (TS 521bb93035324cecbebed2f6f2a630f8 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:50230
I20240502 21:29:05.756040 1255 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 72355dd734ea4aeba031171e8c39696d with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:29:05.756691 2226 raft_consensus.cc:2932] T 72355dd734ea4aeba031171e8c39696d P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 521bb93035324cecbebed2f6f2a630f8 (127.15.255.134) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } }
I20240502 21:29:05.757184 1954 raft_consensus.cc:2932] T 72355dd734ea4aeba031171e8c39696d P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 521bb93035324cecbebed2f6f2a630f8 (127.15.255.134) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } }
I20240502 21:29:05.760787 4613 tablet_replica.cc:330] T 72355dd734ea4aeba031171e8c39696d P 521bb93035324cecbebed2f6f2a630f8: stopping tablet replica
I20240502 21:29:05.762094 4613 raft_consensus.cc:2229] T 72355dd734ea4aeba031171e8c39696d P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:29:05.762936 4613 raft_consensus.cc:2258] T 72355dd734ea4aeba031171e8c39696d P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:29:05.765259 4613 ts_tablet_manager.cc:1875] T 72355dd734ea4aeba031171e8c39696d P 521bb93035324cecbebed2f6f2a630f8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:29:05.768362 4504 raft_consensus.cc:2932] T 1aa57620996e42b6adf941e9d606f7e2 P 521bb93035324cecbebed2f6f2a630f8 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, d8d7297186694f77bb98de9174e4279c (127.15.255.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } attrs { replace: true } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } }
I20240502 21:29:05.770320 2226 raft_consensus.cc:2932] T 1aa57620996e42b6adf941e9d606f7e2 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, d8d7297186694f77bb98de9174e4279c (127.15.255.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } attrs { replace: true } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } }
I20240502 21:29:05.773728 4613 ts_tablet_manager.cc:1888] T 72355dd734ea4aeba031171e8c39696d P 521bb93035324cecbebed2f6f2a630f8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240502 21:29:05.773954 4613 log.cc:1196] T 72355dd734ea4aeba031171e8c39696d P 521bb93035324cecbebed2f6f2a630f8: Deleting WAL directory at /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-5/wal/wals/72355dd734ea4aeba031171e8c39696d
I20240502 21:29:05.773587 3855 raft_consensus.cc:2932] T 1aa57620996e42b6adf941e9d606f7e2 P d8d7297186694f77bb98de9174e4279c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, d8d7297186694f77bb98de9174e4279c (127.15.255.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } attrs { replace: true } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } }
I20240502 21:29:05.776830 1254 catalog_manager.cc:4830] TS 521bb93035324cecbebed2f6f2a630f8 (127.15.255.134:40821): tablet 72355dd734ea4aeba031171e8c39696d (table rebalance_test_table_5 [id=9cb803e5bb2545c8b992c90a894085af]) successfully deleted
I20240502 21:29:05.777868 1265 catalog_manager.cc:5484] T 1aa57620996e42b6adf941e9d606f7e2 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 reported cstate change: config changed from index 5 to 6, d8d7297186694f77bb98de9174e4279c (127.15.255.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "521bb93035324cecbebed2f6f2a630f8" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } attrs { replace: true } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } }
I20240502 21:29:05.778638 1818 raft_consensus.cc:2932] T 1aa57620996e42b6adf941e9d606f7e2 P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, d8d7297186694f77bb98de9174e4279c (127.15.255.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } attrs { replace: true } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } }
I20240502 21:29:05.780090 1684 raft_consensus.cc:2932] T 1aa57620996e42b6adf941e9d606f7e2 P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, d8d7297186694f77bb98de9174e4279c (127.15.255.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } attrs { replace: true } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } }
I20240502 21:29:05.782995 1955 raft_consensus.cc:2932] T 1aa57620996e42b6adf941e9d606f7e2 P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, d8d7297186694f77bb98de9174e4279c (127.15.255.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } attrs { replace: true } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } }
I20240502 21:29:05.783531 1414 raft_consensus.cc:2932] T 1aa57620996e42b6adf941e9d606f7e2 P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, d8d7297186694f77bb98de9174e4279c (127.15.255.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } attrs { replace: true } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } }
I20240502 21:29:05.787993 2089 consensus_queue.cc:237] T 1aa57620996e42b6adf941e9d606f7e2 P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } }
I20240502 21:29:05.788033 1549 raft_consensus.cc:2932] T 1aa57620996e42b6adf941e9d606f7e2 P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, d8d7297186694f77bb98de9174e4279c (127.15.255.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } attrs { replace: true } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } }
I20240502 21:29:05.796975 3855 raft_consensus.cc:1261] T 1aa57620996e42b6adf941e9d606f7e2 P d8d7297186694f77bb98de9174e4279c [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:29:05.798475 1818 raft_consensus.cc:1261] T 1aa57620996e42b6adf941e9d606f7e2 P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:29:05.798615 3765 consensus_queue.cc:1035] T 1aa57620996e42b6adf941e9d606f7e2 P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:29:05.799316 3545 consensus_queue.cc:1035] T 1aa57620996e42b6adf941e9d606f7e2 P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:29:05.799847 1415 raft_consensus.cc:1261] T 1aa57620996e42b6adf941e9d606f7e2 P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:29:05.800172 1685 raft_consensus.cc:1261] T 1aa57620996e42b6adf941e9d606f7e2 P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:29:05.800467 3765 consensus_queue.cc:1035] T 1aa57620996e42b6adf941e9d606f7e2 P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:29:05.800911 4524 consensus_queue.cc:1035] T 1aa57620996e42b6adf941e9d606f7e2 P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:29:05.801316 2225 raft_consensus.cc:1261] T 1aa57620996e42b6adf941e9d606f7e2 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:29:05.803105 3765 consensus_queue.cc:1035] T 1aa57620996e42b6adf941e9d606f7e2 P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:29:05.805004 4383 raft_consensus.cc:2932] T 1aa57620996e42b6adf941e9d606f7e2 P 521bb93035324cecbebed2f6f2a630f8 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER e632c553ff4341328acf1cbaca678392 (127.15.255.133) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } }
I20240502 21:29:05.805689 1414 raft_consensus.cc:2932] T 1aa57620996e42b6adf941e9d606f7e2 P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER e632c553ff4341328acf1cbaca678392 (127.15.255.133) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } }
I20240502 21:29:05.806280 2226 raft_consensus.cc:2932] T 1aa57620996e42b6adf941e9d606f7e2 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER e632c553ff4341328acf1cbaca678392 (127.15.255.133) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } }
I20240502 21:29:05.812301 1270 catalog_manager.cc:5484] T 1aa57620996e42b6adf941e9d606f7e2 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 reported cstate change: config changed from index 6 to 7, VOTER e632c553ff4341328acf1cbaca678392 (127.15.255.133) evicted. New cstate: current_term: 1 leader_uuid: "521bb93035324cecbebed2f6f2a630f8" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } }
I20240502 21:29:05.813109 1685 raft_consensus.cc:2932] T 1aa57620996e42b6adf941e9d606f7e2 P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER e632c553ff4341328acf1cbaca678392 (127.15.255.133) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } }
I20240502 21:29:05.819959 1931 tablet_service.cc:1514] Processing DeleteTablet for tablet 1aa57620996e42b6adf941e9d606f7e2 with delete_type TABLET_DATA_TOMBSTONED (TS e632c553ff4341328acf1cbaca678392 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:51198
I20240502 21:29:05.820488 1254 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 1aa57620996e42b6adf941e9d606f7e2 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:29:05.822088 3855 raft_consensus.cc:2932] T 1aa57620996e42b6adf941e9d606f7e2 P d8d7297186694f77bb98de9174e4279c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER e632c553ff4341328acf1cbaca678392 (127.15.255.133) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } }
I20240502 21:29:05.822903 1818 raft_consensus.cc:2932] T 1aa57620996e42b6adf941e9d606f7e2 P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER e632c553ff4341328acf1cbaca678392 (127.15.255.133) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } }
I20240502 21:29:05.828984 1549 raft_consensus.cc:1261] T 1aa57620996e42b6adf941e9d606f7e2 P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:29:05.830309 3765 consensus_queue.cc:1035] T 1aa57620996e42b6adf941e9d606f7e2 P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:29:05.832027 1547 raft_consensus.cc:2932] T 1aa57620996e42b6adf941e9d606f7e2 P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER e632c553ff4341328acf1cbaca678392 (127.15.255.133) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } }
I20240502 21:29:05.839887 4625 tablet_replica.cc:330] T 1aa57620996e42b6adf941e9d606f7e2 P e632c553ff4341328acf1cbaca678392: stopping tablet replica
I20240502 21:29:05.840476 4625 raft_consensus.cc:2229] T 1aa57620996e42b6adf941e9d606f7e2 P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:29:05.841109 4625 raft_consensus.cc:2258] T 1aa57620996e42b6adf941e9d606f7e2 P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:29:05.842710 4625 ts_tablet_manager.cc:1875] T 1aa57620996e42b6adf941e9d606f7e2 P e632c553ff4341328acf1cbaca678392: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:29:05.848810 4625 ts_tablet_manager.cc:1888] T 1aa57620996e42b6adf941e9d606f7e2 P e632c553ff4341328acf1cbaca678392: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240502 21:29:05.849099 4625 log.cc:1196] T 1aa57620996e42b6adf941e9d606f7e2 P e632c553ff4341328acf1cbaca678392: Deleting WAL directory at /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-4/wal/wals/1aa57620996e42b6adf941e9d606f7e2
I20240502 21:29:05.851329 1254 catalog_manager.cc:4830] TS e632c553ff4341328acf1cbaca678392 (127.15.255.133:37439): tablet 1aa57620996e42b6adf941e9d606f7e2 (table rebalance_test_table_2 [id=900316782c71465f9538d1bbe31708d3]) successfully deleted
I20240502 21:29:05.864871 4579 tablet_bootstrap.cc:492] T e45af71e3fc14c579b32ba4dbe2adf78 P 16a97c1394f346f0b987f97cce7371b6: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:29:05.865214 4579 tablet_bootstrap.cc:492] T e45af71e3fc14c579b32ba4dbe2adf78 P 16a97c1394f346f0b987f97cce7371b6: Bootstrap complete.
I20240502 21:29:05.865463 4579 ts_tablet_manager.cc:1367] T e45af71e3fc14c579b32ba4dbe2adf78 P 16a97c1394f346f0b987f97cce7371b6: Time spent bootstrapping tablet: real 0.134s user 0.040s sys 0.008s
I20240502 21:29:05.866147 4579 raft_consensus.cc:348] T e45af71e3fc14c579b32ba4dbe2adf78 P 16a97c1394f346f0b987f97cce7371b6 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } attrs { replace: true } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: NON_VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: true } }
I20240502 21:29:05.866453 4579 raft_consensus.cc:729] T e45af71e3fc14c579b32ba4dbe2adf78 P 16a97c1394f346f0b987f97cce7371b6 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 16a97c1394f346f0b987f97cce7371b6, State: Initialized, Role: LEARNER
I20240502 21:29:05.866662 4579 consensus_queue.cc:260] T e45af71e3fc14c579b32ba4dbe2adf78 P 16a97c1394f346f0b987f97cce7371b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } attrs { replace: true } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: NON_VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: true } }
I20240502 21:29:05.867360 4579 ts_tablet_manager.cc:1398] T e45af71e3fc14c579b32ba4dbe2adf78 P 16a97c1394f346f0b987f97cce7371b6: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240502 21:29:05.868036 2113 tablet_copy_service.cc:342] P 521bb93035324cecbebed2f6f2a630f8: Request end of tablet copy session 16a97c1394f346f0b987f97cce7371b6-e45af71e3fc14c579b32ba4dbe2adf78 received from {username='slave'} at 127.15.255.137:45019
I20240502 21:29:05.868180 2113 tablet_copy_service.cc:433] P 521bb93035324cecbebed2f6f2a630f8: ending tablet copy session 16a97c1394f346f0b987f97cce7371b6-e45af71e3fc14c579b32ba4dbe2adf78 on tablet e45af71e3fc14c579b32ba4dbe2adf78 with peer 16a97c1394f346f0b987f97cce7371b6
I20240502 21:29:05.874433 3988 raft_consensus.cc:1203] T bcddb146822e4573837a5697d74134f7 P 16a97c1394f346f0b987f97cce7371b6 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240502 21:29:05.910724 3855 raft_consensus.cc:1203] T 3c88daaac3734d918955a774829532c6 P d8d7297186694f77bb98de9174e4279c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240502 21:29:06.018455 3765 raft_consensus.cc:1050] T 8f86d1517df74ead9e397ca9bed44c3f P 521bb93035324cecbebed2f6f2a630f8: attempting to promote NON_VOTER 1fab5794cf1740288abd82f2cebb9b1a to VOTER
I20240502 21:29:06.019320 3765 consensus_queue.cc:237] T 8f86d1517df74ead9e397ca9bed44c3f P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } }
I20240502 21:29:06.022672 1818 raft_consensus.cc:1261] T 8f86d1517df74ead9e397ca9bed44c3f P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:29:06.023329 4524 consensus_queue.cc:1035] T 8f86d1517df74ead9e397ca9bed44c3f P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:29:06.023528 1955 raft_consensus.cc:1261] T 8f86d1517df74ead9e397ca9bed44c3f P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:29:06.025792 3545 consensus_queue.cc:1035] T 8f86d1517df74ead9e397ca9bed44c3f P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:29:06.028841 1685 raft_consensus.cc:1261] T 8f86d1517df74ead9e397ca9bed44c3f P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:29:06.029140 2226 raft_consensus.cc:1261] T 8f86d1517df74ead9e397ca9bed44c3f P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:29:06.029683 1549 raft_consensus.cc:1261] T 8f86d1517df74ead9e397ca9bed44c3f P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:29:06.029827 4383 consensus_queue.cc:1035] T 8f86d1517df74ead9e397ca9bed44c3f P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:29:06.030956 4121 raft_consensus.cc:1261] T 8f86d1517df74ead9e397ca9bed44c3f P 1fab5794cf1740288abd82f2cebb9b1a [term 1 LEARNER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:29:06.032169 4524 consensus_queue.cc:1035] T 8f86d1517df74ead9e397ca9bed44c3f P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:29:06.033138 4504 consensus_queue.cc:1035] T 8f86d1517df74ead9e397ca9bed44c3f P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:29:06.033643 4504 consensus_queue.cc:1035] T 8f86d1517df74ead9e397ca9bed44c3f P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:29:06.036733 1414 raft_consensus.cc:1261] T 8f86d1517df74ead9e397ca9bed44c3f P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:29:06.037387 4637 consensus_queue.cc:1035] T 8f86d1517df74ead9e397ca9bed44c3f P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:29:06.039904 4524 raft_consensus.cc:2932] T 8f86d1517df74ead9e397ca9bed44c3f P 521bb93035324cecbebed2f6f2a630f8 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } }
I20240502 21:29:06.040402 1955 raft_consensus.cc:2932] T 8f86d1517df74ead9e397ca9bed44c3f P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } }
I20240502 21:29:06.046980 1270 catalog_manager.cc:5484] T 8f86d1517df74ead9e397ca9bed44c3f P e632c553ff4341328acf1cbaca678392 reported cstate change: config changed from index 5 to 6, 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "521bb93035324cecbebed2f6f2a630f8" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } }
I20240502 21:29:06.048280 1414 raft_consensus.cc:2932] T 8f86d1517df74ead9e397ca9bed44c3f P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } }
I20240502 21:29:06.049434 4121 raft_consensus.cc:2932] T 8f86d1517df74ead9e397ca9bed44c3f P 1fab5794cf1740288abd82f2cebb9b1a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } }
I20240502 21:29:06.053512 2226 raft_consensus.cc:2932] T 8f86d1517df74ead9e397ca9bed44c3f P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } }
I20240502 21:29:06.053966 1818 raft_consensus.cc:2932] T 8f86d1517df74ead9e397ca9bed44c3f P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } }
I20240502 21:29:06.054397 1684 raft_consensus.cc:2932] T 8f86d1517df74ead9e397ca9bed44c3f P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } }
I20240502 21:29:06.066236 1549 raft_consensus.cc:2932] T 8f86d1517df74ead9e397ca9bed44c3f P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } }
I20240502 21:29:06.070724 2089 consensus_queue.cc:237] T 8f86d1517df74ead9e397ca9bed44c3f P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } }
I20240502 21:29:06.073300 1684 raft_consensus.cc:1261] T 8f86d1517df74ead9e397ca9bed44c3f P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:29:06.073442 2226 raft_consensus.cc:1261] T 8f86d1517df74ead9e397ca9bed44c3f P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:29:06.073496 1549 raft_consensus.cc:1261] T 8f86d1517df74ead9e397ca9bed44c3f P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:29:06.073819 1414 raft_consensus.cc:1261] T 8f86d1517df74ead9e397ca9bed44c3f P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:29:06.074049 3545 consensus_queue.cc:1035] T 8f86d1517df74ead9e397ca9bed44c3f P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:29:06.074255 3765 consensus_queue.cc:1035] T 8f86d1517df74ead9e397ca9bed44c3f P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:29:06.074520 4383 consensus_queue.cc:1035] T 8f86d1517df74ead9e397ca9bed44c3f P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:29:06.074589 4121 raft_consensus.cc:1261] T 8f86d1517df74ead9e397ca9bed44c3f P 1fab5794cf1740288abd82f2cebb9b1a [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:29:06.074892 1955 raft_consensus.cc:1261] T 8f86d1517df74ead9e397ca9bed44c3f P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:29:06.075225 3545 consensus_queue.cc:1035] T 8f86d1517df74ead9e397ca9bed44c3f P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:29:06.075599 4524 consensus_queue.cc:1035] T 8f86d1517df74ead9e397ca9bed44c3f P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:29:06.078588 4524 consensus_queue.cc:1035] T 8f86d1517df74ead9e397ca9bed44c3f P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:29:06.079320 4637 raft_consensus.cc:2932] T 8f86d1517df74ead9e397ca9bed44c3f P 521bb93035324cecbebed2f6f2a630f8 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } }
I20240502 21:29:06.079869 1413 raft_consensus.cc:2932] T 8f86d1517df74ead9e397ca9bed44c3f P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } }
I20240502 21:29:06.082734 2225 raft_consensus.cc:2932] T 8f86d1517df74ead9e397ca9bed44c3f P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } }
I20240502 21:29:06.083246 4121 raft_consensus.cc:2932] T 8f86d1517df74ead9e397ca9bed44c3f P 1fab5794cf1740288abd82f2cebb9b1a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } }
I20240502 21:29:06.085160 1254 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 8f86d1517df74ead9e397ca9bed44c3f with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:29:06.088330 1547 raft_consensus.cc:2932] T 8f86d1517df74ead9e397ca9bed44c3f P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } }
I20240502 21:29:06.089243 1685 raft_consensus.cc:2932] T 8f86d1517df74ead9e397ca9bed44c3f P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } }
I20240502 21:29:06.089869 1270 catalog_manager.cc:5484] T 8f86d1517df74ead9e397ca9bed44c3f P 7d1e5c1b047443cbb28d7b0a9744babf reported cstate change: config changed from index 6 to 7, VOTER 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132) evicted. New cstate: current_term: 1 leader_uuid: "521bb93035324cecbebed2f6f2a630f8" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } }
I20240502 21:29:06.093863 1955 raft_consensus.cc:2932] T 8f86d1517df74ead9e397ca9bed44c3f P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } }
I20240502 21:29:06.100740 1798 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f86d1517df74ead9e397ca9bed44c3f with delete_type TABLET_DATA_TOMBSTONED (TS 153026dd7f8943cc93fedf6f48450fb4 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:60256
I20240502 21:29:06.101581 4646 tablet_replica.cc:330] T 8f86d1517df74ead9e397ca9bed44c3f P 153026dd7f8943cc93fedf6f48450fb4: stopping tablet replica
I20240502 21:29:06.102003 4646 raft_consensus.cc:2229] T 8f86d1517df74ead9e397ca9bed44c3f P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:29:06.102264 4646 raft_consensus.cc:2258] T 8f86d1517df74ead9e397ca9bed44c3f P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:29:06.103482 4646 ts_tablet_manager.cc:1875] T 8f86d1517df74ead9e397ca9bed44c3f P 153026dd7f8943cc93fedf6f48450fb4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:29:06.109283 4646 ts_tablet_manager.cc:1888] T 8f86d1517df74ead9e397ca9bed44c3f P 153026dd7f8943cc93fedf6f48450fb4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240502 21:29:06.109443 4646 log.cc:1196] T 8f86d1517df74ead9e397ca9bed44c3f P 153026dd7f8943cc93fedf6f48450fb4: Deleting WAL directory at /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-3/wal/wals/8f86d1517df74ead9e397ca9bed44c3f
I20240502 21:29:06.110332 1256 catalog_manager.cc:4830] TS 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132:44947): tablet 8f86d1517df74ead9e397ca9bed44c3f (table rebalance_test_table_0 [id=82737b0de4b945dc8ca0c7d6b92af37c]) successfully deleted
I20240502 21:29:06.130676 4603 raft_consensus.cc:1050] T f8bee9bdcbff404280f000bdaeadf465 P 7d1e5c1b047443cbb28d7b0a9744babf: attempting to promote NON_VOTER d8d7297186694f77bb98de9174e4279c to VOTER
I20240502 21:29:06.131880 4603 consensus_queue.cc:237] T f8bee9bdcbff404280f000bdaeadf465 P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } }
I20240502 21:29:06.136132 1684 raft_consensus.cc:1261] T f8bee9bdcbff404280f000bdaeadf465 P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:29:06.136929 3856 raft_consensus.cc:1261] T f8bee9bdcbff404280f000bdaeadf465 P d8d7297186694f77bb98de9174e4279c [term 1 LEARNER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:29:06.137101 1818 raft_consensus.cc:1261] T f8bee9bdcbff404280f000bdaeadf465 P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:29:06.137542 4590 consensus_queue.cc:1035] T f8bee9bdcbff404280f000bdaeadf465 P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:29:06.138288 1955 raft_consensus.cc:1261] T f8bee9bdcbff404280f000bdaeadf465 P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:29:06.138192 4592 consensus_queue.cc:1035] T f8bee9bdcbff404280f000bdaeadf465 P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:29:06.138777 2089 raft_consensus.cc:1261] T f8bee9bdcbff404280f000bdaeadf465 P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:29:06.139817 2225 raft_consensus.cc:1261] T f8bee9bdcbff404280f000bdaeadf465 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:29:06.140645 1549 raft_consensus.cc:1261] T f8bee9bdcbff404280f000bdaeadf465 P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:29:06.141942 4369 consensus_queue.cc:1035] T f8bee9bdcbff404280f000bdaeadf465 P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:29:06.142266 3988 raft_consensus.cc:1203] T e45af71e3fc14c579b32ba4dbe2adf78 P 16a97c1394f346f0b987f97cce7371b6 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240502 21:29:06.142326 4368 consensus_queue.cc:1035] T f8bee9bdcbff404280f000bdaeadf465 P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:29:06.142828 4368 consensus_queue.cc:1035] T f8bee9bdcbff404280f000bdaeadf465 P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:29:06.143318 4368 consensus_queue.cc:1035] T f8bee9bdcbff404280f000bdaeadf465 P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:29:06.143674 4369 consensus_queue.cc:1035] T f8bee9bdcbff404280f000bdaeadf465 P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:29:06.159384 4590 raft_consensus.cc:2932] T f8bee9bdcbff404280f000bdaeadf465 P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, d8d7297186694f77bb98de9174e4279c (127.15.255.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } }
I20240502 21:29:06.160019 1819 raft_consensus.cc:2932] T f8bee9bdcbff404280f000bdaeadf465 P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, d8d7297186694f77bb98de9174e4279c (127.15.255.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } }
I20240502 21:29:06.164247 2225 raft_consensus.cc:2932] T f8bee9bdcbff404280f000bdaeadf465 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, d8d7297186694f77bb98de9174e4279c (127.15.255.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } }
I20240502 21:29:06.167050 1685 raft_consensus.cc:2932] T f8bee9bdcbff404280f000bdaeadf465 P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, d8d7297186694f77bb98de9174e4279c (127.15.255.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } }
I20240502 21:29:06.167626 1270 catalog_manager.cc:5484] T f8bee9bdcbff404280f000bdaeadf465 P 153026dd7f8943cc93fedf6f48450fb4 reported cstate change: config changed from index 5 to 6, d8d7297186694f77bb98de9174e4279c (127.15.255.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } }
I20240502 21:29:06.168437 1954 raft_consensus.cc:2932] T f8bee9bdcbff404280f000bdaeadf465 P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, d8d7297186694f77bb98de9174e4279c (127.15.255.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } }
I20240502 21:29:06.169004 3856 raft_consensus.cc:2932] T f8bee9bdcbff404280f000bdaeadf465 P d8d7297186694f77bb98de9174e4279c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, d8d7297186694f77bb98de9174e4279c (127.15.255.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } }
I20240502 21:29:06.181461 1547 raft_consensus.cc:2932] T f8bee9bdcbff404280f000bdaeadf465 P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, d8d7297186694f77bb98de9174e4279c (127.15.255.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } }
I20240502 21:29:06.190331 2089 raft_consensus.cc:2932] T f8bee9bdcbff404280f000bdaeadf465 P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, d8d7297186694f77bb98de9174e4279c (127.15.255.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } }
I20240502 21:29:06.193722 1413 consensus_queue.cc:237] T f8bee9bdcbff404280f000bdaeadf465 P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } }
I20240502 21:29:06.197383 3856 raft_consensus.cc:1261] T f8bee9bdcbff404280f000bdaeadf465 P d8d7297186694f77bb98de9174e4279c [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:29:06.197902 4369 consensus_queue.cc:1035] T f8bee9bdcbff404280f000bdaeadf465 P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:29:06.198148 1685 raft_consensus.cc:1261] T f8bee9bdcbff404280f000bdaeadf465 P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:29:06.198655 4369 consensus_queue.cc:1035] T f8bee9bdcbff404280f000bdaeadf465 P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:29:06.199155 1547 raft_consensus.cc:1261] T f8bee9bdcbff404280f000bdaeadf465 P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:29:06.199666 2225 raft_consensus.cc:1261] T f8bee9bdcbff404280f000bdaeadf465 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:29:06.201675 4592 consensus_queue.cc:1035] T f8bee9bdcbff404280f000bdaeadf465 P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:29:06.201999 4590 consensus_queue.cc:1035] T f8bee9bdcbff404280f000bdaeadf465 P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:29:06.203622 1954 raft_consensus.cc:1261] T f8bee9bdcbff404280f000bdaeadf465 P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:29:06.204722 4590 consensus_queue.cc:1035] T f8bee9bdcbff404280f000bdaeadf465 P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:29:06.205242 4590 raft_consensus.cc:2932] T f8bee9bdcbff404280f000bdaeadf465 P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } }
I20240502 21:29:06.206350 2089 raft_consensus.cc:1261] T f8bee9bdcbff404280f000bdaeadf465 P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:29:06.206429 2225 raft_consensus.cc:2932] T f8bee9bdcbff404280f000bdaeadf465 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } }
I20240502 21:29:06.208907 4592 consensus_queue.cc:1035] T f8bee9bdcbff404280f000bdaeadf465 P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:29:06.209286 1255 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f8bee9bdcbff404280f000bdaeadf465 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:29:06.210404 3856 raft_consensus.cc:2932] T f8bee9bdcbff404280f000bdaeadf465 P d8d7297186694f77bb98de9174e4279c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } }
I20240502 21:29:06.210968 1685 raft_consensus.cc:2932] T f8bee9bdcbff404280f000bdaeadf465 P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } }
I20240502 21:29:06.211877 1547 raft_consensus.cc:2932] T f8bee9bdcbff404280f000bdaeadf465 P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } }
I20240502 21:29:06.216732 1955 raft_consensus.cc:2932] T f8bee9bdcbff404280f000bdaeadf465 P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } }
I20240502 21:29:06.219432 2089 raft_consensus.cc:2932] T f8bee9bdcbff404280f000bdaeadf465 P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } }
I20240502 21:29:06.222859 1265 catalog_manager.cc:5484] T f8bee9bdcbff404280f000bdaeadf465 P 7d1e5c1b047443cbb28d7b0a9744babf reported cstate change: config changed from index 6 to 7, VOTER 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132) evicted. New cstate: current_term: 1 leader_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:29:06.234855 1798 tablet_service.cc:1514] Processing DeleteTablet for tablet f8bee9bdcbff404280f000bdaeadf465 with delete_type TABLET_DATA_TOMBSTONED (TS 153026dd7f8943cc93fedf6f48450fb4 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:60256
I20240502 21:29:06.235291 4646 tablet_replica.cc:330] T f8bee9bdcbff404280f000bdaeadf465 P 153026dd7f8943cc93fedf6f48450fb4: stopping tablet replica
I20240502 21:29:06.235495 4646 raft_consensus.cc:2229] T f8bee9bdcbff404280f000bdaeadf465 P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:29:06.235715 4646 raft_consensus.cc:2258] T f8bee9bdcbff404280f000bdaeadf465 P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:29:06.236721 4646 ts_tablet_manager.cc:1875] T f8bee9bdcbff404280f000bdaeadf465 P 153026dd7f8943cc93fedf6f48450fb4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:29:06.243067 4646 ts_tablet_manager.cc:1888] T f8bee9bdcbff404280f000bdaeadf465 P 153026dd7f8943cc93fedf6f48450fb4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240502 21:29:06.243182 4646 log.cc:1196] T f8bee9bdcbff404280f000bdaeadf465 P 153026dd7f8943cc93fedf6f48450fb4: Deleting WAL directory at /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-3/wal/wals/f8bee9bdcbff404280f000bdaeadf465
I20240502 21:29:06.243846 1256 catalog_manager.cc:4830] TS 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132:44947): tablet f8bee9bdcbff404280f000bdaeadf465 (table rebalance_test_table_3 [id=a205513324dd4db5a5ee4ed1928aefa6]) successfully deleted
I20240502 21:29:06.257416 4486 raft_consensus.cc:1050] T 4dad8773f5b54efa9ff527252bac24aa P e632c553ff4341328acf1cbaca678392: attempting to promote NON_VOTER 16a97c1394f346f0b987f97cce7371b6 to VOTER
I20240502 21:29:06.258613 4486 consensus_queue.cc:237] T 4dad8773f5b54efa9ff527252bac24aa P e632c553ff4341328acf1cbaca678392 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } }
I20240502 21:29:06.261713 3988 raft_consensus.cc:1261] T 4dad8773f5b54efa9ff527252bac24aa P 16a97c1394f346f0b987f97cce7371b6 [term 1 LEARNER]: Refusing update from remote peer e632c553ff4341328acf1cbaca678392: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:29:06.261904 1685 raft_consensus.cc:1261] T 4dad8773f5b54efa9ff527252bac24aa P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Refusing update from remote peer e632c553ff4341328acf1cbaca678392: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:29:06.262434 4488 consensus_queue.cc:1035] T 4dad8773f5b54efa9ff527252bac24aa P e632c553ff4341328acf1cbaca678392 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:29:06.262789 4685 consensus_queue.cc:1035] T 4dad8773f5b54efa9ff527252bac24aa P e632c553ff4341328acf1cbaca678392 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:29:06.274664 2089 raft_consensus.cc:1261] T 4dad8773f5b54efa9ff527252bac24aa P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Refusing update from remote peer e632c553ff4341328acf1cbaca678392: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:29:06.274791 2225 raft_consensus.cc:1261] T 4dad8773f5b54efa9ff527252bac24aa P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Refusing update from remote peer e632c553ff4341328acf1cbaca678392: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:29:06.275209 1413 raft_consensus.cc:1261] T 4dad8773f5b54efa9ff527252bac24aa P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Refusing update from remote peer e632c553ff4341328acf1cbaca678392: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:29:06.275533 4693 consensus_queue.cc:1035] T 4dad8773f5b54efa9ff527252bac24aa P e632c553ff4341328acf1cbaca678392 [LEADER]: Connected to new peer: Peer: permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:29:06.275738 1547 raft_consensus.cc:1261] T 4dad8773f5b54efa9ff527252bac24aa P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Refusing update from remote peer e632c553ff4341328acf1cbaca678392: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:29:06.275869 4684 consensus_queue.cc:1035] T 4dad8773f5b54efa9ff527252bac24aa P e632c553ff4341328acf1cbaca678392 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:29:06.276283 4685 consensus_queue.cc:1035] T 4dad8773f5b54efa9ff527252bac24aa P e632c553ff4341328acf1cbaca678392 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:29:06.276958 1818 raft_consensus.cc:1261] T 4dad8773f5b54efa9ff527252bac24aa P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Refusing update from remote peer e632c553ff4341328acf1cbaca678392: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:29:06.277328 4486 consensus_queue.cc:1035] T 4dad8773f5b54efa9ff527252bac24aa P e632c553ff4341328acf1cbaca678392 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:29:06.278417 4486 consensus_queue.cc:1035] T 4dad8773f5b54efa9ff527252bac24aa P e632c553ff4341328acf1cbaca678392 [LEADER]: Connected to new peer: Peer: permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:29:06.293531 4693 raft_consensus.cc:2932] T 4dad8773f5b54efa9ff527252bac24aa P e632c553ff4341328acf1cbaca678392 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 16a97c1394f346f0b987f97cce7371b6 (127.15.255.137) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } }
I20240502 21:29:06.294183 2089 raft_consensus.cc:2932] T 4dad8773f5b54efa9ff527252bac24aa P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 16a97c1394f346f0b987f97cce7371b6 (127.15.255.137) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } }
I20240502 21:29:06.298610 3988 raft_consensus.cc:2932] T 4dad8773f5b54efa9ff527252bac24aa P 16a97c1394f346f0b987f97cce7371b6 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 16a97c1394f346f0b987f97cce7371b6 (127.15.255.137) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } }
I20240502 21:29:06.300307 1265 catalog_manager.cc:5484] T 4dad8773f5b54efa9ff527252bac24aa P 521bb93035324cecbebed2f6f2a630f8 reported cstate change: config changed from index 5 to 6, 16a97c1394f346f0b987f97cce7371b6 (127.15.255.137) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e632c553ff4341328acf1cbaca678392" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } }
I20240502 21:29:06.307282 1547 raft_consensus.cc:2932] T 4dad8773f5b54efa9ff527252bac24aa P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 16a97c1394f346f0b987f97cce7371b6 (127.15.255.137) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } }
I20240502 21:29:06.306566 1685 raft_consensus.cc:2932] T 4dad8773f5b54efa9ff527252bac24aa P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 16a97c1394f346f0b987f97cce7371b6 (127.15.255.137) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } }
I20240502 21:29:06.317919 1413 raft_consensus.cc:2932] T 4dad8773f5b54efa9ff527252bac24aa P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 16a97c1394f346f0b987f97cce7371b6 (127.15.255.137) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } }
I20240502 21:29:06.325263 1819 raft_consensus.cc:2932] T 4dad8773f5b54efa9ff527252bac24aa P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 16a97c1394f346f0b987f97cce7371b6 (127.15.255.137) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } }
I20240502 21:29:06.345089 2225 raft_consensus.cc:2932] T 4dad8773f5b54efa9ff527252bac24aa P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 16a97c1394f346f0b987f97cce7371b6 (127.15.255.137) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } }
I20240502 21:29:06.351042 1955 consensus_queue.cc:237] T 4dad8773f5b54efa9ff527252bac24aa P e632c553ff4341328acf1cbaca678392 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } }
I20240502 21:29:06.354036 3988 raft_consensus.cc:1261] T 4dad8773f5b54efa9ff527252bac24aa P 16a97c1394f346f0b987f97cce7371b6 [term 1 FOLLOWER]: Refusing update from remote peer e632c553ff4341328acf1cbaca678392: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:29:06.354970 1413 raft_consensus.cc:1261] T 4dad8773f5b54efa9ff527252bac24aa P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Refusing update from remote peer e632c553ff4341328acf1cbaca678392: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:29:06.354995 4684 consensus_queue.cc:1035] T 4dad8773f5b54efa9ff527252bac24aa P e632c553ff4341328acf1cbaca678392 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:29:06.355604 4685 consensus_queue.cc:1035] T 4dad8773f5b54efa9ff527252bac24aa P e632c553ff4341328acf1cbaca678392 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:29:06.356158 2089 raft_consensus.cc:1261] T 4dad8773f5b54efa9ff527252bac24aa P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Refusing update from remote peer e632c553ff4341328acf1cbaca678392: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:29:06.357723 2225 raft_consensus.cc:1261] T 4dad8773f5b54efa9ff527252bac24aa P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Refusing update from remote peer e632c553ff4341328acf1cbaca678392: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:29:06.358393 1547 raft_consensus.cc:1261] T 4dad8773f5b54efa9ff527252bac24aa P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Refusing update from remote peer e632c553ff4341328acf1cbaca678392: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:29:06.358516 4684 consensus_queue.cc:1035] T 4dad8773f5b54efa9ff527252bac24aa P e632c553ff4341328acf1cbaca678392 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:29:06.358958 4684 consensus_queue.cc:1035] T 4dad8773f5b54efa9ff527252bac24aa P e632c553ff4341328acf1cbaca678392 [LEADER]: Connected to new peer: Peer: permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:29:06.359421 4684 consensus_queue.cc:1035] T 4dad8773f5b54efa9ff527252bac24aa P e632c553ff4341328acf1cbaca678392 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:29:06.361476 1685 raft_consensus.cc:1261] T 4dad8773f5b54efa9ff527252bac24aa P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Refusing update from remote peer e632c553ff4341328acf1cbaca678392: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:29:06.362341 4704 consensus_queue.cc:1035] T 4dad8773f5b54efa9ff527252bac24aa P e632c553ff4341328acf1cbaca678392 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:29:06.369454 4693 raft_consensus.cc:2932] T 4dad8773f5b54efa9ff527252bac24aa P e632c553ff4341328acf1cbaca678392 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } }
I20240502 21:29:06.370067 2226 raft_consensus.cc:2932] T 4dad8773f5b54efa9ff527252bac24aa P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } }
I20240502 21:29:06.370370 3988 raft_consensus.cc:2932] T 4dad8773f5b54efa9ff527252bac24aa P 16a97c1394f346f0b987f97cce7371b6 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } }
I20240502 21:29:06.376735 1254 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 4dad8773f5b54efa9ff527252bac24aa with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:29:06.377327 2089 raft_consensus.cc:2932] T 4dad8773f5b54efa9ff527252bac24aa P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } }
I20240502 21:29:06.380879 1265 catalog_manager.cc:5484] T 4dad8773f5b54efa9ff527252bac24aa P 6ab8df3c796f4b6cb3e29d5045ad3ea0 reported cstate change: config changed from index 6 to 7, VOTER 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132) evicted. New cstate: current_term: 1 leader_uuid: "e632c553ff4341328acf1cbaca678392" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } }
I20240502 21:29:06.386704 1413 raft_consensus.cc:2932] T 4dad8773f5b54efa9ff527252bac24aa P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } }
I20240502 21:29:06.388636 1549 raft_consensus.cc:2932] T 4dad8773f5b54efa9ff527252bac24aa P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } }
I20240502 21:29:06.393775 1685 raft_consensus.cc:2932] T 4dad8773f5b54efa9ff527252bac24aa P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } }
I20240502 21:29:06.403018 1798 tablet_service.cc:1514] Processing DeleteTablet for tablet 4dad8773f5b54efa9ff527252bac24aa with delete_type TABLET_DATA_TOMBSTONED (TS 153026dd7f8943cc93fedf6f48450fb4 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:60256
I20240502 21:29:06.403369 4646 tablet_replica.cc:330] T 4dad8773f5b54efa9ff527252bac24aa P 153026dd7f8943cc93fedf6f48450fb4: stopping tablet replica
I20240502 21:29:06.403581 4646 raft_consensus.cc:2229] T 4dad8773f5b54efa9ff527252bac24aa P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:29:06.403934 4646 raft_consensus.cc:2258] T 4dad8773f5b54efa9ff527252bac24aa P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:29:06.404845 4646 ts_tablet_manager.cc:1875] T 4dad8773f5b54efa9ff527252bac24aa P 153026dd7f8943cc93fedf6f48450fb4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:29:06.409629 4646 ts_tablet_manager.cc:1888] T 4dad8773f5b54efa9ff527252bac24aa P 153026dd7f8943cc93fedf6f48450fb4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240502 21:29:06.409768 4646 log.cc:1196] T 4dad8773f5b54efa9ff527252bac24aa P 153026dd7f8943cc93fedf6f48450fb4: Deleting WAL directory at /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-3/wal/wals/4dad8773f5b54efa9ff527252bac24aa
I20240502 21:29:06.410415 1256 catalog_manager.cc:4830] TS 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132:44947): tablet 4dad8773f5b54efa9ff527252bac24aa (table rebalance_test_table_5 [id=9cb803e5bb2545c8b992c90a894085af]) successfully deleted
I20240502 21:29:06.466401 4552 raft_consensus.cc:1050] T bcddb146822e4573837a5697d74134f7 P 6ab8df3c796f4b6cb3e29d5045ad3ea0: attempting to promote NON_VOTER 16a97c1394f346f0b987f97cce7371b6 to VOTER
I20240502 21:29:06.467466 4552 consensus_queue.cc:237] T bcddb146822e4573837a5697d74134f7 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } }
I20240502 21:29:06.473349 3988 raft_consensus.cc:1261] T bcddb146822e4573837a5697d74134f7 P 16a97c1394f346f0b987f97cce7371b6 [term 1 LEARNER]: Refusing update from remote peer 6ab8df3c796f4b6cb3e29d5045ad3ea0: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:29:06.473424 1413 raft_consensus.cc:1261] T bcddb146822e4573837a5697d74134f7 P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Refusing update from remote peer 6ab8df3c796f4b6cb3e29d5045ad3ea0: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:29:06.473493 1685 raft_consensus.cc:1261] T bcddb146822e4573837a5697d74134f7 P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Refusing update from remote peer 6ab8df3c796f4b6cb3e29d5045ad3ea0: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:29:06.473847 1819 raft_consensus.cc:1261] T bcddb146822e4573837a5697d74134f7 P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Refusing update from remote peer 6ab8df3c796f4b6cb3e29d5045ad3ea0: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:29:06.473933 4553 consensus_queue.cc:1035] T bcddb146822e4573837a5697d74134f7 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:29:06.474226 4710 consensus_queue.cc:1035] T bcddb146822e4573837a5697d74134f7 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:29:06.474778 4710 consensus_queue.cc:1035] T bcddb146822e4573837a5697d74134f7 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:29:06.475503 1549 raft_consensus.cc:1261] T bcddb146822e4573837a5697d74134f7 P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Refusing update from remote peer 6ab8df3c796f4b6cb3e29d5045ad3ea0: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:29:06.477280 4553 consensus_queue.cc:1035] T bcddb146822e4573837a5697d74134f7 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:29:06.477766 4553 consensus_queue.cc:1035] T bcddb146822e4573837a5697d74134f7 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:29:06.484336 2089 raft_consensus.cc:1261] T bcddb146822e4573837a5697d74134f7 P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Refusing update from remote peer 6ab8df3c796f4b6cb3e29d5045ad3ea0: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:29:06.484880 1955 raft_consensus.cc:1261] T bcddb146822e4573837a5697d74134f7 P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Refusing update from remote peer 6ab8df3c796f4b6cb3e29d5045ad3ea0: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:29:06.485193 4713 raft_consensus.cc:1050] T 3c88daaac3734d918955a774829532c6 P a62a935ff43d494ca35d0c417943133d: attempting to promote NON_VOTER d8d7297186694f77bb98de9174e4279c to VOTER
I20240502 21:29:06.486035 4715 consensus_queue.cc:1035] T bcddb146822e4573837a5697d74134f7 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:29:06.486342 4553 consensus_queue.cc:1035] T bcddb146822e4573837a5697d74134f7 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:29:06.486487 4713 consensus_queue.cc:237] T 3c88daaac3734d918955a774829532c6 P a62a935ff43d494ca35d0c417943133d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } attrs { replace: true } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } }
I20240502 21:29:06.497413 3856 raft_consensus.cc:1261] T 3c88daaac3734d918955a774829532c6 P d8d7297186694f77bb98de9174e4279c [term 1 LEARNER]: Refusing update from remote peer a62a935ff43d494ca35d0c417943133d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:29:06.498255 1685 raft_consensus.cc:1261] T 3c88daaac3734d918955a774829532c6 P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Refusing update from remote peer a62a935ff43d494ca35d0c417943133d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:29:06.498347 4443 consensus_queue.cc:1035] T 3c88daaac3734d918955a774829532c6 P a62a935ff43d494ca35d0c417943133d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:29:06.500468 4443 consensus_queue.cc:1035] T 3c88daaac3734d918955a774829532c6 P a62a935ff43d494ca35d0c417943133d [LEADER]: Connected to new peer: Peer: permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:29:06.502279 4553 raft_consensus.cc:2932] T bcddb146822e4573837a5697d74134f7 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 16a97c1394f346f0b987f97cce7371b6 (127.15.255.137) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } }
I20240502 21:29:06.502772 3988 raft_consensus.cc:2932] T bcddb146822e4573837a5697d74134f7 P 16a97c1394f346f0b987f97cce7371b6 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 16a97c1394f346f0b987f97cce7371b6 (127.15.255.137) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } }
I20240502 21:29:06.507120 1414 raft_consensus.cc:1261] T 3c88daaac3734d918955a774829532c6 P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Refusing update from remote peer a62a935ff43d494ca35d0c417943133d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:29:06.509145 1265 catalog_manager.cc:5484] T bcddb146822e4573837a5697d74134f7 P 16a97c1394f346f0b987f97cce7371b6 reported cstate change: config changed from index 5 to 6, 16a97c1394f346f0b987f97cce7371b6 (127.15.255.137) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } }
I20240502 21:29:06.509814 1686 raft_consensus.cc:2932] T bcddb146822e4573837a5697d74134f7 P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 16a97c1394f346f0b987f97cce7371b6 (127.15.255.137) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } }
I20240502 21:29:06.510972 4443 consensus_queue.cc:1035] T 3c88daaac3734d918955a774829532c6 P a62a935ff43d494ca35d0c417943133d [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:29:06.510519 1549 raft_consensus.cc:2932] T bcddb146822e4573837a5697d74134f7 P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 16a97c1394f346f0b987f97cce7371b6 (127.15.255.137) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } }
I20240502 21:29:06.516678 1955 raft_consensus.cc:2932] T bcddb146822e4573837a5697d74134f7 P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 16a97c1394f346f0b987f97cce7371b6 (127.15.255.137) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } }
I20240502 21:29:06.517537 2089 raft_consensus.cc:2932] T bcddb146822e4573837a5697d74134f7 P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 16a97c1394f346f0b987f97cce7371b6 (127.15.255.137) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } }
I20240502 21:29:06.524636 1819 raft_consensus.cc:2932] T bcddb146822e4573837a5697d74134f7 P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 16a97c1394f346f0b987f97cce7371b6 (127.15.255.137) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } }
I20240502 21:29:06.534027 2226 raft_consensus.cc:1261] T 3c88daaac3734d918955a774829532c6 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Refusing update from remote peer a62a935ff43d494ca35d0c417943133d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:29:06.534716 2087 raft_consensus.cc:1261] T 3c88daaac3734d918955a774829532c6 P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Refusing update from remote peer a62a935ff43d494ca35d0c417943133d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:29:06.534659 4443 consensus_queue.cc:1035] T 3c88daaac3734d918955a774829532c6 P a62a935ff43d494ca35d0c417943133d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:29:06.535460 4443 consensus_queue.cc:1035] T 3c88daaac3734d918955a774829532c6 P a62a935ff43d494ca35d0c417943133d [LEADER]: Connected to new peer: Peer: permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:29:06.536514 1955 raft_consensus.cc:1261] T 3c88daaac3734d918955a774829532c6 P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Refusing update from remote peer a62a935ff43d494ca35d0c417943133d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:29:06.537045 1819 raft_consensus.cc:1261] T 3c88daaac3734d918955a774829532c6 P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Refusing update from remote peer a62a935ff43d494ca35d0c417943133d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:29:06.537313 4443 consensus_queue.cc:1035] T 3c88daaac3734d918955a774829532c6 P a62a935ff43d494ca35d0c417943133d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:29:06.537899 4730 consensus_queue.cc:1035] T 3c88daaac3734d918955a774829532c6 P a62a935ff43d494ca35d0c417943133d [LEADER]: Connected to new peer: Peer: permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20240502 21:29:06.542749 1413 raft_consensus.cc:2932] T bcddb146822e4573837a5697d74134f7 P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 16a97c1394f346f0b987f97cce7371b6 (127.15.255.137) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } }
I20240502 21:29:06.545831 2226 consensus_queue.cc:237] T bcddb146822e4573837a5697d74134f7 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } }
I20240502 21:29:06.548585 1686 raft_consensus.cc:1261] T bcddb146822e4573837a5697d74134f7 P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Refusing update from remote peer 6ab8df3c796f4b6cb3e29d5045ad3ea0: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:29:06.548858 1549 raft_consensus.cc:1261] T bcddb146822e4573837a5697d74134f7 P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Refusing update from remote peer 6ab8df3c796f4b6cb3e29d5045ad3ea0: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:29:06.549106 3988 raft_consensus.cc:1261] T bcddb146822e4573837a5697d74134f7 P 16a97c1394f346f0b987f97cce7371b6 [term 1 FOLLOWER]: Refusing update from remote peer 6ab8df3c796f4b6cb3e29d5045ad3ea0: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:29:06.549695 4552 consensus_queue.cc:1035] T bcddb146822e4573837a5697d74134f7 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:29:06.550030 4710 consensus_queue.cc:1035] T bcddb146822e4573837a5697d74134f7 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:29:06.550253 4714 consensus_queue.cc:1035] T bcddb146822e4573837a5697d74134f7 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:29:06.550973 1819 raft_consensus.cc:1261] T bcddb146822e4573837a5697d74134f7 P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Refusing update from remote peer 6ab8df3c796f4b6cb3e29d5045ad3ea0: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:29:06.551421 4710 consensus_queue.cc:1035] T bcddb146822e4573837a5697d74134f7 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:29:06.554461 1954 raft_consensus.cc:1261] T bcddb146822e4573837a5697d74134f7 P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Refusing update from remote peer 6ab8df3c796f4b6cb3e29d5045ad3ea0: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:29:06.555611 4552 raft_consensus.cc:2932] T bcddb146822e4573837a5697d74134f7 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 521bb93035324cecbebed2f6f2a630f8 (127.15.255.134) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } }
I20240502 21:29:06.556679 4716 consensus_queue.cc:1035] T bcddb146822e4573837a5697d74134f7 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:29:06.557126 4729 raft_consensus.cc:2932] T 3c88daaac3734d918955a774829532c6 P a62a935ff43d494ca35d0c417943133d [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, d8d7297186694f77bb98de9174e4279c (127.15.255.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } attrs { replace: true } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } }
I20240502 21:29:06.558387 1954 raft_consensus.cc:2932] T bcddb146822e4573837a5697d74134f7 P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 521bb93035324cecbebed2f6f2a630f8 (127.15.255.134) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } }
I20240502 21:29:06.557812 3856 raft_consensus.cc:2932] T 3c88daaac3734d918955a774829532c6 P d8d7297186694f77bb98de9174e4279c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, d8d7297186694f77bb98de9174e4279c (127.15.255.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } attrs { replace: true } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } }
I20240502 21:29:06.561558 1686 raft_consensus.cc:2932] T bcddb146822e4573837a5697d74134f7 P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 521bb93035324cecbebed2f6f2a630f8 (127.15.255.134) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } }
I20240502 21:29:06.563005 3988 raft_consensus.cc:2932] T bcddb146822e4573837a5697d74134f7 P 16a97c1394f346f0b987f97cce7371b6 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 521bb93035324cecbebed2f6f2a630f8 (127.15.255.134) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } }
I20240502 21:29:06.567868 2226 raft_consensus.cc:2932] T 3c88daaac3734d918955a774829532c6 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, d8d7297186694f77bb98de9174e4279c (127.15.255.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } attrs { replace: true } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } }
I20240502 21:29:06.568688 1953 raft_consensus.cc:2932] T 3c88daaac3734d918955a774829532c6 P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, d8d7297186694f77bb98de9174e4279c (127.15.255.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } attrs { replace: true } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } }
I20240502 21:29:06.568953 2087 raft_consensus.cc:2932] T 3c88daaac3734d918955a774829532c6 P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, d8d7297186694f77bb98de9174e4279c (127.15.255.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } attrs { replace: true } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } }
I20240502 21:29:06.573777 1269 catalog_manager.cc:5484] T 3c88daaac3734d918955a774829532c6 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 reported cstate change: config changed from index 5 to 6, d8d7297186694f77bb98de9174e4279c (127.15.255.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a62a935ff43d494ca35d0c417943133d" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } attrs { replace: true } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } }
I20240502 21:29:06.574860 1255 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet bcddb146822e4573837a5697d74134f7 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:29:06.574494 1686 raft_consensus.cc:2932] T 3c88daaac3734d918955a774829532c6 P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, d8d7297186694f77bb98de9174e4279c (127.15.255.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } attrs { replace: true } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } }
I20240502 21:29:06.575492 1262 catalog_manager.cc:5484] T bcddb146822e4573837a5697d74134f7 P 16b3a337bc5e462f83ba895ad8660deb reported cstate change: config changed from index 6 to 7, VOTER 521bb93035324cecbebed2f6f2a630f8 (127.15.255.134) evicted. New cstate: current_term: 1 leader_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } }
I20240502 21:29:06.579444 1818 raft_consensus.cc:2932] T 3c88daaac3734d918955a774829532c6 P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, d8d7297186694f77bb98de9174e4279c (127.15.255.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } attrs { replace: true } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } }
I20240502 21:29:06.579560 1549 raft_consensus.cc:2932] T bcddb146822e4573837a5697d74134f7 P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 521bb93035324cecbebed2f6f2a630f8 (127.15.255.134) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } }
I20240502 21:29:06.580087 1819 raft_consensus.cc:2932] T bcddb146822e4573837a5697d74134f7 P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 521bb93035324cecbebed2f6f2a630f8 (127.15.255.134) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } }
I20240502 21:29:06.582538 2068 tablet_service.cc:1514] Processing DeleteTablet for tablet bcddb146822e4573837a5697d74134f7 with delete_type TABLET_DATA_TOMBSTONED (TS 521bb93035324cecbebed2f6f2a630f8 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:50230
I20240502 21:29:06.587435 4735 tablet_replica.cc:330] T bcddb146822e4573837a5697d74134f7 P 521bb93035324cecbebed2f6f2a630f8: stopping tablet replica
I20240502 21:29:06.587842 4735 raft_consensus.cc:2229] T bcddb146822e4573837a5697d74134f7 P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:29:06.588399 4735 raft_consensus.cc:2258] T bcddb146822e4573837a5697d74134f7 P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:29:06.589514 4735 ts_tablet_manager.cc:1875] T bcddb146822e4573837a5697d74134f7 P 521bb93035324cecbebed2f6f2a630f8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:29:06.591250 1415 raft_consensus.cc:1261] T bcddb146822e4573837a5697d74134f7 P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Refusing update from remote peer 6ab8df3c796f4b6cb3e29d5045ad3ea0: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:29:06.591923 4552 consensus_queue.cc:1035] T bcddb146822e4573837a5697d74134f7 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:29:06.594444 1415 raft_consensus.cc:2932] T bcddb146822e4573837a5697d74134f7 P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 521bb93035324cecbebed2f6f2a630f8 (127.15.255.134) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } }
I20240502 21:29:06.598098 1549 consensus_queue.cc:237] T 3c88daaac3734d918955a774829532c6 P a62a935ff43d494ca35d0c417943133d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } }
I20240502 21:29:06.601183 1686 raft_consensus.cc:1261] T 3c88daaac3734d918955a774829532c6 P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Refusing update from remote peer a62a935ff43d494ca35d0c417943133d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:29:06.601449 2226 raft_consensus.cc:1261] T 3c88daaac3734d918955a774829532c6 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Refusing update from remote peer a62a935ff43d494ca35d0c417943133d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:29:06.602052 4735 ts_tablet_manager.cc:1888] T bcddb146822e4573837a5697d74134f7 P 521bb93035324cecbebed2f6f2a630f8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240502 21:29:06.602193 4735 log.cc:1196] T bcddb146822e4573837a5697d74134f7 P 521bb93035324cecbebed2f6f2a630f8: Deleting WAL directory at /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-5/wal/wals/bcddb146822e4573837a5697d74134f7
I20240502 21:29:06.602102 4712 consensus_queue.cc:1035] T 3c88daaac3734d918955a774829532c6 P a62a935ff43d494ca35d0c417943133d [LEADER]: Connected to new peer: Peer: permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:29:06.602423 4443 consensus_queue.cc:1035] T 3c88daaac3734d918955a774829532c6 P a62a935ff43d494ca35d0c417943133d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:29:06.602983 1254 catalog_manager.cc:4830] TS 521bb93035324cecbebed2f6f2a630f8 (127.15.255.134:40821): tablet bcddb146822e4573837a5697d74134f7 (table rebalance_test_table_3 [id=a205513324dd4db5a5ee4ed1928aefa6]) successfully deleted
I20240502 21:29:06.603075 1413 raft_consensus.cc:1261] T 3c88daaac3734d918955a774829532c6 P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Refusing update from remote peer a62a935ff43d494ca35d0c417943133d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:29:06.603889 4729 consensus_queue.cc:1035] T 3c88daaac3734d918955a774829532c6 P a62a935ff43d494ca35d0c417943133d [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:29:06.604465 1819 raft_consensus.cc:1261] T 3c88daaac3734d918955a774829532c6 P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Refusing update from remote peer a62a935ff43d494ca35d0c417943133d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:29:06.605016 4730 consensus_queue.cc:1035] T 3c88daaac3734d918955a774829532c6 P a62a935ff43d494ca35d0c417943133d [LEADER]: Connected to new peer: Peer: permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:29:06.605096 2087 raft_consensus.cc:1261] T 3c88daaac3734d918955a774829532c6 P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Refusing update from remote peer a62a935ff43d494ca35d0c417943133d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:29:06.605420 3856 raft_consensus.cc:1261] T 3c88daaac3734d918955a774829532c6 P d8d7297186694f77bb98de9174e4279c [term 1 FOLLOWER]: Refusing update from remote peer a62a935ff43d494ca35d0c417943133d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:29:06.605846 4730 consensus_queue.cc:1035] T 3c88daaac3734d918955a774829532c6 P a62a935ff43d494ca35d0c417943133d [LEADER]: Connected to new peer: Peer: permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:29:06.606578 4730 consensus_queue.cc:1035] T 3c88daaac3734d918955a774829532c6 P a62a935ff43d494ca35d0c417943133d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:29:06.606459 1413 raft_consensus.cc:2932] T 3c88daaac3734d918955a774829532c6 P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, d8d7297186694f77bb98de9174e4279c (127.15.255.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } attrs { replace: true } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } }
I20240502 21:29:06.609215 4712 raft_consensus.cc:2932] T 3c88daaac3734d918955a774829532c6 P a62a935ff43d494ca35d0c417943133d [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER e632c553ff4341328acf1cbaca678392 (127.15.255.133) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } }
I20240502 21:29:06.609980 1819 raft_consensus.cc:2932] T 3c88daaac3734d918955a774829532c6 P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER e632c553ff4341328acf1cbaca678392 (127.15.255.133) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } }
I20240502 21:29:06.615079 1255 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 3c88daaac3734d918955a774829532c6 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:29:06.614740 3856 raft_consensus.cc:2932] T 3c88daaac3734d918955a774829532c6 P d8d7297186694f77bb98de9174e4279c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER e632c553ff4341328acf1cbaca678392 (127.15.255.133) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } }
I20240502 21:29:06.615844 2087 raft_consensus.cc:2932] T 3c88daaac3734d918955a774829532c6 P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER e632c553ff4341328acf1cbaca678392 (127.15.255.133) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } }
I20240502 21:29:06.618220 1263 catalog_manager.cc:5484] T 3c88daaac3734d918955a774829532c6 P 153026dd7f8943cc93fedf6f48450fb4 reported cstate change: config changed from index 6 to 7, VOTER e632c553ff4341328acf1cbaca678392 (127.15.255.133) evicted. New cstate: current_term: 1 leader_uuid: "a62a935ff43d494ca35d0c417943133d" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } }
I20240502 21:29:06.622467 1931 tablet_service.cc:1514] Processing DeleteTablet for tablet 3c88daaac3734d918955a774829532c6 with delete_type TABLET_DATA_TOMBSTONED (TS e632c553ff4341328acf1cbaca678392 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:51198
I20240502 21:29:06.624174 1686 raft_consensus.cc:2932] T 3c88daaac3734d918955a774829532c6 P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER e632c553ff4341328acf1cbaca678392 (127.15.255.133) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } }
I20240502 21:29:06.629737 2226 raft_consensus.cc:2932] T 3c88daaac3734d918955a774829532c6 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER e632c553ff4341328acf1cbaca678392 (127.15.255.133) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } }
I20240502 21:29:06.634824 1413 raft_consensus.cc:2932] T 3c88daaac3734d918955a774829532c6 P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER e632c553ff4341328acf1cbaca678392 (127.15.255.133) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } }
I20240502 21:29:06.645015 4738 tablet_replica.cc:330] T 3c88daaac3734d918955a774829532c6 P e632c553ff4341328acf1cbaca678392: stopping tablet replica
I20240502 21:29:06.645201 4738 raft_consensus.cc:2229] T 3c88daaac3734d918955a774829532c6 P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:29:06.645545 4738 raft_consensus.cc:2258] T 3c88daaac3734d918955a774829532c6 P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:29:06.646337 4738 ts_tablet_manager.cc:1875] T 3c88daaac3734d918955a774829532c6 P e632c553ff4341328acf1cbaca678392: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:29:06.650986 4738 ts_tablet_manager.cc:1888] T 3c88daaac3734d918955a774829532c6 P e632c553ff4341328acf1cbaca678392: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240502 21:29:06.651109 4738 log.cc:1196] T 3c88daaac3734d918955a774829532c6 P e632c553ff4341328acf1cbaca678392: Deleting WAL directory at /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-4/wal/wals/3c88daaac3734d918955a774829532c6
I20240502 21:29:06.650832 2087 consensus_queue.cc:237] T b300ad3948074c728a25f053512bc37c P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } }
I20240502 21:29:06.651782 1254 catalog_manager.cc:4830] TS e632c553ff4341328acf1cbaca678392 (127.15.255.133:37439): tablet 3c88daaac3734d918955a774829532c6 (table rebalance_test_table_0 [id=82737b0de4b945dc8ca0c7d6b92af37c]) successfully deleted
I20240502 21:29:06.653589 1955 raft_consensus.cc:1261] T b300ad3948074c728a25f053512bc37c P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:29:06.653849 2226 raft_consensus.cc:1261] T b300ad3948074c728a25f053512bc37c P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:29:06.653896 1686 raft_consensus.cc:1261] T b300ad3948074c728a25f053512bc37c P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:29:06.654400 3545 consensus_queue.cc:1035] T b300ad3948074c728a25f053512bc37c P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:06.654759 1549 raft_consensus.cc:1261] T b300ad3948074c728a25f053512bc37c P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:29:06.654968 4504 consensus_queue.cc:1035] T b300ad3948074c728a25f053512bc37c P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:06.655532 4504 consensus_queue.cc:1035] T b300ad3948074c728a25f053512bc37c P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:06.656378 3545 consensus_queue.cc:1035] T b300ad3948074c728a25f053512bc37c P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:06.658527 1819 raft_consensus.cc:1261] T b300ad3948074c728a25f053512bc37c P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:29:06.659300 3545 consensus_queue.cc:1035] T b300ad3948074c728a25f053512bc37c P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:06.660270 1413 raft_consensus.cc:1261] T b300ad3948074c728a25f053512bc37c P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:29:06.660811 4524 consensus_queue.cc:1035] T b300ad3948074c728a25f053512bc37c P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:06.674911 4743 raft_consensus.cc:2932] T b300ad3948074c728a25f053512bc37c P 521bb93035324cecbebed2f6f2a630f8 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } }
I20240502 21:29:06.676760 1955 raft_consensus.cc:2932] T b300ad3948074c728a25f053512bc37c P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } }
I20240502 21:29:06.678350 1818 raft_consensus.cc:2932] T b300ad3948074c728a25f053512bc37c P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } }
I20240502 21:29:06.681932 1686 raft_consensus.cc:2932] T b300ad3948074c728a25f053512bc37c P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } }
I20240502 21:29:06.682324 1549 raft_consensus.cc:2932] T b300ad3948074c728a25f053512bc37c P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } }
I20240502 21:29:06.682809 1264 catalog_manager.cc:5484] T b300ad3948074c728a25f053512bc37c P e632c553ff4341328acf1cbaca678392 reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "521bb93035324cecbebed2f6f2a630f8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } }
I20240502 21:29:06.695345 2087 consensus_queue.cc:237] T b300ad3948074c728a25f053512bc37c P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: NON_VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: true } }
I20240502 21:29:06.699360 1685 raft_consensus.cc:1261] T b300ad3948074c728a25f053512bc37c P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:29:06.700037 1819 raft_consensus.cc:1261] T b300ad3948074c728a25f053512bc37c P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240502 21:29:06.700099 2022 consensus_peers.cc:485] T b300ad3948074c728a25f053512bc37c P 521bb93035324cecbebed2f6f2a630f8 -> Peer 16a97c1394f346f0b987f97cce7371b6 (127.15.255.137:43717): Couldn't send request to peer 16a97c1394f346f0b987f97cce7371b6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b300ad3948074c728a25f053512bc37c. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:29:06.700284 3545 consensus_queue.cc:1035] T b300ad3948074c728a25f053512bc37c P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:29:06.700711 3545 consensus_queue.cc:1035] T b300ad3948074c728a25f053512bc37c P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:29:06.700814 2225 raft_consensus.cc:1261] T b300ad3948074c728a25f053512bc37c P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:29:06.701830 1415 raft_consensus.cc:1261] T b300ad3948074c728a25f053512bc37c P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:29:06.702205 4504 consensus_queue.cc:1035] T b300ad3948074c728a25f053512bc37c P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:06.703073 4524 consensus_queue.cc:1035] T b300ad3948074c728a25f053512bc37c P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:06.703362 2225 raft_consensus.cc:2932] T b300ad3948074c728a25f053512bc37c P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } }
I20240502 21:29:06.705219 1955 raft_consensus.cc:1261] T b300ad3948074c728a25f053512bc37c P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:29:06.705809 3545 consensus_queue.cc:1035] T b300ad3948074c728a25f053512bc37c P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:29:06.706012 1549 raft_consensus.cc:1261] T b300ad3948074c728a25f053512bc37c P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:29:06.707227 4524 consensus_queue.cc:1035] T b300ad3948074c728a25f053512bc37c P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:29:06.707104 1415 raft_consensus.cc:2932] T b300ad3948074c728a25f053512bc37c P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } }
I20240502 21:29:06.710604 3545 raft_consensus.cc:2932] T b300ad3948074c728a25f053512bc37c P 521bb93035324cecbebed2f6f2a630f8 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 16a97c1394f346f0b987f97cce7371b6 (127.15.255.137) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: NON_VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: true } } }
I20240502 21:29:06.711251 1549 raft_consensus.cc:2932] T b300ad3948074c728a25f053512bc37c P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 16a97c1394f346f0b987f97cce7371b6 (127.15.255.137) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: NON_VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: true } } }
I20240502 21:29:06.714762 1254 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet b300ad3948074c728a25f053512bc37c with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240502 21:29:06.716064 1818 raft_consensus.cc:2932] T b300ad3948074c728a25f053512bc37c P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 16a97c1394f346f0b987f97cce7371b6 (127.15.255.137) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: NON_VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: true } } }
I20240502 21:29:06.716127 1685 raft_consensus.cc:2932] T b300ad3948074c728a25f053512bc37c P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 16a97c1394f346f0b987f97cce7371b6 (127.15.255.137) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: NON_VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: true } } }
I20240502 21:29:06.721634 1262 catalog_manager.cc:5484] T b300ad3948074c728a25f053512bc37c P 16b3a337bc5e462f83ba895ad8660deb reported cstate change: config changed from index 4 to 5, NON_VOTER 16a97c1394f346f0b987f97cce7371b6 (127.15.255.137) added. New cstate: current_term: 1 leader_uuid: "521bb93035324cecbebed2f6f2a630f8" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: NON_VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: true } } }
I20240502 21:29:06.726635 2225 raft_consensus.cc:2932] T b300ad3948074c728a25f053512bc37c P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 16a97c1394f346f0b987f97cce7371b6 (127.15.255.137) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: NON_VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: true } } }
I20240502 21:29:06.727164 1954 raft_consensus.cc:2932] T b300ad3948074c728a25f053512bc37c P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 16a97c1394f346f0b987f97cce7371b6 (127.15.255.137) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: NON_VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: true } } }
I20240502 21:29:06.735694 1415 raft_consensus.cc:2932] T b300ad3948074c728a25f053512bc37c P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 16a97c1394f346f0b987f97cce7371b6 (127.15.255.137) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: NON_VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: true } } }
I20240502 21:29:06.746191 2087 consensus_queue.cc:237] T f8de1fd0869348e6b0aa131b958ce822 P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } attrs { replace: true } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:29:06.748792 1549 raft_consensus.cc:1261] T f8de1fd0869348e6b0aa131b958ce822 P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:29:06.748950 1415 raft_consensus.cc:1261] T f8de1fd0869348e6b0aa131b958ce822 P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:29:06.748961 2225 raft_consensus.cc:1261] T f8de1fd0869348e6b0aa131b958ce822 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:29:06.749420 1685 raft_consensus.cc:1261] T f8de1fd0869348e6b0aa131b958ce822 P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:29:06.749573 1819 raft_consensus.cc:1261] T f8de1fd0869348e6b0aa131b958ce822 P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:29:06.749866 1954 raft_consensus.cc:1261] T f8de1fd0869348e6b0aa131b958ce822 P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:29:06.750253 4504 consensus_queue.cc:1035] T f8de1fd0869348e6b0aa131b958ce822 P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:06.750586 4752 consensus_queue.cc:1035] T f8de1fd0869348e6b0aa131b958ce822 P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:06.750928 4524 consensus_queue.cc:1035] T f8de1fd0869348e6b0aa131b958ce822 P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:06.751439 4524 consensus_queue.cc:1035] T f8de1fd0869348e6b0aa131b958ce822 P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:06.754091 4743 consensus_queue.cc:1035] T f8de1fd0869348e6b0aa131b958ce822 P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:06.754500 3545 consensus_queue.cc:1035] T f8de1fd0869348e6b0aa131b958ce822 P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:06.765882 4752 raft_consensus.cc:1050] T e45af71e3fc14c579b32ba4dbe2adf78 P 521bb93035324cecbebed2f6f2a630f8: attempting to promote NON_VOTER 16a97c1394f346f0b987f97cce7371b6 to VOTER
I20240502 21:29:06.767060 4752 consensus_queue.cc:237] T e45af71e3fc14c579b32ba4dbe2adf78 P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } attrs { replace: true } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } }
I20240502 21:29:06.783164 4504 raft_consensus.cc:2932] T f8de1fd0869348e6b0aa131b958ce822 P 521bb93035324cecbebed2f6f2a630f8 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } attrs { replace: true } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } }
I20240502 21:29:06.783834 2225 raft_consensus.cc:2932] T f8de1fd0869348e6b0aa131b958ce822 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } attrs { replace: true } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } }
I20240502 21:29:06.788055 1685 raft_consensus.cc:2932] T f8de1fd0869348e6b0aa131b958ce822 P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } attrs { replace: true } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } }
I20240502 21:29:06.792239 1262 catalog_manager.cc:5484] T f8de1fd0869348e6b0aa131b958ce822 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "521bb93035324cecbebed2f6f2a630f8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } attrs { replace: true } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } }
I20240502 21:29:06.792716 1549 raft_consensus.cc:2932] T f8de1fd0869348e6b0aa131b958ce822 P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } attrs { replace: true } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } }
I20240502 21:29:06.794879 1819 raft_consensus.cc:2932] T f8de1fd0869348e6b0aa131b958ce822 P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } attrs { replace: true } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } }
I20240502 21:29:06.802672 1819 raft_consensus.cc:1261] T e45af71e3fc14c579b32ba4dbe2adf78 P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:29:06.803090 1413 raft_consensus.cc:1261] T e45af71e3fc14c579b32ba4dbe2adf78 P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:29:06.803170 4524 consensus_queue.cc:1035] T e45af71e3fc14c579b32ba4dbe2adf78 P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:29:06.803393 2087 consensus_queue.cc:237] T f8de1fd0869348e6b0aa131b958ce822 P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } attrs { replace: true } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: NON_VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: true } }
I20240502 21:29:06.804769 2225 raft_consensus.cc:1261] T e45af71e3fc14c579b32ba4dbe2adf78 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:29:06.805315 4743 consensus_queue.cc:1035] T e45af71e3fc14c579b32ba4dbe2adf78 P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:29:06.807588 4743 consensus_queue.cc:1035] T e45af71e3fc14c579b32ba4dbe2adf78 P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:29:06.810184 3988 raft_consensus.cc:1261] T e45af71e3fc14c579b32ba4dbe2adf78 P 16a97c1394f346f0b987f97cce7371b6 [term 1 LEARNER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
W20240502 21:29:06.811125 2025 consensus_peers.cc:485] T f8de1fd0869348e6b0aa131b958ce822 P 521bb93035324cecbebed2f6f2a630f8 -> Peer 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138:33375): Couldn't send request to peer 1fab5794cf1740288abd82f2cebb9b1a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f8de1fd0869348e6b0aa131b958ce822. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:29:06.811197 4504 consensus_queue.cc:1035] T e45af71e3fc14c579b32ba4dbe2adf78 P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:29:06.812275 2226 raft_consensus.cc:1261] T f8de1fd0869348e6b0aa131b958ce822 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:29:06.812451 1685 raft_consensus.cc:1261] T e45af71e3fc14c579b32ba4dbe2adf78 P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:29:06.813012 4743 consensus_queue.cc:1035] T e45af71e3fc14c579b32ba4dbe2adf78 P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:29:06.813809 4743 consensus_queue.cc:1035] T f8de1fd0869348e6b0aa131b958ce822 P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:29:06.823046 1414 raft_consensus.cc:1261] T f8de1fd0869348e6b0aa131b958ce822 P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:29:06.824775 1819 raft_consensus.cc:1261] T f8de1fd0869348e6b0aa131b958ce822 P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:29:06.825419 4743 consensus_queue.cc:1035] T f8de1fd0869348e6b0aa131b958ce822 P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:29:06.825852 4743 consensus_queue.cc:1035] T f8de1fd0869348e6b0aa131b958ce822 P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:06.830808 1549 raft_consensus.cc:1261] T e45af71e3fc14c579b32ba4dbe2adf78 P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:29:06.831027 1415 raft_consensus.cc:2932] T f8de1fd0869348e6b0aa131b958ce822 P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } attrs { replace: true } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } }
I20240502 21:29:06.832754 1549 raft_consensus.cc:1261] T f8de1fd0869348e6b0aa131b958ce822 P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:29:06.834121 1686 raft_consensus.cc:1261] T f8de1fd0869348e6b0aa131b958ce822 P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:29:06.834723 4743 consensus_queue.cc:1035] T f8de1fd0869348e6b0aa131b958ce822 P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:29:06.834959 4757 consensus_queue.cc:1035] T f8de1fd0869348e6b0aa131b958ce822 P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:29:06.835107 4769 consensus_queue.cc:1035] T e45af71e3fc14c579b32ba4dbe2adf78 P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.003s
I20240502 21:29:06.837393 3545 raft_consensus.cc:2932] T f8de1fd0869348e6b0aa131b958ce822 P 521bb93035324cecbebed2f6f2a630f8 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } attrs { replace: true } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: NON_VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: true } } }
I20240502 21:29:06.841856 1254 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet f8de1fd0869348e6b0aa131b958ce822 with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240502 21:29:06.843015 1549 raft_consensus.cc:2932] T f8de1fd0869348e6b0aa131b958ce822 P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } attrs { replace: true } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: NON_VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: true } } }
I20240502 21:29:06.843004 1817 raft_consensus.cc:2932] T f8de1fd0869348e6b0aa131b958ce822 P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } attrs { replace: true } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: NON_VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: true } } }
I20240502 21:29:06.843310 2227 raft_consensus.cc:2932] T f8de1fd0869348e6b0aa131b958ce822 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } attrs { replace: true } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: NON_VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: true } } }
I20240502 21:29:06.849376 1265 catalog_manager.cc:5484] T f8de1fd0869348e6b0aa131b958ce822 P 153026dd7f8943cc93fedf6f48450fb4 reported cstate change: config changed from index 4 to 5, NON_VOTER 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138) added. New cstate: current_term: 1 leader_uuid: "521bb93035324cecbebed2f6f2a630f8" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } attrs { replace: true } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: NON_VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: true } } }
I20240502 21:29:06.851703 1686 raft_consensus.cc:2932] T f8de1fd0869348e6b0aa131b958ce822 P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } attrs { replace: true } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: NON_VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: true } } }
I20240502 21:29:06.872838 4743 raft_consensus.cc:2932] T e45af71e3fc14c579b32ba4dbe2adf78 P 521bb93035324cecbebed2f6f2a630f8 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 16a97c1394f346f0b987f97cce7371b6 (127.15.255.137) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } attrs { replace: true } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } }
I20240502 21:29:06.880537 1686 raft_consensus.cc:2932] T e45af71e3fc14c579b32ba4dbe2adf78 P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 16a97c1394f346f0b987f97cce7371b6 (127.15.255.137) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } attrs { replace: true } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } }
I20240502 21:29:06.886237 1263 catalog_manager.cc:5484] T e45af71e3fc14c579b32ba4dbe2adf78 P 16b3a337bc5e462f83ba895ad8660deb reported cstate change: config changed from index 5 to 6, 16a97c1394f346f0b987f97cce7371b6 (127.15.255.137) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "521bb93035324cecbebed2f6f2a630f8" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } attrs { replace: true } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } }
I20240502 21:29:06.891149 1414 raft_consensus.cc:2932] T f8de1fd0869348e6b0aa131b958ce822 P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } attrs { replace: true } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: NON_VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: true } } }
I20240502 21:29:06.893189 1955 raft_consensus.cc:1261] T f8de1fd0869348e6b0aa131b958ce822 P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:29:06.894171 4504 consensus_queue.cc:1035] T f8de1fd0869348e6b0aa131b958ce822 P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:06.895310 1953 raft_consensus.cc:2932] T f8de1fd0869348e6b0aa131b958ce822 P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } attrs { replace: true } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } }
I20240502 21:29:06.898016 2227 raft_consensus.cc:2932] T e45af71e3fc14c579b32ba4dbe2adf78 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 16a97c1394f346f0b987f97cce7371b6 (127.15.255.137) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } attrs { replace: true } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } }
I20240502 21:29:06.899749 1413 raft_consensus.cc:2932] T e45af71e3fc14c579b32ba4dbe2adf78 P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 16a97c1394f346f0b987f97cce7371b6 (127.15.255.137) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } attrs { replace: true } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } }
I20240502 21:29:06.905316 1953 raft_consensus.cc:2932] T f8de1fd0869348e6b0aa131b958ce822 P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } attrs { replace: true } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: NON_VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: true } } }
I20240502 21:29:06.918243 1953 raft_consensus.cc:1261] T e45af71e3fc14c579b32ba4dbe2adf78 P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:29:06.918973 1818 raft_consensus.cc:2932] T e45af71e3fc14c579b32ba4dbe2adf78 P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 16a97c1394f346f0b987f97cce7371b6 (127.15.255.137) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } attrs { replace: true } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } }
I20240502 21:29:06.921211 1549 raft_consensus.cc:2932] T e45af71e3fc14c579b32ba4dbe2adf78 P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 16a97c1394f346f0b987f97cce7371b6 (127.15.255.137) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } attrs { replace: true } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } }
I20240502 21:29:06.924324 4757 consensus_queue.cc:1035] T e45af71e3fc14c579b32ba4dbe2adf78 P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:29:06.924823 3988 raft_consensus.cc:2932] T e45af71e3fc14c579b32ba4dbe2adf78 P 16a97c1394f346f0b987f97cce7371b6 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 16a97c1394f346f0b987f97cce7371b6 (127.15.255.137) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } attrs { replace: true } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } }
I20240502 21:29:06.929296 2087 consensus_queue.cc:237] T e45af71e3fc14c579b32ba4dbe2adf78 P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } }
I20240502 21:29:06.932044 1685 raft_consensus.cc:1261] T e45af71e3fc14c579b32ba4dbe2adf78 P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:29:06.932621 4769 consensus_queue.cc:1035] T e45af71e3fc14c579b32ba4dbe2adf78 P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:29:06.933202 2227 raft_consensus.cc:1261] T e45af71e3fc14c579b32ba4dbe2adf78 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:29:06.933946 1413 raft_consensus.cc:1261] T e45af71e3fc14c579b32ba4dbe2adf78 P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:29:06.934655 4769 consensus_queue.cc:1035] T e45af71e3fc14c579b32ba4dbe2adf78 P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:29:06.935164 4766 consensus_queue.cc:1035] T e45af71e3fc14c579b32ba4dbe2adf78 P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:29:06.936477 3987 raft_consensus.cc:1261] T e45af71e3fc14c579b32ba4dbe2adf78 P 16a97c1394f346f0b987f97cce7371b6 [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:29:06.937940 4769 consensus_queue.cc:1035] T e45af71e3fc14c579b32ba4dbe2adf78 P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:29:06.940197 1817 raft_consensus.cc:1261] T e45af71e3fc14c579b32ba4dbe2adf78 P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:29:06.940892 4752 consensus_queue.cc:1035] T e45af71e3fc14c579b32ba4dbe2adf78 P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:29:06.945791 4769 raft_consensus.cc:2932] T e45af71e3fc14c579b32ba4dbe2adf78 P 521bb93035324cecbebed2f6f2a630f8 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER e632c553ff4341328acf1cbaca678392 (127.15.255.133) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } }
I20240502 21:29:06.948348 1818 raft_consensus.cc:2932] T e45af71e3fc14c579b32ba4dbe2adf78 P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER e632c553ff4341328acf1cbaca678392 (127.15.255.133) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } }
I20240502 21:29:06.949155 3987 raft_consensus.cc:2932] T e45af71e3fc14c579b32ba4dbe2adf78 P 16a97c1394f346f0b987f97cce7371b6 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER e632c553ff4341328acf1cbaca678392 (127.15.255.133) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } }
I20240502 21:29:06.949663 2227 raft_consensus.cc:2932] T e45af71e3fc14c579b32ba4dbe2adf78 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER e632c553ff4341328acf1cbaca678392 (127.15.255.133) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } }
I20240502 21:29:06.951819 1685 raft_consensus.cc:2932] T e45af71e3fc14c579b32ba4dbe2adf78 P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER e632c553ff4341328acf1cbaca678392 (127.15.255.133) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } }
I20240502 21:29:06.952607 1254 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet e45af71e3fc14c579b32ba4dbe2adf78 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:29:06.952190 1413 raft_consensus.cc:2932] T e45af71e3fc14c579b32ba4dbe2adf78 P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER e632c553ff4341328acf1cbaca678392 (127.15.255.133) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } }
I20240502 21:29:06.953207 1262 catalog_manager.cc:5484] T e45af71e3fc14c579b32ba4dbe2adf78 P 521bb93035324cecbebed2f6f2a630f8 reported cstate change: config changed from index 6 to 7, VOTER e632c553ff4341328acf1cbaca678392 (127.15.255.133) evicted. New cstate: current_term: 1 leader_uuid: "521bb93035324cecbebed2f6f2a630f8" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:29:06.970124 1931 tablet_service.cc:1514] Processing DeleteTablet for tablet e45af71e3fc14c579b32ba4dbe2adf78 with delete_type TABLET_DATA_TOMBSTONED (TS e632c553ff4341328acf1cbaca678392 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:51198
I20240502 21:29:06.974367 1953 raft_consensus.cc:2932] T e45af71e3fc14c579b32ba4dbe2adf78 P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 16a97c1394f346f0b987f97cce7371b6 (127.15.255.137) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } attrs { replace: true } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } }
I20240502 21:29:06.977857 4738 tablet_replica.cc:330] T e45af71e3fc14c579b32ba4dbe2adf78 P e632c553ff4341328acf1cbaca678392: stopping tablet replica
I20240502 21:29:06.977643 2227 consensus_queue.cc:237] T 4555e51179344acda81670fc9019d968 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } }
I20240502 21:29:06.978072 4738 raft_consensus.cc:2229] T e45af71e3fc14c579b32ba4dbe2adf78 P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:29:06.978294 4738 raft_consensus.cc:2258] T e45af71e3fc14c579b32ba4dbe2adf78 P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:29:06.979705 4738 ts_tablet_manager.cc:1875] T e45af71e3fc14c579b32ba4dbe2adf78 P e632c553ff4341328acf1cbaca678392: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:29:06.985700 4738 ts_tablet_manager.cc:1888] T e45af71e3fc14c579b32ba4dbe2adf78 P e632c553ff4341328acf1cbaca678392: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240502 21:29:06.987308 1547 raft_consensus.cc:1261] T 4555e51179344acda81670fc9019d968 P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Refusing update from remote peer 6ab8df3c796f4b6cb3e29d5045ad3ea0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:29:06.987773 2087 raft_consensus.cc:1261] T 4555e51179344acda81670fc9019d968 P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Refusing update from remote peer 6ab8df3c796f4b6cb3e29d5045ad3ea0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:29:06.987908 4710 consensus_queue.cc:1035] T 4555e51179344acda81670fc9019d968 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:06.988003 1413 raft_consensus.cc:1261] T 4555e51179344acda81670fc9019d968 P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Refusing update from remote peer 6ab8df3c796f4b6cb3e29d5045ad3ea0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:29:06.988679 4710 consensus_queue.cc:1035] T 4555e51179344acda81670fc9019d968 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:06.989012 1685 raft_consensus.cc:1261] T 4555e51179344acda81670fc9019d968 P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Refusing update from remote peer 6ab8df3c796f4b6cb3e29d5045ad3ea0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:29:06.989894 4710 consensus_queue.cc:1035] T 4555e51179344acda81670fc9019d968 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20240502 21:29:06.987324 4738 log.cc:1196] T e45af71e3fc14c579b32ba4dbe2adf78 P e632c553ff4341328acf1cbaca678392: Deleting WAL directory at /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-4/wal/wals/e45af71e3fc14c579b32ba4dbe2adf78
I20240502 21:29:06.992923 4710 consensus_queue.cc:1035] T 4555e51179344acda81670fc9019d968 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:06.988566 1953 raft_consensus.cc:1261] T 4555e51179344acda81670fc9019d968 P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Refusing update from remote peer 6ab8df3c796f4b6cb3e29d5045ad3ea0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:29:06.993932 1549 raft_consensus.cc:1261] T e45af71e3fc14c579b32ba4dbe2adf78 P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:29:06.993897 1254 catalog_manager.cc:4830] TS e632c553ff4341328acf1cbaca678392 (127.15.255.133:37439): tablet e45af71e3fc14c579b32ba4dbe2adf78 (table rebalance_test_table_5 [id=9cb803e5bb2545c8b992c90a894085af]) successfully deleted
I20240502 21:29:06.995287 4757 consensus_queue.cc:1035] T e45af71e3fc14c579b32ba4dbe2adf78 P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:29:06.996989 4552 consensus_queue.cc:1035] T 4555e51179344acda81670fc9019d968 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:06.997057 1549 raft_consensus.cc:2932] T e45af71e3fc14c579b32ba4dbe2adf78 P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER e632c553ff4341328acf1cbaca678392 (127.15.255.133) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } }
I20240502 21:29:07.005240 1818 raft_consensus.cc:1261] T 4555e51179344acda81670fc9019d968 P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Refusing update from remote peer 6ab8df3c796f4b6cb3e29d5045ad3ea0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:29:07.006052 4552 consensus_queue.cc:1035] T 4555e51179344acda81670fc9019d968 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:07.021770 4710 raft_consensus.cc:2932] T 4555e51179344acda81670fc9019d968 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } }
I20240502 21:29:07.023666 2087 raft_consensus.cc:2932] T 4555e51179344acda81670fc9019d968 P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } }
I20240502 21:29:07.028347 1413 raft_consensus.cc:2932] T 4555e51179344acda81670fc9019d968 P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } }
I20240502 21:29:07.028440 1953 raft_consensus.cc:2932] T 4555e51179344acda81670fc9019d968 P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } }
I20240502 21:29:07.033221 1262 catalog_manager.cc:5484] T 4555e51179344acda81670fc9019d968 P 521bb93035324cecbebed2f6f2a630f8 reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } }
I20240502 21:29:07.035629 1686 raft_consensus.cc:2932] T 4555e51179344acda81670fc9019d968 P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } }
I20240502 21:29:07.036908 1820 raft_consensus.cc:2932] T 4555e51179344acda81670fc9019d968 P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } }
I20240502 21:29:07.042480 1547 raft_consensus.cc:2932] T 4555e51179344acda81670fc9019d968 P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } }
I20240502 21:29:07.062063 2227 consensus_queue.cc:237] T 4555e51179344acda81670fc9019d968 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: NON_VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: true } }
I20240502 21:29:07.081363 1413 raft_consensus.cc:1261] T 4555e51179344acda81670fc9019d968 P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Refusing update from remote peer 6ab8df3c796f4b6cb3e29d5045ad3ea0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:29:07.082130 4552 consensus_queue.cc:1035] T 4555e51179344acda81670fc9019d968 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:29:07.083660 1820 raft_consensus.cc:1261] T 4555e51179344acda81670fc9019d968 P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Refusing update from remote peer 6ab8df3c796f4b6cb3e29d5045ad3ea0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:29:07.084285 4552 consensus_queue.cc:1035] T 4555e51179344acda81670fc9019d968 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:29:07.086127 1547 raft_consensus.cc:1261] T 4555e51179344acda81670fc9019d968 P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Refusing update from remote peer 6ab8df3c796f4b6cb3e29d5045ad3ea0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:29:07.086694 4552 consensus_queue.cc:1035] T 4555e51179344acda81670fc9019d968 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:29:07.088124 1953 raft_consensus.cc:1261] T 4555e51179344acda81670fc9019d968 P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Refusing update from remote peer 6ab8df3c796f4b6cb3e29d5045ad3ea0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:29:07.088805 4552 consensus_queue.cc:1035] T 4555e51179344acda81670fc9019d968 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:29:07.089223 2087 raft_consensus.cc:1261] T 4555e51179344acda81670fc9019d968 P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Refusing update from remote peer 6ab8df3c796f4b6cb3e29d5045ad3ea0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:29:07.089710 4552 consensus_queue.cc:1035] T 4555e51179344acda81670fc9019d968 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:29:07.092054 4710 raft_consensus.cc:2932] T 4555e51179344acda81670fc9019d968 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: NON_VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: true } } }
I20240502 21:29:07.095216 4794 ts_tablet_manager.cc:902] T b300ad3948074c728a25f053512bc37c P 16a97c1394f346f0b987f97cce7371b6: Initiating tablet copy from peer 521bb93035324cecbebed2f6f2a630f8 (127.15.255.134:40821)
I20240502 21:29:07.096464 2087 raft_consensus.cc:2932] T 4555e51179344acda81670fc9019d968 P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: NON_VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: true } } }
I20240502 21:29:07.096738 1953 raft_consensus.cc:2932] T 4555e51179344acda81670fc9019d968 P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: NON_VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: true } } }
I20240502 21:29:07.101770 1549 raft_consensus.cc:2932] T 4555e51179344acda81670fc9019d968 P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: NON_VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: true } } }
I20240502 21:29:07.102906 1263 catalog_manager.cc:5484] T 4555e51179344acda81670fc9019d968 P e632c553ff4341328acf1cbaca678392 reported cstate change: config changed from index 4 to 5, NON_VOTER 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138) added. New cstate: current_term: 1 leader_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: NON_VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: true } } }
I20240502 21:29:07.108855 1413 raft_consensus.cc:2932] T 4555e51179344acda81670fc9019d968 P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: NON_VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: true } } }
I20240502 21:29:07.108906 1820 raft_consensus.cc:2932] T 4555e51179344acda81670fc9019d968 P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: NON_VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: true } } }
I20240502 21:29:07.113461 4794 tablet_copy_client.cc:308] T b300ad3948074c728a25f053512bc37c P 16a97c1394f346f0b987f97cce7371b6: tablet copy: Beginning tablet copy session from remote peer at address 127.15.255.134:40821
I20240502 21:29:07.114897 2113 tablet_copy_service.cc:140] P 521bb93035324cecbebed2f6f2a630f8: Received BeginTabletCopySession request for tablet b300ad3948074c728a25f053512bc37c from peer 16a97c1394f346f0b987f97cce7371b6 ({username='slave'} at 127.15.255.137:45019)
I20240502 21:29:07.115082 2113 tablet_copy_service.cc:161] P 521bb93035324cecbebed2f6f2a630f8: Beginning new tablet copy session on tablet b300ad3948074c728a25f053512bc37c from peer 16a97c1394f346f0b987f97cce7371b6 at {username='slave'} at 127.15.255.137:45019: session id = 16a97c1394f346f0b987f97cce7371b6-b300ad3948074c728a25f053512bc37c
I20240502 21:29:07.117444 2113 tablet_copy_source_session.cc:205] T b300ad3948074c728a25f053512bc37c P 521bb93035324cecbebed2f6f2a630f8: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:29:07.120326 4794 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b300ad3948074c728a25f053512bc37c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:29:07.122269 1255 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 4555e51179344acda81670fc9019d968 with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240502 21:29:07.122550 1686 raft_consensus.cc:1261] T 4555e51179344acda81670fc9019d968 P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Refusing update from remote peer 6ab8df3c796f4b6cb3e29d5045ad3ea0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:29:07.127380 4794 tablet_copy_client.cc:783] T b300ad3948074c728a25f053512bc37c P 16a97c1394f346f0b987f97cce7371b6: tablet copy: Starting download of 0 data blocks...
I20240502 21:29:07.127656 4794 tablet_copy_client.cc:647] T b300ad3948074c728a25f053512bc37c P 16a97c1394f346f0b987f97cce7371b6: tablet copy: Starting download of 1 WAL segments...
I20240502 21:29:07.127822 4715 consensus_queue.cc:1035] T 4555e51179344acda81670fc9019d968 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:29:07.129863 4794 tablet_copy_client.cc:523] T b300ad3948074c728a25f053512bc37c P 16a97c1394f346f0b987f97cce7371b6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:29:07.129770 1686 raft_consensus.cc:2932] T 4555e51179344acda81670fc9019d968 P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: NON_VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: true } } }
I20240502 21:29:07.133471 4794 tablet_bootstrap.cc:492] T b300ad3948074c728a25f053512bc37c P 16a97c1394f346f0b987f97cce7371b6: Bootstrap starting.
W20240502 21:29:07.136896 2161 consensus_peers.cc:485] T 4555e51179344acda81670fc9019d968 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 -> Peer 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138:33375): Couldn't send request to peer 1fab5794cf1740288abd82f2cebb9b1a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4555e51179344acda81670fc9019d968. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:29:07.172380 4794 tablet_bootstrap.cc:492] T b300ad3948074c728a25f053512bc37c P 16a97c1394f346f0b987f97cce7371b6: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:29:07.172802 4794 tablet_bootstrap.cc:492] T b300ad3948074c728a25f053512bc37c P 16a97c1394f346f0b987f97cce7371b6: Bootstrap complete.
I20240502 21:29:07.173069 4794 ts_tablet_manager.cc:1367] T b300ad3948074c728a25f053512bc37c P 16a97c1394f346f0b987f97cce7371b6: Time spent bootstrapping tablet: real 0.040s user 0.031s sys 0.007s
I20240502 21:29:07.173856 4794 raft_consensus.cc:348] T b300ad3948074c728a25f053512bc37c P 16a97c1394f346f0b987f97cce7371b6 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: NON_VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: true } }
I20240502 21:29:07.174119 4794 raft_consensus.cc:729] T b300ad3948074c728a25f053512bc37c P 16a97c1394f346f0b987f97cce7371b6 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 16a97c1394f346f0b987f97cce7371b6, State: Initialized, Role: LEARNER
I20240502 21:29:07.174335 4794 consensus_queue.cc:260] T b300ad3948074c728a25f053512bc37c P 16a97c1394f346f0b987f97cce7371b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: NON_VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: true } }
I20240502 21:29:07.174865 4794 ts_tablet_manager.cc:1398] T b300ad3948074c728a25f053512bc37c P 16a97c1394f346f0b987f97cce7371b6: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 21:29:07.175549 2113 tablet_copy_service.cc:342] P 521bb93035324cecbebed2f6f2a630f8: Request end of tablet copy session 16a97c1394f346f0b987f97cce7371b6-b300ad3948074c728a25f053512bc37c received from {username='slave'} at 127.15.255.137:45019
I20240502 21:29:07.175674 2113 tablet_copy_service.cc:433] P 521bb93035324cecbebed2f6f2a630f8: ending tablet copy session 16a97c1394f346f0b987f97cce7371b6-b300ad3948074c728a25f053512bc37c on tablet b300ad3948074c728a25f053512bc37c with peer 16a97c1394f346f0b987f97cce7371b6
I20240502 21:29:07.300743 4835 ts_tablet_manager.cc:902] T f8de1fd0869348e6b0aa131b958ce822 P 1fab5794cf1740288abd82f2cebb9b1a: Initiating tablet copy from peer 521bb93035324cecbebed2f6f2a630f8 (127.15.255.134:40821)
I20240502 21:29:07.301889 4835 tablet_copy_client.cc:308] T f8de1fd0869348e6b0aa131b958ce822 P 1fab5794cf1740288abd82f2cebb9b1a: tablet copy: Beginning tablet copy session from remote peer at address 127.15.255.134:40821
I20240502 21:29:07.302536 2113 tablet_copy_service.cc:140] P 521bb93035324cecbebed2f6f2a630f8: Received BeginTabletCopySession request for tablet f8de1fd0869348e6b0aa131b958ce822 from peer 1fab5794cf1740288abd82f2cebb9b1a ({username='slave'} at 127.15.255.138:37811)
I20240502 21:29:07.302671 2113 tablet_copy_service.cc:161] P 521bb93035324cecbebed2f6f2a630f8: Beginning new tablet copy session on tablet f8de1fd0869348e6b0aa131b958ce822 from peer 1fab5794cf1740288abd82f2cebb9b1a at {username='slave'} at 127.15.255.138:37811: session id = 1fab5794cf1740288abd82f2cebb9b1a-f8de1fd0869348e6b0aa131b958ce822
I20240502 21:29:07.304701 2113 tablet_copy_source_session.cc:205] T f8de1fd0869348e6b0aa131b958ce822 P 521bb93035324cecbebed2f6f2a630f8: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:29:07.305836 4835 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8de1fd0869348e6b0aa131b958ce822. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:29:07.310729 4835 tablet_copy_client.cc:783] T f8de1fd0869348e6b0aa131b958ce822 P 1fab5794cf1740288abd82f2cebb9b1a: tablet copy: Starting download of 0 data blocks...
I20240502 21:29:07.310940 4835 tablet_copy_client.cc:647] T f8de1fd0869348e6b0aa131b958ce822 P 1fab5794cf1740288abd82f2cebb9b1a: tablet copy: Starting download of 1 WAL segments...
I20240502 21:29:07.312409 4835 tablet_copy_client.cc:523] T f8de1fd0869348e6b0aa131b958ce822 P 1fab5794cf1740288abd82f2cebb9b1a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:29:07.315047 4835 tablet_bootstrap.cc:492] T f8de1fd0869348e6b0aa131b958ce822 P 1fab5794cf1740288abd82f2cebb9b1a: Bootstrap starting.
I20240502 21:29:07.351349 4835 tablet_bootstrap.cc:492] T f8de1fd0869348e6b0aa131b958ce822 P 1fab5794cf1740288abd82f2cebb9b1a: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:29:07.351760 4835 tablet_bootstrap.cc:492] T f8de1fd0869348e6b0aa131b958ce822 P 1fab5794cf1740288abd82f2cebb9b1a: Bootstrap complete.
I20240502 21:29:07.351998 4835 ts_tablet_manager.cc:1367] T f8de1fd0869348e6b0aa131b958ce822 P 1fab5794cf1740288abd82f2cebb9b1a: Time spent bootstrapping tablet: real 0.037s user 0.025s sys 0.010s
I20240502 21:29:07.353116 4835 raft_consensus.cc:348] T f8de1fd0869348e6b0aa131b958ce822 P 1fab5794cf1740288abd82f2cebb9b1a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } attrs { replace: true } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: NON_VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: true } }
I20240502 21:29:07.353446 4835 raft_consensus.cc:729] T f8de1fd0869348e6b0aa131b958ce822 P 1fab5794cf1740288abd82f2cebb9b1a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1fab5794cf1740288abd82f2cebb9b1a, State: Initialized, Role: LEARNER
I20240502 21:29:07.353668 4835 consensus_queue.cc:260] T f8de1fd0869348e6b0aa131b958ce822 P 1fab5794cf1740288abd82f2cebb9b1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } attrs { replace: true } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: NON_VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: true } }
I20240502 21:29:07.354583 4835 ts_tablet_manager.cc:1398] T f8de1fd0869348e6b0aa131b958ce822 P 1fab5794cf1740288abd82f2cebb9b1a: Time spent starting tablet: real 0.002s user 0.005s sys 0.001s
I20240502 21:29:07.355365 2113 tablet_copy_service.cc:342] P 521bb93035324cecbebed2f6f2a630f8: Request end of tablet copy session 1fab5794cf1740288abd82f2cebb9b1a-f8de1fd0869348e6b0aa131b958ce822 received from {username='slave'} at 127.15.255.138:37811
I20240502 21:29:07.355497 2113 tablet_copy_service.cc:433] P 521bb93035324cecbebed2f6f2a630f8: ending tablet copy session 1fab5794cf1740288abd82f2cebb9b1a-f8de1fd0869348e6b0aa131b958ce822 on tablet f8de1fd0869348e6b0aa131b958ce822 with peer 1fab5794cf1740288abd82f2cebb9b1a
I20240502 21:29:07.407089 1953 consensus_queue.cc:237] T 0658ecbf32754656876452b6f021926b P e632c553ff4341328acf1cbaca678392 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:29:07.410409 1686 raft_consensus.cc:1261] T 0658ecbf32754656876452b6f021926b P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Refusing update from remote peer e632c553ff4341328acf1cbaca678392: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:29:07.410804 2087 raft_consensus.cc:1261] T 0658ecbf32754656876452b6f021926b P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Refusing update from remote peer e632c553ff4341328acf1cbaca678392: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:29:07.411010 1413 raft_consensus.cc:1261] T 0658ecbf32754656876452b6f021926b P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Refusing update from remote peer e632c553ff4341328acf1cbaca678392: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:29:07.411576 4693 consensus_queue.cc:1035] T 0658ecbf32754656876452b6f021926b P e632c553ff4341328acf1cbaca678392 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:07.412009 1820 raft_consensus.cc:1261] T 0658ecbf32754656876452b6f021926b P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Refusing update from remote peer e632c553ff4341328acf1cbaca678392: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:29:07.412091 4693 consensus_queue.cc:1035] T 0658ecbf32754656876452b6f021926b P e632c553ff4341328acf1cbaca678392 [LEADER]: Connected to new peer: Peer: permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:07.412623 4693 consensus_queue.cc:1035] T 0658ecbf32754656876452b6f021926b P e632c553ff4341328acf1cbaca678392 [LEADER]: Connected to new peer: Peer: permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:07.417616 4693 raft_consensus.cc:2932] T 0658ecbf32754656876452b6f021926b P e632c553ff4341328acf1cbaca678392 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } }
I20240502 21:29:07.418416 2087 raft_consensus.cc:2932] T 0658ecbf32754656876452b6f021926b P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } }
I20240502 21:29:07.421262 2226 raft_consensus.cc:1261] T 0658ecbf32754656876452b6f021926b P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Refusing update from remote peer e632c553ff4341328acf1cbaca678392: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:29:07.421291 1547 raft_consensus.cc:1261] T 0658ecbf32754656876452b6f021926b P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Refusing update from remote peer e632c553ff4341328acf1cbaca678392: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:29:07.421836 4842 consensus_queue.cc:1035] T 0658ecbf32754656876452b6f021926b P e632c553ff4341328acf1cbaca678392 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:07.423327 1820 raft_consensus.cc:2932] T 0658ecbf32754656876452b6f021926b P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } }
I20240502 21:29:07.423627 1686 raft_consensus.cc:2932] T 0658ecbf32754656876452b6f021926b P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } }
I20240502 21:29:07.425276 1263 catalog_manager.cc:5484] T 0658ecbf32754656876452b6f021926b P 521bb93035324cecbebed2f6f2a630f8 reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "e632c553ff4341328acf1cbaca678392" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } }
I20240502 21:29:07.430665 4842 consensus_queue.cc:1035] T 0658ecbf32754656876452b6f021926b P e632c553ff4341328acf1cbaca678392 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:07.433341 4842 consensus_queue.cc:1035] T 0658ecbf32754656876452b6f021926b P e632c553ff4341328acf1cbaca678392 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:07.433068 1413 raft_consensus.cc:2932] T 0658ecbf32754656876452b6f021926b P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } }
I20240502 21:29:07.435648 2226 raft_consensus.cc:2932] T 0658ecbf32754656876452b6f021926b P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } }
I20240502 21:29:07.445492 1547 raft_consensus.cc:2932] T 0658ecbf32754656876452b6f021926b P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } }
I20240502 21:29:07.451992 1953 consensus_queue.cc:237] T 0658ecbf32754656876452b6f021926b P e632c553ff4341328acf1cbaca678392 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: NON_VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: true } }
I20240502 21:29:07.455588 1820 raft_consensus.cc:1261] T 0658ecbf32754656876452b6f021926b P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Refusing update from remote peer e632c553ff4341328acf1cbaca678392: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:29:07.456207 2087 raft_consensus.cc:1261] T 0658ecbf32754656876452b6f021926b P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Refusing update from remote peer e632c553ff4341328acf1cbaca678392: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:29:07.456197 4693 consensus_queue.cc:1035] T 0658ecbf32754656876452b6f021926b P e632c553ff4341328acf1cbaca678392 [LEADER]: Connected to new peer: Peer: permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:29:07.456797 4842 consensus_queue.cc:1035] T 0658ecbf32754656876452b6f021926b P e632c553ff4341328acf1cbaca678392 [LEADER]: Connected to new peer: Peer: permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:29:07.459137 2226 raft_consensus.cc:1261] T 0658ecbf32754656876452b6f021926b P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Refusing update from remote peer e632c553ff4341328acf1cbaca678392: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:29:07.462044 1547 raft_consensus.cc:1261] T 0658ecbf32754656876452b6f021926b P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Refusing update from remote peer e632c553ff4341328acf1cbaca678392: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:29:07.462430 4853 consensus_queue.cc:1035] T 0658ecbf32754656876452b6f021926b P e632c553ff4341328acf1cbaca678392 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:29:07.462855 4853 consensus_queue.cc:1035] T 0658ecbf32754656876452b6f021926b P e632c553ff4341328acf1cbaca678392 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:29:07.465071 1686 raft_consensus.cc:1261] T 0658ecbf32754656876452b6f021926b P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Refusing update from remote peer e632c553ff4341328acf1cbaca678392: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:29:07.465675 4849 consensus_queue.cc:1035] T 0658ecbf32754656876452b6f021926b P e632c553ff4341328acf1cbaca678392 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:29:07.468143 4693 raft_consensus.cc:2932] T 0658ecbf32754656876452b6f021926b P e632c553ff4341328acf1cbaca678392 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: NON_VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: true } } }
W20240502 21:29:07.474851 1887 consensus_peers.cc:485] T 0658ecbf32754656876452b6f021926b P e632c553ff4341328acf1cbaca678392 -> Peer 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138:33375): Couldn't send request to peer 1fab5794cf1740288abd82f2cebb9b1a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0658ecbf32754656876452b6f021926b. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:29:07.475740 1254 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 0658ecbf32754656876452b6f021926b with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240502 21:29:07.476459 1820 raft_consensus.cc:2932] T 0658ecbf32754656876452b6f021926b P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: NON_VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: true } } }
I20240502 21:29:07.476763 2087 raft_consensus.cc:2932] T 0658ecbf32754656876452b6f021926b P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: NON_VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: true } } }
I20240502 21:29:07.482620 1263 catalog_manager.cc:5484] T 0658ecbf32754656876452b6f021926b P 153026dd7f8943cc93fedf6f48450fb4 reported cstate change: config changed from index 4 to 5, NON_VOTER 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138) added. New cstate: current_term: 1 leader_uuid: "e632c553ff4341328acf1cbaca678392" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: NON_VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: true } } }
I20240502 21:29:07.483422 1413 raft_consensus.cc:1261] T 0658ecbf32754656876452b6f021926b P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Refusing update from remote peer e632c553ff4341328acf1cbaca678392: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:29:07.483985 1686 raft_consensus.cc:2932] T 0658ecbf32754656876452b6f021926b P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: NON_VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: true } } }
I20240502 21:29:07.485216 4842 consensus_queue.cc:1035] T 0658ecbf32754656876452b6f021926b P e632c553ff4341328acf1cbaca678392 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:29:07.486047 1547 raft_consensus.cc:2932] T 0658ecbf32754656876452b6f021926b P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: NON_VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: true } } }
I20240502 21:29:07.487208 1413 raft_consensus.cc:2932] T 0658ecbf32754656876452b6f021926b P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: NON_VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: true } } }
I20240502 21:29:07.492066 2226 raft_consensus.cc:2932] T 0658ecbf32754656876452b6f021926b P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: NON_VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: true } } }
I20240502 21:29:07.550206 3987 raft_consensus.cc:1203] T b300ad3948074c728a25f053512bc37c P 16a97c1394f346f0b987f97cce7371b6 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240502 21:29:07.619460 4835 ts_tablet_manager.cc:902] T 4555e51179344acda81670fc9019d968 P 1fab5794cf1740288abd82f2cebb9b1a: Initiating tablet copy from peer 6ab8df3c796f4b6cb3e29d5045ad3ea0 (127.15.255.135:34701)
I20240502 21:29:07.620625 4835 tablet_copy_client.cc:308] T 4555e51179344acda81670fc9019d968 P 1fab5794cf1740288abd82f2cebb9b1a: tablet copy: Beginning tablet copy session from remote peer at address 127.15.255.135:34701
I20240502 21:29:07.627002 2249 tablet_copy_service.cc:140] P 6ab8df3c796f4b6cb3e29d5045ad3ea0: Received BeginTabletCopySession request for tablet 4555e51179344acda81670fc9019d968 from peer 1fab5794cf1740288abd82f2cebb9b1a ({username='slave'} at 127.15.255.138:49711)
I20240502 21:29:07.627144 2249 tablet_copy_service.cc:161] P 6ab8df3c796f4b6cb3e29d5045ad3ea0: Beginning new tablet copy session on tablet 4555e51179344acda81670fc9019d968 from peer 1fab5794cf1740288abd82f2cebb9b1a at {username='slave'} at 127.15.255.138:49711: session id = 1fab5794cf1740288abd82f2cebb9b1a-4555e51179344acda81670fc9019d968
I20240502 21:29:07.629567 2249 tablet_copy_source_session.cc:205] T 4555e51179344acda81670fc9019d968 P 6ab8df3c796f4b6cb3e29d5045ad3ea0: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:29:07.630676 4835 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4555e51179344acda81670fc9019d968. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:29:07.635464 4835 tablet_copy_client.cc:783] T 4555e51179344acda81670fc9019d968 P 1fab5794cf1740288abd82f2cebb9b1a: tablet copy: Starting download of 0 data blocks...
I20240502 21:29:07.635651 4835 tablet_copy_client.cc:647] T 4555e51179344acda81670fc9019d968 P 1fab5794cf1740288abd82f2cebb9b1a: tablet copy: Starting download of 1 WAL segments...
I20240502 21:29:07.637198 4835 tablet_copy_client.cc:523] T 4555e51179344acda81670fc9019d968 P 1fab5794cf1740288abd82f2cebb9b1a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:29:07.639398 4835 tablet_bootstrap.cc:492] T 4555e51179344acda81670fc9019d968 P 1fab5794cf1740288abd82f2cebb9b1a: Bootstrap starting.
I20240502 21:29:07.668375 4835 tablet_bootstrap.cc:492] T 4555e51179344acda81670fc9019d968 P 1fab5794cf1740288abd82f2cebb9b1a: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:29:07.668776 4835 tablet_bootstrap.cc:492] T 4555e51179344acda81670fc9019d968 P 1fab5794cf1740288abd82f2cebb9b1a: Bootstrap complete.
I20240502 21:29:07.669075 4835 ts_tablet_manager.cc:1367] T 4555e51179344acda81670fc9019d968 P 1fab5794cf1740288abd82f2cebb9b1a: Time spent bootstrapping tablet: real 0.030s user 0.018s sys 0.013s
I20240502 21:29:07.669788 4835 raft_consensus.cc:348] T 4555e51179344acda81670fc9019d968 P 1fab5794cf1740288abd82f2cebb9b1a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: NON_VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: true } }
I20240502 21:29:07.670014 4835 raft_consensus.cc:729] T 4555e51179344acda81670fc9019d968 P 1fab5794cf1740288abd82f2cebb9b1a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1fab5794cf1740288abd82f2cebb9b1a, State: Initialized, Role: LEARNER
I20240502 21:29:07.670214 4835 consensus_queue.cc:260] T 4555e51179344acda81670fc9019d968 P 1fab5794cf1740288abd82f2cebb9b1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: NON_VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: true } }
I20240502 21:29:07.670709 4835 ts_tablet_manager.cc:1398] T 4555e51179344acda81670fc9019d968 P 1fab5794cf1740288abd82f2cebb9b1a: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 21:29:07.671473 2249 tablet_copy_service.cc:342] P 6ab8df3c796f4b6cb3e29d5045ad3ea0: Request end of tablet copy session 1fab5794cf1740288abd82f2cebb9b1a-4555e51179344acda81670fc9019d968 received from {username='slave'} at 127.15.255.138:49711
I20240502 21:29:07.671605 2249 tablet_copy_service.cc:433] P 6ab8df3c796f4b6cb3e29d5045ad3ea0: ending tablet copy session 1fab5794cf1740288abd82f2cebb9b1a-4555e51179344acda81670fc9019d968 on tablet 4555e51179344acda81670fc9019d968 with peer 1fab5794cf1740288abd82f2cebb9b1a
I20240502 21:29:07.765892 4121 raft_consensus.cc:1203] T f8de1fd0869348e6b0aa131b958ce822 P 1fab5794cf1740288abd82f2cebb9b1a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240502 21:29:08.040683 4504 raft_consensus.cc:1050] T b300ad3948074c728a25f053512bc37c P 521bb93035324cecbebed2f6f2a630f8: attempting to promote NON_VOTER 16a97c1394f346f0b987f97cce7371b6 to VOTER
I20240502 21:29:08.041586 4504 consensus_queue.cc:237] T b300ad3948074c728a25f053512bc37c P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } }
I20240502 21:29:08.044281 3987 raft_consensus.cc:1261] T b300ad3948074c728a25f053512bc37c P 16a97c1394f346f0b987f97cce7371b6 [term 1 LEARNER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:29:08.044411 1820 raft_consensus.cc:1261] T b300ad3948074c728a25f053512bc37c P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:29:08.044623 2227 raft_consensus.cc:1261] T b300ad3948074c728a25f053512bc37c P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:29:08.045118 4769 consensus_queue.cc:1035] T b300ad3948074c728a25f053512bc37c P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:29:08.045418 4757 consensus_queue.cc:1035] T b300ad3948074c728a25f053512bc37c P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:29:08.045994 4757 consensus_queue.cc:1035] T b300ad3948074c728a25f053512bc37c P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:29:08.050762 1953 raft_consensus.cc:1261] T b300ad3948074c728a25f053512bc37c P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:29:08.052213 1685 raft_consensus.cc:1261] T b300ad3948074c728a25f053512bc37c P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:29:08.052842 4757 consensus_queue.cc:1035] T b300ad3948074c728a25f053512bc37c P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:29:08.053241 4769 consensus_queue.cc:1035] T b300ad3948074c728a25f053512bc37c P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:29:08.054360 4121 raft_consensus.cc:1203] T 4555e51179344acda81670fc9019d968 P 1fab5794cf1740288abd82f2cebb9b1a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240502 21:29:08.060564 1413 raft_consensus.cc:1261] T b300ad3948074c728a25f053512bc37c P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:29:08.060729 1549 raft_consensus.cc:1261] T b300ad3948074c728a25f053512bc37c P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:29:08.061290 4757 consensus_queue.cc:1035] T b300ad3948074c728a25f053512bc37c P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:29:08.061767 4757 consensus_queue.cc:1035] T b300ad3948074c728a25f053512bc37c P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:29:08.064949 4835 ts_tablet_manager.cc:902] T 0658ecbf32754656876452b6f021926b P 1fab5794cf1740288abd82f2cebb9b1a: Initiating tablet copy from peer e632c553ff4341328acf1cbaca678392 (127.15.255.133:37439)
I20240502 21:29:08.070392 1686 raft_consensus.cc:2932] T b300ad3948074c728a25f053512bc37c P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 16a97c1394f346f0b987f97cce7371b6 (127.15.255.137) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } }
I20240502 21:29:08.069643 4769 raft_consensus.cc:2932] T b300ad3948074c728a25f053512bc37c P 521bb93035324cecbebed2f6f2a630f8 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 16a97c1394f346f0b987f97cce7371b6 (127.15.255.137) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } }
I20240502 21:29:08.076867 1262 catalog_manager.cc:5484] T b300ad3948074c728a25f053512bc37c P 16b3a337bc5e462f83ba895ad8660deb reported cstate change: config changed from index 5 to 6, 16a97c1394f346f0b987f97cce7371b6 (127.15.255.137) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "521bb93035324cecbebed2f6f2a630f8" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } }
I20240502 21:29:08.077404 1820 raft_consensus.cc:2932] T b300ad3948074c728a25f053512bc37c P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 16a97c1394f346f0b987f97cce7371b6 (127.15.255.137) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } }
I20240502 21:29:08.076990 1953 raft_consensus.cc:2932] T b300ad3948074c728a25f053512bc37c P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 16a97c1394f346f0b987f97cce7371b6 (127.15.255.137) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } }
I20240502 21:29:08.078759 3987 raft_consensus.cc:2932] T b300ad3948074c728a25f053512bc37c P 16a97c1394f346f0b987f97cce7371b6 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 16a97c1394f346f0b987f97cce7371b6 (127.15.255.137) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } }
I20240502 21:29:08.086242 1414 raft_consensus.cc:2932] T b300ad3948074c728a25f053512bc37c P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 16a97c1394f346f0b987f97cce7371b6 (127.15.255.137) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } }
I20240502 21:29:08.088541 1549 raft_consensus.cc:2932] T b300ad3948074c728a25f053512bc37c P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 16a97c1394f346f0b987f97cce7371b6 (127.15.255.137) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } }
I20240502 21:29:08.095785 2226 raft_consensus.cc:2932] T b300ad3948074c728a25f053512bc37c P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 16a97c1394f346f0b987f97cce7371b6 (127.15.255.137) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } }
I20240502 21:29:08.096277 4835 tablet_copy_client.cc:308] T 0658ecbf32754656876452b6f021926b P 1fab5794cf1740288abd82f2cebb9b1a: tablet copy: Beginning tablet copy session from remote peer at address 127.15.255.133:37439
I20240502 21:29:08.105648 1975 tablet_copy_service.cc:140] P e632c553ff4341328acf1cbaca678392: Received BeginTabletCopySession request for tablet 0658ecbf32754656876452b6f021926b from peer 1fab5794cf1740288abd82f2cebb9b1a ({username='slave'} at 127.15.255.138:55671)
I20240502 21:29:08.105844 1975 tablet_copy_service.cc:161] P e632c553ff4341328acf1cbaca678392: Beginning new tablet copy session on tablet 0658ecbf32754656876452b6f021926b from peer 1fab5794cf1740288abd82f2cebb9b1a at {username='slave'} at 127.15.255.138:55671: session id = 1fab5794cf1740288abd82f2cebb9b1a-0658ecbf32754656876452b6f021926b
I20240502 21:29:08.108637 1975 tablet_copy_source_session.cc:205] T 0658ecbf32754656876452b6f021926b P e632c553ff4341328acf1cbaca678392: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:29:08.109943 4835 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0658ecbf32754656876452b6f021926b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:29:08.111945 2087 consensus_queue.cc:237] T b300ad3948074c728a25f053512bc37c P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } }
I20240502 21:29:08.114732 3987 raft_consensus.cc:1261] T b300ad3948074c728a25f053512bc37c P 16a97c1394f346f0b987f97cce7371b6 [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:29:08.115209 2226 raft_consensus.cc:1261] T b300ad3948074c728a25f053512bc37c P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:29:08.115314 4835 tablet_copy_client.cc:783] T 0658ecbf32754656876452b6f021926b P 1fab5794cf1740288abd82f2cebb9b1a: tablet copy: Starting download of 0 data blocks...
I20240502 21:29:08.115511 1549 raft_consensus.cc:1261] T b300ad3948074c728a25f053512bc37c P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:29:08.115538 4835 tablet_copy_client.cc:647] T 0658ecbf32754656876452b6f021926b P 1fab5794cf1740288abd82f2cebb9b1a: tablet copy: Starting download of 1 WAL segments...
I20240502 21:29:08.115727 4504 consensus_queue.cc:1035] T b300ad3948074c728a25f053512bc37c P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:29:08.116515 4504 consensus_queue.cc:1035] T b300ad3948074c728a25f053512bc37c P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:29:08.116668 1686 raft_consensus.cc:1261] T b300ad3948074c728a25f053512bc37c P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:29:08.117137 4835 tablet_copy_client.cc:523] T 0658ecbf32754656876452b6f021926b P 1fab5794cf1740288abd82f2cebb9b1a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:29:08.117589 1953 raft_consensus.cc:1261] T b300ad3948074c728a25f053512bc37c P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:29:08.117620 4769 consensus_queue.cc:1035] T b300ad3948074c728a25f053512bc37c P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:29:08.118259 4932 consensus_queue.cc:1035] T b300ad3948074c728a25f053512bc37c P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:29:08.119329 4932 consensus_queue.cc:1035] T b300ad3948074c728a25f053512bc37c P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:29:08.120397 4835 tablet_bootstrap.cc:492] T 0658ecbf32754656876452b6f021926b P 1fab5794cf1740288abd82f2cebb9b1a: Bootstrap starting.
I20240502 21:29:08.123250 1414 raft_consensus.cc:1261] T b300ad3948074c728a25f053512bc37c P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:29:08.124316 4932 consensus_queue.cc:1035] T b300ad3948074c728a25f053512bc37c P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:29:08.123793 4949 raft_consensus.cc:2932] T b300ad3948074c728a25f053512bc37c P 521bb93035324cecbebed2f6f2a630f8 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } }
I20240502 21:29:08.125401 1685 raft_consensus.cc:2932] T b300ad3948074c728a25f053512bc37c P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } }
I20240502 21:29:08.126046 1549 raft_consensus.cc:2932] T b300ad3948074c728a25f053512bc37c P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } }
I20240502 21:29:08.126456 1414 raft_consensus.cc:2932] T b300ad3948074c728a25f053512bc37c P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } }
I20240502 21:29:08.129392 1254 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet b300ad3948074c728a25f053512bc37c with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:29:08.130851 3987 raft_consensus.cc:2932] T b300ad3948074c728a25f053512bc37c P 16a97c1394f346f0b987f97cce7371b6 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } }
I20240502 21:29:08.131561 1268 catalog_manager.cc:5484] T b300ad3948074c728a25f053512bc37c P 16b3a337bc5e462f83ba895ad8660deb reported cstate change: config changed from index 6 to 7, VOTER 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132) evicted. New cstate: current_term: 1 leader_uuid: "521bb93035324cecbebed2f6f2a630f8" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } }
I20240502 21:29:08.134083 2226 raft_consensus.cc:2932] T b300ad3948074c728a25f053512bc37c P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } }
I20240502 21:29:08.135622 1953 raft_consensus.cc:2932] T b300ad3948074c728a25f053512bc37c P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } }
I20240502 21:29:08.141717 1798 tablet_service.cc:1514] Processing DeleteTablet for tablet b300ad3948074c728a25f053512bc37c with delete_type TABLET_DATA_TOMBSTONED (TS 153026dd7f8943cc93fedf6f48450fb4 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:60256
I20240502 21:29:08.150364 4504 raft_consensus.cc:1050] T f8de1fd0869348e6b0aa131b958ce822 P 521bb93035324cecbebed2f6f2a630f8: attempting to promote NON_VOTER 1fab5794cf1740288abd82f2cebb9b1a to VOTER
I20240502 21:29:08.151810 4504 consensus_queue.cc:237] T f8de1fd0869348e6b0aa131b958ce822 P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } attrs { replace: true } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } }
I20240502 21:29:08.162298 4950 tablet_replica.cc:330] T b300ad3948074c728a25f053512bc37c P 153026dd7f8943cc93fedf6f48450fb4: stopping tablet replica
I20240502 21:29:08.162534 4950 raft_consensus.cc:2229] T b300ad3948074c728a25f053512bc37c P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:29:08.163004 4950 raft_consensus.cc:2258] T b300ad3948074c728a25f053512bc37c P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:29:08.163816 1685 raft_consensus.cc:1261] T f8de1fd0869348e6b0aa131b958ce822 P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:29:08.164444 4757 consensus_queue.cc:1035] T f8de1fd0869348e6b0aa131b958ce822 P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:29:08.165135 4950 ts_tablet_manager.cc:1875] T b300ad3948074c728a25f053512bc37c P 153026dd7f8943cc93fedf6f48450fb4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:29:08.168485 1414 raft_consensus.cc:1261] T f8de1fd0869348e6b0aa131b958ce822 P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:29:08.169157 4757 consensus_queue.cc:1035] T f8de1fd0869348e6b0aa131b958ce822 P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:29:08.172278 4950 ts_tablet_manager.cc:1888] T b300ad3948074c728a25f053512bc37c P 153026dd7f8943cc93fedf6f48450fb4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240502 21:29:08.172487 4950 log.cc:1196] T b300ad3948074c728a25f053512bc37c P 153026dd7f8943cc93fedf6f48450fb4: Deleting WAL directory at /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-3/wal/wals/b300ad3948074c728a25f053512bc37c
I20240502 21:29:08.173383 1256 catalog_manager.cc:4830] TS 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132:44947): tablet b300ad3948074c728a25f053512bc37c (table rebalance_test_table_2 [id=900316782c71465f9538d1bbe31708d3]) successfully deleted
I20240502 21:29:08.175678 1820 raft_consensus.cc:1261] T f8de1fd0869348e6b0aa131b958ce822 P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:29:08.175760 4121 raft_consensus.cc:1261] T f8de1fd0869348e6b0aa131b958ce822 P 1fab5794cf1740288abd82f2cebb9b1a [term 1 LEARNER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:29:08.176314 4949 consensus_queue.cc:1035] T f8de1fd0869348e6b0aa131b958ce822 P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:29:08.177202 2226 raft_consensus.cc:1261] T f8de1fd0869348e6b0aa131b958ce822 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:29:08.177795 4932 consensus_queue.cc:1035] T f8de1fd0869348e6b0aa131b958ce822 P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:29:08.179818 4757 consensus_queue.cc:1035] T f8de1fd0869348e6b0aa131b958ce822 P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:29:08.186151 1953 raft_consensus.cc:1261] T f8de1fd0869348e6b0aa131b958ce822 P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:29:08.186740 4504 consensus_queue.cc:1035] T f8de1fd0869348e6b0aa131b958ce822 P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:29:08.187556 1549 raft_consensus.cc:1261] T f8de1fd0869348e6b0aa131b958ce822 P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:29:08.188231 4769 consensus_queue.cc:1035] T f8de1fd0869348e6b0aa131b958ce822 P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:29:08.191243 4949 raft_consensus.cc:2932] T f8de1fd0869348e6b0aa131b958ce822 P 521bb93035324cecbebed2f6f2a630f8 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } attrs { replace: true } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } }
I20240502 21:29:08.191911 1820 raft_consensus.cc:2932] T f8de1fd0869348e6b0aa131b958ce822 P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } attrs { replace: true } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } }
I20240502 21:29:08.196352 4121 raft_consensus.cc:2932] T f8de1fd0869348e6b0aa131b958ce822 P 1fab5794cf1740288abd82f2cebb9b1a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } attrs { replace: true } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } }
I20240502 21:29:08.196916 1414 raft_consensus.cc:2932] T f8de1fd0869348e6b0aa131b958ce822 P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } attrs { replace: true } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } }
I20240502 21:29:08.198251 1264 catalog_manager.cc:5484] T f8de1fd0869348e6b0aa131b958ce822 P 521bb93035324cecbebed2f6f2a630f8 reported cstate change: config changed from index 5 to 6, 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "521bb93035324cecbebed2f6f2a630f8" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:29:08.203302 2226 raft_consensus.cc:2932] T f8de1fd0869348e6b0aa131b958ce822 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } attrs { replace: true } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } }
I20240502 21:29:08.211232 1685 raft_consensus.cc:2932] T f8de1fd0869348e6b0aa131b958ce822 P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } attrs { replace: true } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } }
I20240502 21:29:08.224285 1953 raft_consensus.cc:2932] T f8de1fd0869348e6b0aa131b958ce822 P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } attrs { replace: true } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } }
I20240502 21:29:08.227902 4835 tablet_bootstrap.cc:492] T 0658ecbf32754656876452b6f021926b P 1fab5794cf1740288abd82f2cebb9b1a: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:29:08.228384 4835 tablet_bootstrap.cc:492] T 0658ecbf32754656876452b6f021926b P 1fab5794cf1740288abd82f2cebb9b1a: Bootstrap complete.
I20240502 21:29:08.228727 4835 ts_tablet_manager.cc:1367] T 0658ecbf32754656876452b6f021926b P 1fab5794cf1740288abd82f2cebb9b1a: Time spent bootstrapping tablet: real 0.108s user 0.052s sys 0.021s
I20240502 21:29:08.229568 4835 raft_consensus.cc:348] T 0658ecbf32754656876452b6f021926b P 1fab5794cf1740288abd82f2cebb9b1a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: NON_VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: true } }
I20240502 21:29:08.229969 4835 raft_consensus.cc:729] T 0658ecbf32754656876452b6f021926b P 1fab5794cf1740288abd82f2cebb9b1a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1fab5794cf1740288abd82f2cebb9b1a, State: Initialized, Role: LEARNER
I20240502 21:29:08.230252 4835 consensus_queue.cc:260] T 0658ecbf32754656876452b6f021926b P 1fab5794cf1740288abd82f2cebb9b1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: NON_VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: true } }
I20240502 21:29:08.231076 4835 ts_tablet_manager.cc:1398] T 0658ecbf32754656876452b6f021926b P 1fab5794cf1740288abd82f2cebb9b1a: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 21:29:08.231891 1975 tablet_copy_service.cc:342] P e632c553ff4341328acf1cbaca678392: Request end of tablet copy session 1fab5794cf1740288abd82f2cebb9b1a-0658ecbf32754656876452b6f021926b received from {username='slave'} at 127.15.255.138:55671
I20240502 21:29:08.232049 1975 tablet_copy_service.cc:433] P e632c553ff4341328acf1cbaca678392: ending tablet copy session 1fab5794cf1740288abd82f2cebb9b1a-0658ecbf32754656876452b6f021926b on tablet 0658ecbf32754656876452b6f021926b with peer 1fab5794cf1740288abd82f2cebb9b1a
I20240502 21:29:08.240370 1547 raft_consensus.cc:2932] T f8de1fd0869348e6b0aa131b958ce822 P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } attrs { replace: true } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } }
I20240502 21:29:08.246358 2087 consensus_queue.cc:237] T f8de1fd0869348e6b0aa131b958ce822 P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } }
I20240502 21:29:08.249341 1414 raft_consensus.cc:1261] T f8de1fd0869348e6b0aa131b958ce822 P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:29:08.249763 1820 raft_consensus.cc:1261] T f8de1fd0869348e6b0aa131b958ce822 P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:29:08.250349 4121 raft_consensus.cc:1261] T f8de1fd0869348e6b0aa131b958ce822 P 1fab5794cf1740288abd82f2cebb9b1a [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:29:08.250340 4757 consensus_queue.cc:1035] T f8de1fd0869348e6b0aa131b958ce822 P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:29:08.251284 4504 consensus_queue.cc:1035] T f8de1fd0869348e6b0aa131b958ce822 P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:29:08.251740 2226 raft_consensus.cc:1261] T f8de1fd0869348e6b0aa131b958ce822 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:29:08.253362 4504 consensus_queue.cc:1035] T f8de1fd0869348e6b0aa131b958ce822 P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:29:08.254037 4757 consensus_queue.cc:1035] T f8de1fd0869348e6b0aa131b958ce822 P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:29:08.251775 1547 raft_consensus.cc:1261] T f8de1fd0869348e6b0aa131b958ce822 P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:29:08.257503 1685 raft_consensus.cc:1261] T f8de1fd0869348e6b0aa131b958ce822 P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:29:08.258234 4757 consensus_queue.cc:1035] T f8de1fd0869348e6b0aa131b958ce822 P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:29:08.258832 4757 consensus_queue.cc:1035] T f8de1fd0869348e6b0aa131b958ce822 P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:29:08.259892 4504 raft_consensus.cc:2932] T f8de1fd0869348e6b0aa131b958ce822 P 521bb93035324cecbebed2f6f2a630f8 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER e632c553ff4341328acf1cbaca678392 (127.15.255.133) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } }
I20240502 21:29:08.263572 1254 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f8de1fd0869348e6b0aa131b958ce822 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:29:08.264539 2226 raft_consensus.cc:2932] T f8de1fd0869348e6b0aa131b958ce822 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER e632c553ff4341328acf1cbaca678392 (127.15.255.133) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } }
I20240502 21:29:08.265836 4121 raft_consensus.cc:2932] T f8de1fd0869348e6b0aa131b958ce822 P 1fab5794cf1740288abd82f2cebb9b1a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER e632c553ff4341328acf1cbaca678392 (127.15.255.133) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } }
I20240502 21:29:08.270237 1414 raft_consensus.cc:2932] T f8de1fd0869348e6b0aa131b958ce822 P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER e632c553ff4341328acf1cbaca678392 (127.15.255.133) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } }
I20240502 21:29:08.270771 1266 catalog_manager.cc:5484] T f8de1fd0869348e6b0aa131b958ce822 P 1fab5794cf1740288abd82f2cebb9b1a reported cstate change: config changed from index 6 to 7, VOTER e632c553ff4341328acf1cbaca678392 (127.15.255.133) evicted. New cstate: current_term: 1 leader_uuid: "521bb93035324cecbebed2f6f2a630f8" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } }
I20240502 21:29:08.272104 1549 raft_consensus.cc:2932] T f8de1fd0869348e6b0aa131b958ce822 P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER e632c553ff4341328acf1cbaca678392 (127.15.255.133) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } }
I20240502 21:29:08.275449 1931 tablet_service.cc:1514] Processing DeleteTablet for tablet f8de1fd0869348e6b0aa131b958ce822 with delete_type TABLET_DATA_TOMBSTONED (TS e632c553ff4341328acf1cbaca678392 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:51198
I20240502 21:29:08.277647 4966 tablet_replica.cc:330] T f8de1fd0869348e6b0aa131b958ce822 P e632c553ff4341328acf1cbaca678392: stopping tablet replica
I20240502 21:29:08.277861 4966 raft_consensus.cc:2229] T f8de1fd0869348e6b0aa131b958ce822 P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:29:08.278139 4966 raft_consensus.cc:2258] T f8de1fd0869348e6b0aa131b958ce822 P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:29:08.279052 4966 ts_tablet_manager.cc:1875] T f8de1fd0869348e6b0aa131b958ce822 P e632c553ff4341328acf1cbaca678392: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:29:08.281592 1820 raft_consensus.cc:2932] T f8de1fd0869348e6b0aa131b958ce822 P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER e632c553ff4341328acf1cbaca678392 (127.15.255.133) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } }
I20240502 21:29:08.281726 1685 raft_consensus.cc:2932] T f8de1fd0869348e6b0aa131b958ce822 P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER e632c553ff4341328acf1cbaca678392 (127.15.255.133) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } }
I20240502 21:29:08.286722 4966 ts_tablet_manager.cc:1888] T f8de1fd0869348e6b0aa131b958ce822 P e632c553ff4341328acf1cbaca678392: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240502 21:29:08.286906 4966 log.cc:1196] T f8de1fd0869348e6b0aa131b958ce822 P e632c553ff4341328acf1cbaca678392: Deleting WAL directory at /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-4/wal/wals/f8de1fd0869348e6b0aa131b958ce822
I20240502 21:29:08.290390 1254 catalog_manager.cc:4830] TS e632c553ff4341328acf1cbaca678392 (127.15.255.133:37439): tablet f8de1fd0869348e6b0aa131b958ce822 (table rebalance_test_table_6 [id=552d53f5242c4a2eb8e89b63e8eb6a7a]) successfully deleted
I20240502 21:29:08.482324 4905 raft_consensus.cc:1050] T 4555e51179344acda81670fc9019d968 P 6ab8df3c796f4b6cb3e29d5045ad3ea0: attempting to promote NON_VOTER 1fab5794cf1740288abd82f2cebb9b1a to VOTER
I20240502 21:29:08.483278 4905 consensus_queue.cc:237] T 4555e51179344acda81670fc9019d968 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } }
I20240502 21:29:08.486354 4121 raft_consensus.cc:1261] T 4555e51179344acda81670fc9019d968 P 1fab5794cf1740288abd82f2cebb9b1a [term 1 LEARNER]: Refusing update from remote peer 6ab8df3c796f4b6cb3e29d5045ad3ea0: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:29:08.486939 1685 raft_consensus.cc:1261] T 4555e51179344acda81670fc9019d968 P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Refusing update from remote peer 6ab8df3c796f4b6cb3e29d5045ad3ea0: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:29:08.487017 1414 raft_consensus.cc:1261] T 4555e51179344acda81670fc9019d968 P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Refusing update from remote peer 6ab8df3c796f4b6cb3e29d5045ad3ea0: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:29:08.487522 1953 raft_consensus.cc:1261] T 4555e51179344acda81670fc9019d968 P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Refusing update from remote peer 6ab8df3c796f4b6cb3e29d5045ad3ea0: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:29:08.487610 4905 consensus_queue.cc:1035] T 4555e51179344acda81670fc9019d968 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:29:08.487907 4716 consensus_queue.cc:1035] T 4555e51179344acda81670fc9019d968 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:29:08.488456 2087 raft_consensus.cc:1261] T 4555e51179344acda81670fc9019d968 P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Refusing update from remote peer 6ab8df3c796f4b6cb3e29d5045ad3ea0: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:29:08.488559 4905 consensus_queue.cc:1035] T 4555e51179344acda81670fc9019d968 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:29:08.488684 1820 raft_consensus.cc:1261] T 4555e51179344acda81670fc9019d968 P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Refusing update from remote peer 6ab8df3c796f4b6cb3e29d5045ad3ea0: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:29:08.488853 4715 consensus_queue.cc:1035] T 4555e51179344acda81670fc9019d968 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:29:08.489187 4716 consensus_queue.cc:1035] T 4555e51179344acda81670fc9019d968 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:29:08.489683 4715 consensus_queue.cc:1035] T 4555e51179344acda81670fc9019d968 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:29:08.491673 1549 raft_consensus.cc:1261] T 4555e51179344acda81670fc9019d968 P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Refusing update from remote peer 6ab8df3c796f4b6cb3e29d5045ad3ea0: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:29:08.492208 4715 consensus_queue.cc:1035] T 4555e51179344acda81670fc9019d968 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:29:08.501462 4716 raft_consensus.cc:2932] T 4555e51179344acda81670fc9019d968 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } }
I20240502 21:29:08.502002 1820 raft_consensus.cc:2932] T 4555e51179344acda81670fc9019d968 P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } }
I20240502 21:29:08.503548 1414 raft_consensus.cc:2932] T 4555e51179344acda81670fc9019d968 P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } }
I20240502 21:29:08.506647 4121 raft_consensus.cc:2932] T 4555e51179344acda81670fc9019d968 P 1fab5794cf1740288abd82f2cebb9b1a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } }
I20240502 21:29:08.509953 1685 raft_consensus.cc:2932] T 4555e51179344acda81670fc9019d968 P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } }
I20240502 21:29:08.510895 2087 raft_consensus.cc:2932] T 4555e51179344acda81670fc9019d968 P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } }
I20240502 21:29:08.511090 1953 raft_consensus.cc:2932] T 4555e51179344acda81670fc9019d968 P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } }
I20240502 21:29:08.513664 1264 catalog_manager.cc:5484] T 4555e51179344acda81670fc9019d968 P 7d1e5c1b047443cbb28d7b0a9744babf reported cstate change: config changed from index 5 to 6, 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } }
I20240502 21:29:08.522190 1547 raft_consensus.cc:2932] T 4555e51179344acda81670fc9019d968 P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } }
I20240502 21:29:08.534255 2226 consensus_queue.cc:237] T 4555e51179344acda81670fc9019d968 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } }
I20240502 21:29:08.537668 2087 raft_consensus.cc:1261] T 4555e51179344acda81670fc9019d968 P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Refusing update from remote peer 6ab8df3c796f4b6cb3e29d5045ad3ea0: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:29:08.538345 1547 raft_consensus.cc:1261] T 4555e51179344acda81670fc9019d968 P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Refusing update from remote peer 6ab8df3c796f4b6cb3e29d5045ad3ea0: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:29:08.538993 4905 consensus_queue.cc:1035] T 4555e51179344acda81670fc9019d968 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:29:08.539186 1953 raft_consensus.cc:1261] T 4555e51179344acda81670fc9019d968 P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Refusing update from remote peer 6ab8df3c796f4b6cb3e29d5045ad3ea0: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:29:08.539346 4715 consensus_queue.cc:1035] T 4555e51179344acda81670fc9019d968 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:29:08.539822 4989 consensus_queue.cc:1035] T 4555e51179344acda81670fc9019d968 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:29:08.541174 4121 raft_consensus.cc:1261] T 4555e51179344acda81670fc9019d968 P 1fab5794cf1740288abd82f2cebb9b1a [term 1 FOLLOWER]: Refusing update from remote peer 6ab8df3c796f4b6cb3e29d5045ad3ea0: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:29:08.541745 1685 raft_consensus.cc:1261] T 4555e51179344acda81670fc9019d968 P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Refusing update from remote peer 6ab8df3c796f4b6cb3e29d5045ad3ea0: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:29:08.541738 4716 consensus_queue.cc:1035] T 4555e51179344acda81670fc9019d968 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:29:08.543412 1414 raft_consensus.cc:1261] T 4555e51179344acda81670fc9019d968 P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Refusing update from remote peer 6ab8df3c796f4b6cb3e29d5045ad3ea0: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:29:08.545593 4716 consensus_queue.cc:1035] T 4555e51179344acda81670fc9019d968 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:29:08.546638 4989 raft_consensus.cc:2932] T 4555e51179344acda81670fc9019d968 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } }
I20240502 21:29:08.549796 4716 consensus_queue.cc:1035] T 4555e51179344acda81670fc9019d968 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:29:08.549634 2087 raft_consensus.cc:2932] T 4555e51179344acda81670fc9019d968 P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } }
I20240502 21:29:08.550174 1953 raft_consensus.cc:2932] T 4555e51179344acda81670fc9019d968 P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } }
I20240502 21:29:08.552678 1255 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 4555e51179344acda81670fc9019d968 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:29:08.552295 1547 raft_consensus.cc:2932] T 4555e51179344acda81670fc9019d968 P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } }
I20240502 21:29:08.553035 1264 catalog_manager.cc:5484] T 4555e51179344acda81670fc9019d968 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 reported cstate change: config changed from index 6 to 7, VOTER 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132) evicted. New cstate: current_term: 1 leader_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:29:08.554198 1685 raft_consensus.cc:2932] T 4555e51179344acda81670fc9019d968 P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } }
I20240502 21:29:08.558259 1798 tablet_service.cc:1514] Processing DeleteTablet for tablet 4555e51179344acda81670fc9019d968 with delete_type TABLET_DATA_TOMBSTONED (TS 153026dd7f8943cc93fedf6f48450fb4 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:60256
I20240502 21:29:08.558671 4950 tablet_replica.cc:330] T 4555e51179344acda81670fc9019d968 P 153026dd7f8943cc93fedf6f48450fb4: stopping tablet replica
I20240502 21:29:08.558883 4950 raft_consensus.cc:2229] T 4555e51179344acda81670fc9019d968 P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:29:08.559262 4950 raft_consensus.cc:2258] T 4555e51179344acda81670fc9019d968 P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:29:08.560153 4950 ts_tablet_manager.cc:1875] T 4555e51179344acda81670fc9019d968 P 153026dd7f8943cc93fedf6f48450fb4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:29:08.560742 4121 raft_consensus.cc:2932] T 4555e51179344acda81670fc9019d968 P 1fab5794cf1740288abd82f2cebb9b1a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } }
I20240502 21:29:08.565606 1414 raft_consensus.cc:2932] T 4555e51179344acda81670fc9019d968 P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } }
I20240502 21:29:08.571022 4950 ts_tablet_manager.cc:1888] T 4555e51179344acda81670fc9019d968 P 153026dd7f8943cc93fedf6f48450fb4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240502 21:29:08.571491 4950 log.cc:1196] T 4555e51179344acda81670fc9019d968 P 153026dd7f8943cc93fedf6f48450fb4: Deleting WAL directory at /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-3/wal/wals/4555e51179344acda81670fc9019d968
I20240502 21:29:08.572322 1256 catalog_manager.cc:4830] TS 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132:44947): tablet 4555e51179344acda81670fc9019d968 (table rebalance_test_table_6 [id=552d53f5242c4a2eb8e89b63e8eb6a7a]) successfully deleted
I20240502 21:29:08.573071 1414 consensus_queue.cc:237] T 815178fbd07b4631b507927683a475c9 P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:29:08.576013 1547 raft_consensus.cc:1261] T 815178fbd07b4631b507927683a475c9 P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:29:08.576264 1953 raft_consensus.cc:1261] T 815178fbd07b4631b507927683a475c9 P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:29:08.576694 4889 consensus_queue.cc:1035] T 815178fbd07b4631b507927683a475c9 P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:08.577167 1685 raft_consensus.cc:1261] T 815178fbd07b4631b507927683a475c9 P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:29:08.577690 4889 consensus_queue.cc:1035] T 815178fbd07b4631b507927683a475c9 P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:08.577955 4592 consensus_queue.cc:1035] T 815178fbd07b4631b507927683a475c9 P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:08.585912 2087 raft_consensus.cc:1261] T 815178fbd07b4631b507927683a475c9 P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:29:08.588052 4592 consensus_queue.cc:1035] T 815178fbd07b4631b507927683a475c9 P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:08.591460 4592 raft_consensus.cc:2932] T 815178fbd07b4631b507927683a475c9 P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } }
I20240502 21:29:08.593147 1547 raft_consensus.cc:2932] T 815178fbd07b4631b507927683a475c9 P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } }
I20240502 21:29:08.595625 2226 raft_consensus.cc:1261] T 815178fbd07b4631b507927683a475c9 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:29:08.595633 1820 raft_consensus.cc:1261] T 815178fbd07b4631b507927683a475c9 P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:29:08.596274 5000 consensus_queue.cc:1035] T 815178fbd07b4631b507927683a475c9 P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:08.596633 4889 consensus_queue.cc:1035] T 815178fbd07b4631b507927683a475c9 P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:08.598901 1685 raft_consensus.cc:2932] T 815178fbd07b4631b507927683a475c9 P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } }
I20240502 21:29:08.600347 1266 catalog_manager.cc:5484] T 815178fbd07b4631b507927683a475c9 P a62a935ff43d494ca35d0c417943133d reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } }
I20240502 21:29:08.604365 4121 raft_consensus.cc:1203] T 0658ecbf32754656876452b6f021926b P 1fab5794cf1740288abd82f2cebb9b1a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240502 21:29:08.598109 1953 raft_consensus.cc:2932] T 815178fbd07b4631b507927683a475c9 P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } }
I20240502 21:29:08.635807 1820 raft_consensus.cc:2932] T 815178fbd07b4631b507927683a475c9 P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } }
I20240502 21:29:08.645224 2226 raft_consensus.cc:2932] T 815178fbd07b4631b507927683a475c9 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } }
I20240502 21:29:08.648029 2089 raft_consensus.cc:2932] T 815178fbd07b4631b507927683a475c9 P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } }
I20240502 21:29:08.656316 1414 consensus_queue.cc:237] T 815178fbd07b4631b507927683a475c9 P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: NON_VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: true } }
I20240502 21:29:08.659827 1820 raft_consensus.cc:1261] T 815178fbd07b4631b507927683a475c9 P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:29:08.660236 1547 raft_consensus.cc:1261] T 815178fbd07b4631b507927683a475c9 P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:29:08.660360 5000 consensus_queue.cc:1035] T 815178fbd07b4631b507927683a475c9 P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:29:08.661070 2226 raft_consensus.cc:1261] T 815178fbd07b4631b507927683a475c9 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:29:08.661746 5000 consensus_queue.cc:1035] T 815178fbd07b4631b507927683a475c9 P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:29:08.662216 5006 consensus_queue.cc:1035] T 815178fbd07b4631b507927683a475c9 P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:29:08.664088 1953 raft_consensus.cc:1261] T 815178fbd07b4631b507927683a475c9 P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:29:08.665244 2089 raft_consensus.cc:1261] T 815178fbd07b4631b507927683a475c9 P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:29:08.665979 5006 consensus_queue.cc:1035] T 815178fbd07b4631b507927683a475c9 P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:29:08.666224 5000 consensus_queue.cc:1035] T 815178fbd07b4631b507927683a475c9 P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:29:08.667690 5006 raft_consensus.cc:2932] T 815178fbd07b4631b507927683a475c9 P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 16a97c1394f346f0b987f97cce7371b6 (127.15.255.137) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: NON_VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: true } } }
I20240502 21:29:08.670572 2227 raft_consensus.cc:2932] T 815178fbd07b4631b507927683a475c9 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 16a97c1394f346f0b987f97cce7371b6 (127.15.255.137) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: NON_VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: true } } }
I20240502 21:29:08.671397 2087 raft_consensus.cc:2932] T 815178fbd07b4631b507927683a475c9 P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 16a97c1394f346f0b987f97cce7371b6 (127.15.255.137) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: NON_VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: true } } }
I20240502 21:29:08.674039 1255 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 815178fbd07b4631b507927683a475c9 with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240502 21:29:08.675853 1953 raft_consensus.cc:2932] T 815178fbd07b4631b507927683a475c9 P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 16a97c1394f346f0b987f97cce7371b6 (127.15.255.137) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: NON_VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: true } } }
W20240502 21:29:08.678735 1347 consensus_peers.cc:485] T 815178fbd07b4631b507927683a475c9 P 7d1e5c1b047443cbb28d7b0a9744babf -> Peer 16a97c1394f346f0b987f97cce7371b6 (127.15.255.137:43717): Couldn't send request to peer 16a97c1394f346f0b987f97cce7371b6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 815178fbd07b4631b507927683a475c9. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:29:08.680550 1265 catalog_manager.cc:5484] T 815178fbd07b4631b507927683a475c9 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 reported cstate change: config changed from index 4 to 5, NON_VOTER 16a97c1394f346f0b987f97cce7371b6 (127.15.255.137) added. New cstate: current_term: 1 leader_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: NON_VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: true } } }
I20240502 21:29:08.690868 1685 raft_consensus.cc:1261] T 815178fbd07b4631b507927683a475c9 P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:29:08.691382 1820 raft_consensus.cc:2932] T 815178fbd07b4631b507927683a475c9 P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 16a97c1394f346f0b987f97cce7371b6 (127.15.255.137) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: NON_VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: true } } }
I20240502 21:29:08.691725 1547 raft_consensus.cc:2932] T 815178fbd07b4631b507927683a475c9 P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 16a97c1394f346f0b987f97cce7371b6 (127.15.255.137) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: NON_VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: true } } }
I20240502 21:29:08.693038 5006 consensus_queue.cc:1035] T 815178fbd07b4631b507927683a475c9 P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:29:08.694530 1685 raft_consensus.cc:2932] T 815178fbd07b4631b507927683a475c9 P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 16a97c1394f346f0b987f97cce7371b6 (127.15.255.137) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: NON_VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: true } } }
I20240502 21:29:08.930859 1414 consensus_queue.cc:237] T 20a7af42e83c4639bba20f22dee69f2a P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:29:08.933439 1549 raft_consensus.cc:1261] T 20a7af42e83c4639bba20f22dee69f2a P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:29:08.933737 1685 raft_consensus.cc:1261] T 20a7af42e83c4639bba20f22dee69f2a P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:29:08.934044 5006 consensus_queue.cc:1035] T 20a7af42e83c4639bba20f22dee69f2a P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:08.934355 5007 consensus_queue.cc:1035] T 20a7af42e83c4639bba20f22dee69f2a P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:08.935446 1953 raft_consensus.cc:1261] T 20a7af42e83c4639bba20f22dee69f2a P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:29:08.935478 2227 raft_consensus.cc:1261] T 20a7af42e83c4639bba20f22dee69f2a P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:29:08.935911 2087 raft_consensus.cc:1261] T 20a7af42e83c4639bba20f22dee69f2a P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:29:08.936488 5006 consensus_queue.cc:1035] T 20a7af42e83c4639bba20f22dee69f2a P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:08.936769 1820 raft_consensus.cc:1261] T 20a7af42e83c4639bba20f22dee69f2a P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:29:08.937047 5006 consensus_queue.cc:1035] T 20a7af42e83c4639bba20f22dee69f2a P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:08.937410 4889 consensus_queue.cc:1035] T 20a7af42e83c4639bba20f22dee69f2a P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:08.939297 5006 consensus_queue.cc:1035] T 20a7af42e83c4639bba20f22dee69f2a P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:08.942209 5000 raft_consensus.cc:2932] T 20a7af42e83c4639bba20f22dee69f2a P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } }
I20240502 21:29:08.945698 1820 raft_consensus.cc:2932] T 20a7af42e83c4639bba20f22dee69f2a P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } }
I20240502 21:29:08.950763 1685 raft_consensus.cc:2932] T 20a7af42e83c4639bba20f22dee69f2a P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } }
I20240502 21:29:08.953192 1265 catalog_manager.cc:5484] T 20a7af42e83c4639bba20f22dee69f2a P 7d1e5c1b047443cbb28d7b0a9744babf reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } health_report { overall_health: HEALTHY } } }
I20240502 21:29:08.954349 1549 raft_consensus.cc:2932] T 20a7af42e83c4639bba20f22dee69f2a P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } }
I20240502 21:29:08.958508 2088 raft_consensus.cc:2932] T 20a7af42e83c4639bba20f22dee69f2a P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } }
I20240502 21:29:08.964347 1953 raft_consensus.cc:2932] T 20a7af42e83c4639bba20f22dee69f2a P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } }
I20240502 21:29:08.965739 2227 raft_consensus.cc:2932] T 20a7af42e83c4639bba20f22dee69f2a P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } }
I20240502 21:29:08.973824 1414 consensus_queue.cc:237] T 20a7af42e83c4639bba20f22dee69f2a P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: NON_VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: true } }
I20240502 21:29:08.976356 1820 raft_consensus.cc:1261] T 20a7af42e83c4639bba20f22dee69f2a P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240502 21:29:08.977371 1347 consensus_peers.cc:485] T 20a7af42e83c4639bba20f22dee69f2a P 7d1e5c1b047443cbb28d7b0a9744babf -> Peer 16a97c1394f346f0b987f97cce7371b6 (127.15.255.137:43717): Couldn't send request to peer 16a97c1394f346f0b987f97cce7371b6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 20a7af42e83c4639bba20f22dee69f2a. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:29:08.977843 5007 consensus_queue.cc:1035] T 20a7af42e83c4639bba20f22dee69f2a P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:29:08.978024 2088 raft_consensus.cc:1261] T 20a7af42e83c4639bba20f22dee69f2a P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:29:08.979199 1953 raft_consensus.cc:1261] T 20a7af42e83c4639bba20f22dee69f2a P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:29:08.979548 1685 raft_consensus.cc:1261] T 20a7af42e83c4639bba20f22dee69f2a P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:29:08.979756 2226 raft_consensus.cc:1261] T 20a7af42e83c4639bba20f22dee69f2a P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:29:08.980295 5007 consensus_queue.cc:1035] T 20a7af42e83c4639bba20f22dee69f2a P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:29:08.980863 5007 consensus_queue.cc:1035] T 20a7af42e83c4639bba20f22dee69f2a P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:29:08.981693 5007 consensus_queue.cc:1035] T 20a7af42e83c4639bba20f22dee69f2a P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:29:08.986295 4889 raft_consensus.cc:2932] T 20a7af42e83c4639bba20f22dee69f2a P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 16a97c1394f346f0b987f97cce7371b6 (127.15.255.137) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: NON_VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: true } } }
I20240502 21:29:08.992297 2088 raft_consensus.cc:2932] T 20a7af42e83c4639bba20f22dee69f2a P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 16a97c1394f346f0b987f97cce7371b6 (127.15.255.137) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: NON_VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: true } } }
I20240502 21:29:08.992902 1685 raft_consensus.cc:2932] T 20a7af42e83c4639bba20f22dee69f2a P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 16a97c1394f346f0b987f97cce7371b6 (127.15.255.137) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: NON_VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: true } } }
I20240502 21:29:08.996479 1255 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 20a7af42e83c4639bba20f22dee69f2a with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240502 21:29:08.997040 1820 raft_consensus.cc:2932] T 20a7af42e83c4639bba20f22dee69f2a P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 16a97c1394f346f0b987f97cce7371b6 (127.15.255.137) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: NON_VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: true } } }
I20240502 21:29:08.998622 1549 raft_consensus.cc:1261] T 20a7af42e83c4639bba20f22dee69f2a P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:29:08.998975 5006 consensus_queue.cc:1035] T 20a7af42e83c4639bba20f22dee69f2a P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:29:09.003222 5006 consensus_queue.cc:1035] T 20a7af42e83c4639bba20f22dee69f2a P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:29:09.004598 1268 catalog_manager.cc:5484] T 20a7af42e83c4639bba20f22dee69f2a P 16b3a337bc5e462f83ba895ad8660deb reported cstate change: config changed from index 4 to 5, NON_VOTER 16a97c1394f346f0b987f97cce7371b6 (127.15.255.137) added. New cstate: current_term: 1 leader_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: NON_VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: true } } }
I20240502 21:29:09.004772 2227 raft_consensus.cc:2932] T 20a7af42e83c4639bba20f22dee69f2a P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 16a97c1394f346f0b987f97cce7371b6 (127.15.255.137) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: NON_VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: true } } }
I20240502 21:29:09.004742 1953 raft_consensus.cc:2932] T 20a7af42e83c4639bba20f22dee69f2a P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 16a97c1394f346f0b987f97cce7371b6 (127.15.255.137) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: NON_VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: true } } }
I20240502 21:29:09.012081 1549 raft_consensus.cc:2932] T 20a7af42e83c4639bba20f22dee69f2a P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 16a97c1394f346f0b987f97cce7371b6 (127.15.255.137) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: NON_VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: true } } }
I20240502 21:29:09.184527 5058 ts_tablet_manager.cc:902] T 815178fbd07b4631b507927683a475c9 P 16a97c1394f346f0b987f97cce7371b6: Initiating tablet copy from peer 7d1e5c1b047443cbb28d7b0a9744babf (127.15.255.129:37897)
I20240502 21:29:09.185554 5058 tablet_copy_client.cc:308] T 815178fbd07b4631b507927683a475c9 P 16a97c1394f346f0b987f97cce7371b6: tablet copy: Beginning tablet copy session from remote peer at address 127.15.255.129:37897
I20240502 21:29:09.191143 1435 tablet_copy_service.cc:140] P 7d1e5c1b047443cbb28d7b0a9744babf: Received BeginTabletCopySession request for tablet 815178fbd07b4631b507927683a475c9 from peer 16a97c1394f346f0b987f97cce7371b6 ({username='slave'} at 127.15.255.137:43115)
I20240502 21:29:09.191282 1435 tablet_copy_service.cc:161] P 7d1e5c1b047443cbb28d7b0a9744babf: Beginning new tablet copy session on tablet 815178fbd07b4631b507927683a475c9 from peer 16a97c1394f346f0b987f97cce7371b6 at {username='slave'} at 127.15.255.137:43115: session id = 16a97c1394f346f0b987f97cce7371b6-815178fbd07b4631b507927683a475c9
I20240502 21:29:09.193400 1435 tablet_copy_source_session.cc:205] T 815178fbd07b4631b507927683a475c9 P 7d1e5c1b047443cbb28d7b0a9744babf: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:29:09.194579 5058 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 815178fbd07b4631b507927683a475c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:29:09.198889 5058 tablet_copy_client.cc:783] T 815178fbd07b4631b507927683a475c9 P 16a97c1394f346f0b987f97cce7371b6: tablet copy: Starting download of 0 data blocks...
I20240502 21:29:09.199120 5058 tablet_copy_client.cc:647] T 815178fbd07b4631b507927683a475c9 P 16a97c1394f346f0b987f97cce7371b6: tablet copy: Starting download of 1 WAL segments...
I20240502 21:29:09.200656 5058 tablet_copy_client.cc:523] T 815178fbd07b4631b507927683a475c9 P 16a97c1394f346f0b987f97cce7371b6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:29:09.202901 5058 tablet_bootstrap.cc:492] T 815178fbd07b4631b507927683a475c9 P 16a97c1394f346f0b987f97cce7371b6: Bootstrap starting.
I20240502 21:29:09.226397 4849 raft_consensus.cc:1050] T 0658ecbf32754656876452b6f021926b P e632c553ff4341328acf1cbaca678392: attempting to promote NON_VOTER 1fab5794cf1740288abd82f2cebb9b1a to VOTER
I20240502 21:29:09.227792 4849 consensus_queue.cc:237] T 0658ecbf32754656876452b6f021926b P e632c553ff4341328acf1cbaca678392 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } }
I20240502 21:29:09.245489 1414 raft_consensus.cc:1261] T 0658ecbf32754656876452b6f021926b P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Refusing update from remote peer e632c553ff4341328acf1cbaca678392: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:29:09.246698 4693 consensus_queue.cc:1035] T 0658ecbf32754656876452b6f021926b P e632c553ff4341328acf1cbaca678392 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:29:09.248239 5058 tablet_bootstrap.cc:492] T 815178fbd07b4631b507927683a475c9 P 16a97c1394f346f0b987f97cce7371b6: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:29:09.248739 5058 tablet_bootstrap.cc:492] T 815178fbd07b4631b507927683a475c9 P 16a97c1394f346f0b987f97cce7371b6: Bootstrap complete.
I20240502 21:29:09.249070 5058 ts_tablet_manager.cc:1367] T 815178fbd07b4631b507927683a475c9 P 16a97c1394f346f0b987f97cce7371b6: Time spent bootstrapping tablet: real 0.046s user 0.032s sys 0.012s
I20240502 21:29:09.249912 5058 raft_consensus.cc:348] T 815178fbd07b4631b507927683a475c9 P 16a97c1394f346f0b987f97cce7371b6 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: NON_VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: true } }
I20240502 21:29:09.250229 5058 raft_consensus.cc:729] T 815178fbd07b4631b507927683a475c9 P 16a97c1394f346f0b987f97cce7371b6 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 16a97c1394f346f0b987f97cce7371b6, State: Initialized, Role: LEARNER
I20240502 21:29:09.250471 5058 consensus_queue.cc:260] T 815178fbd07b4631b507927683a475c9 P 16a97c1394f346f0b987f97cce7371b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: NON_VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: true } }
I20240502 21:29:09.255553 5058 ts_tablet_manager.cc:1398] T 815178fbd07b4631b507927683a475c9 P 16a97c1394f346f0b987f97cce7371b6: Time spent starting tablet: real 0.006s user 0.002s sys 0.004s
I20240502 21:29:09.256182 1435 tablet_copy_service.cc:342] P 7d1e5c1b047443cbb28d7b0a9744babf: Request end of tablet copy session 16a97c1394f346f0b987f97cce7371b6-815178fbd07b4631b507927683a475c9 received from {username='slave'} at 127.15.255.137:43115
I20240502 21:29:09.256340 1435 tablet_copy_service.cc:433] P 7d1e5c1b047443cbb28d7b0a9744babf: ending tablet copy session 16a97c1394f346f0b987f97cce7371b6-815178fbd07b4631b507927683a475c9 on tablet 815178fbd07b4631b507927683a475c9 with peer 16a97c1394f346f0b987f97cce7371b6
I20240502 21:29:09.260169 1685 raft_consensus.cc:1261] T 0658ecbf32754656876452b6f021926b P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Refusing update from remote peer e632c553ff4341328acf1cbaca678392: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:29:09.261636 4693 consensus_queue.cc:1035] T 0658ecbf32754656876452b6f021926b P e632c553ff4341328acf1cbaca678392 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:29:09.281947 1549 raft_consensus.cc:1261] T 0658ecbf32754656876452b6f021926b P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Refusing update from remote peer e632c553ff4341328acf1cbaca678392: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:29:09.282696 4693 consensus_queue.cc:1035] T 0658ecbf32754656876452b6f021926b P e632c553ff4341328acf1cbaca678392 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:29:09.284649 4121 raft_consensus.cc:1261] T 0658ecbf32754656876452b6f021926b P 1fab5794cf1740288abd82f2cebb9b1a [term 1 LEARNER]: Refusing update from remote peer e632c553ff4341328acf1cbaca678392: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:29:09.284884 2227 raft_consensus.cc:1261] T 0658ecbf32754656876452b6f021926b P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Refusing update from remote peer e632c553ff4341328acf1cbaca678392: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:29:09.285599 4693 consensus_queue.cc:1035] T 0658ecbf32754656876452b6f021926b P e632c553ff4341328acf1cbaca678392 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:29:09.285975 5066 consensus_queue.cc:1035] T 0658ecbf32754656876452b6f021926b P e632c553ff4341328acf1cbaca678392 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:29:09.287526 2088 raft_consensus.cc:1261] T 0658ecbf32754656876452b6f021926b P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Refusing update from remote peer e632c553ff4341328acf1cbaca678392: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:29:09.289139 4693 consensus_queue.cc:1035] T 0658ecbf32754656876452b6f021926b P e632c553ff4341328acf1cbaca678392 [LEADER]: Connected to new peer: Peer: permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:29:09.293808 4693 raft_consensus.cc:2932] T 0658ecbf32754656876452b6f021926b P e632c553ff4341328acf1cbaca678392 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } }
I20240502 21:29:09.294265 1550 raft_consensus.cc:2932] T 0658ecbf32754656876452b6f021926b P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } }
I20240502 21:29:09.302338 1264 catalog_manager.cc:5484] T 0658ecbf32754656876452b6f021926b P a62a935ff43d494ca35d0c417943133d reported cstate change: config changed from index 5 to 6, 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e632c553ff4341328acf1cbaca678392" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } }
I20240502 21:29:09.309144 4121 raft_consensus.cc:2932] T 0658ecbf32754656876452b6f021926b P 1fab5794cf1740288abd82f2cebb9b1a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } }
I20240502 21:29:09.310359 1820 raft_consensus.cc:1261] T 0658ecbf32754656876452b6f021926b P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Refusing update from remote peer e632c553ff4341328acf1cbaca678392: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:29:09.310905 2227 raft_consensus.cc:2932] T 0658ecbf32754656876452b6f021926b P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } }
I20240502 21:29:09.313261 5066 consensus_queue.cc:1035] T 0658ecbf32754656876452b6f021926b P e632c553ff4341328acf1cbaca678392 [LEADER]: Connected to new peer: Peer: permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:29:09.315807 1685 raft_consensus.cc:2932] T 0658ecbf32754656876452b6f021926b P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } }
I20240502 21:29:09.334838 2088 raft_consensus.cc:2932] T 0658ecbf32754656876452b6f021926b P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } }
I20240502 21:29:09.342262 1820 raft_consensus.cc:2932] T 0658ecbf32754656876452b6f021926b P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } }
I20240502 21:29:09.343132 1414 raft_consensus.cc:2932] T 0658ecbf32754656876452b6f021926b P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } }
I20240502 21:29:09.352083 1953 consensus_queue.cc:237] T 0658ecbf32754656876452b6f021926b P e632c553ff4341328acf1cbaca678392 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } }
I20240502 21:29:09.355044 2088 raft_consensus.cc:1261] T 0658ecbf32754656876452b6f021926b P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Refusing update from remote peer e632c553ff4341328acf1cbaca678392: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:29:09.355070 4121 raft_consensus.cc:1261] T 0658ecbf32754656876452b6f021926b P 1fab5794cf1740288abd82f2cebb9b1a [term 1 FOLLOWER]: Refusing update from remote peer e632c553ff4341328acf1cbaca678392: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:29:09.355583 5081 consensus_queue.cc:1035] T 0658ecbf32754656876452b6f021926b P e632c553ff4341328acf1cbaca678392 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:29:09.356949 1685 raft_consensus.cc:1261] T 0658ecbf32754656876452b6f021926b P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Refusing update from remote peer e632c553ff4341328acf1cbaca678392: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:29:09.357591 4693 consensus_queue.cc:1035] T 0658ecbf32754656876452b6f021926b P e632c553ff4341328acf1cbaca678392 [LEADER]: Connected to new peer: Peer: permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:29:09.358801 1550 raft_consensus.cc:1261] T 0658ecbf32754656876452b6f021926b P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Refusing update from remote peer e632c553ff4341328acf1cbaca678392: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:29:09.359387 4693 consensus_queue.cc:1035] T 0658ecbf32754656876452b6f021926b P e632c553ff4341328acf1cbaca678392 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:29:09.359565 1414 raft_consensus.cc:1261] T 0658ecbf32754656876452b6f021926b P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Refusing update from remote peer e632c553ff4341328acf1cbaca678392: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:29:09.360018 4693 consensus_queue.cc:1035] T 0658ecbf32754656876452b6f021926b P e632c553ff4341328acf1cbaca678392 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:29:09.361106 5080 consensus_queue.cc:1035] T 0658ecbf32754656876452b6f021926b P e632c553ff4341328acf1cbaca678392 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:29:09.362079 4849 raft_consensus.cc:2932] T 0658ecbf32754656876452b6f021926b P e632c553ff4341328acf1cbaca678392 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } }
I20240502 21:29:09.362584 1550 raft_consensus.cc:2932] T 0658ecbf32754656876452b6f021926b P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } }
I20240502 21:29:09.364785 1686 raft_consensus.cc:2932] T 0658ecbf32754656876452b6f021926b P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } }
I20240502 21:29:09.368577 2088 raft_consensus.cc:2932] T 0658ecbf32754656876452b6f021926b P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } }
I20240502 21:29:09.370249 1254 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 0658ecbf32754656876452b6f021926b with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:29:09.370515 4121 raft_consensus.cc:2932] T 0658ecbf32754656876452b6f021926b P 1fab5794cf1740288abd82f2cebb9b1a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } }
I20240502 21:29:09.371693 1414 raft_consensus.cc:2932] T 0658ecbf32754656876452b6f021926b P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } }
I20240502 21:29:09.373241 1264 catalog_manager.cc:5484] T 0658ecbf32754656876452b6f021926b P a62a935ff43d494ca35d0c417943133d reported cstate change: config changed from index 6 to 7, VOTER 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132) evicted. New cstate: current_term: 1 leader_uuid: "e632c553ff4341328acf1cbaca678392" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } }
I20240502 21:29:09.377244 1798 tablet_service.cc:1514] Processing DeleteTablet for tablet 0658ecbf32754656876452b6f021926b with delete_type TABLET_DATA_TOMBSTONED (TS 153026dd7f8943cc93fedf6f48450fb4 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:60256
W20240502 21:29:09.378229 1254 catalog_manager.cc:5007] ChangeConfig:REMOVE_PEER RPC for tablet 0658ecbf32754656876452b6f021926b with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER failed with leader e632c553ff4341328acf1cbaca678392 (127.15.255.133:37439) due to CAS failure; no further retry: Illegal state: Request specified cas_config_opid_index of 6 but the committed config has opid_index of 7
I20240502 21:29:09.392395 2227 raft_consensus.cc:1261] T 0658ecbf32754656876452b6f021926b P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Refusing update from remote peer e632c553ff4341328acf1cbaca678392: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:29:09.393057 5076 consensus_queue.cc:1035] T 0658ecbf32754656876452b6f021926b P e632c553ff4341328acf1cbaca678392 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:29:09.394661 2227 raft_consensus.cc:2932] T 0658ecbf32754656876452b6f021926b P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } }
I20240502 21:29:09.399354 5086 tablet_replica.cc:330] T 0658ecbf32754656876452b6f021926b P 153026dd7f8943cc93fedf6f48450fb4: stopping tablet replica
I20240502 21:29:09.401288 5086 raft_consensus.cc:2229] T 0658ecbf32754656876452b6f021926b P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:29:09.401649 5086 raft_consensus.cc:2258] T 0658ecbf32754656876452b6f021926b P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:29:09.402730 5086 ts_tablet_manager.cc:1875] T 0658ecbf32754656876452b6f021926b P 153026dd7f8943cc93fedf6f48450fb4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:29:09.409283 5086 ts_tablet_manager.cc:1888] T 0658ecbf32754656876452b6f021926b P 153026dd7f8943cc93fedf6f48450fb4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240502 21:29:09.409476 5086 log.cc:1196] T 0658ecbf32754656876452b6f021926b P 153026dd7f8943cc93fedf6f48450fb4: Deleting WAL directory at /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-3/wal/wals/0658ecbf32754656876452b6f021926b
I20240502 21:29:09.410382 1256 catalog_manager.cc:4830] TS 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132:44947): tablet 0658ecbf32754656876452b6f021926b (table rebalance_test_table_2 [id=900316782c71465f9538d1bbe31708d3]) successfully deleted
I20240502 21:29:09.585809 5058 ts_tablet_manager.cc:902] T 20a7af42e83c4639bba20f22dee69f2a P 16a97c1394f346f0b987f97cce7371b6: Initiating tablet copy from peer 7d1e5c1b047443cbb28d7b0a9744babf (127.15.255.129:37897)
I20240502 21:29:09.587888 5058 tablet_copy_client.cc:308] T 20a7af42e83c4639bba20f22dee69f2a P 16a97c1394f346f0b987f97cce7371b6: tablet copy: Beginning tablet copy session from remote peer at address 127.15.255.129:37897
I20240502 21:29:09.588654 1435 tablet_copy_service.cc:140] P 7d1e5c1b047443cbb28d7b0a9744babf: Received BeginTabletCopySession request for tablet 20a7af42e83c4639bba20f22dee69f2a from peer 16a97c1394f346f0b987f97cce7371b6 ({username='slave'} at 127.15.255.137:43115)
I20240502 21:29:09.588845 1435 tablet_copy_service.cc:161] P 7d1e5c1b047443cbb28d7b0a9744babf: Beginning new tablet copy session on tablet 20a7af42e83c4639bba20f22dee69f2a from peer 16a97c1394f346f0b987f97cce7371b6 at {username='slave'} at 127.15.255.137:43115: session id = 16a97c1394f346f0b987f97cce7371b6-20a7af42e83c4639bba20f22dee69f2a
I20240502 21:29:09.591332 1435 tablet_copy_source_session.cc:205] T 20a7af42e83c4639bba20f22dee69f2a P 7d1e5c1b047443cbb28d7b0a9744babf: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:29:09.592641 5058 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20a7af42e83c4639bba20f22dee69f2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:29:09.597220 5058 tablet_copy_client.cc:783] T 20a7af42e83c4639bba20f22dee69f2a P 16a97c1394f346f0b987f97cce7371b6: tablet copy: Starting download of 0 data blocks...
I20240502 21:29:09.597440 5058 tablet_copy_client.cc:647] T 20a7af42e83c4639bba20f22dee69f2a P 16a97c1394f346f0b987f97cce7371b6: tablet copy: Starting download of 1 WAL segments...
I20240502 21:29:09.598718 5058 tablet_copy_client.cc:523] T 20a7af42e83c4639bba20f22dee69f2a P 16a97c1394f346f0b987f97cce7371b6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:29:09.600937 5058 tablet_bootstrap.cc:492] T 20a7af42e83c4639bba20f22dee69f2a P 16a97c1394f346f0b987f97cce7371b6: Bootstrap starting.
I20240502 21:29:09.635192 5058 tablet_bootstrap.cc:492] T 20a7af42e83c4639bba20f22dee69f2a P 16a97c1394f346f0b987f97cce7371b6: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:29:09.635692 5058 tablet_bootstrap.cc:492] T 20a7af42e83c4639bba20f22dee69f2a P 16a97c1394f346f0b987f97cce7371b6: Bootstrap complete.
I20240502 21:29:09.635957 5058 ts_tablet_manager.cc:1367] T 20a7af42e83c4639bba20f22dee69f2a P 16a97c1394f346f0b987f97cce7371b6: Time spent bootstrapping tablet: real 0.035s user 0.023s sys 0.009s
I20240502 21:29:09.636900 5058 raft_consensus.cc:348] T 20a7af42e83c4639bba20f22dee69f2a P 16a97c1394f346f0b987f97cce7371b6 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: NON_VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: true } }
I20240502 21:29:09.637235 5058 raft_consensus.cc:729] T 20a7af42e83c4639bba20f22dee69f2a P 16a97c1394f346f0b987f97cce7371b6 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 16a97c1394f346f0b987f97cce7371b6, State: Initialized, Role: LEARNER
I20240502 21:29:09.637468 5058 consensus_queue.cc:260] T 20a7af42e83c4639bba20f22dee69f2a P 16a97c1394f346f0b987f97cce7371b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: NON_VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: true } }
I20240502 21:29:09.638129 5058 ts_tablet_manager.cc:1398] T 20a7af42e83c4639bba20f22dee69f2a P 16a97c1394f346f0b987f97cce7371b6: Time spent starting tablet: real 0.002s user 0.005s sys 0.000s
I20240502 21:29:09.639384 1435 tablet_copy_service.cc:342] P 7d1e5c1b047443cbb28d7b0a9744babf: Request end of tablet copy session 16a97c1394f346f0b987f97cce7371b6-20a7af42e83c4639bba20f22dee69f2a received from {username='slave'} at 127.15.255.137:43115
I20240502 21:29:09.639493 1435 tablet_copy_service.cc:433] P 7d1e5c1b047443cbb28d7b0a9744babf: ending tablet copy session 16a97c1394f346f0b987f97cce7371b6-20a7af42e83c4639bba20f22dee69f2a on tablet 20a7af42e83c4639bba20f22dee69f2a with peer 16a97c1394f346f0b987f97cce7371b6
I20240502 21:29:09.652249 2088 consensus_queue.cc:237] T e44bf56d64b24e328803ebd0a3a54ceb P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:29:09.655133 1550 raft_consensus.cc:1261] T e44bf56d64b24e328803ebd0a3a54ceb P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:29:09.655139 1686 raft_consensus.cc:1261] T e44bf56d64b24e328803ebd0a3a54ceb P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:29:09.655331 1953 raft_consensus.cc:1261] T e44bf56d64b24e328803ebd0a3a54ceb P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:29:09.655714 5087 consensus_queue.cc:1035] T e44bf56d64b24e328803ebd0a3a54ceb P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:09.656141 4949 consensus_queue.cc:1035] T e44bf56d64b24e328803ebd0a3a54ceb P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:09.657068 1414 raft_consensus.cc:1261] T e44bf56d64b24e328803ebd0a3a54ceb P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:29:09.657146 4949 consensus_queue.cc:1035] T e44bf56d64b24e328803ebd0a3a54ceb P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:09.659322 5087 consensus_queue.cc:1035] T e44bf56d64b24e328803ebd0a3a54ceb P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:09.661712 5087 raft_consensus.cc:2932] T e44bf56d64b24e328803ebd0a3a54ceb P 521bb93035324cecbebed2f6f2a630f8 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } }
I20240502 21:29:09.662236 1953 raft_consensus.cc:2932] T e44bf56d64b24e328803ebd0a3a54ceb P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } }
I20240502 21:29:09.667467 1820 raft_consensus.cc:1261] T e44bf56d64b24e328803ebd0a3a54ceb P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:29:09.667433 1686 raft_consensus.cc:2932] T e44bf56d64b24e328803ebd0a3a54ceb P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } }
I20240502 21:29:09.667812 1264 catalog_manager.cc:5484] T e44bf56d64b24e328803ebd0a3a54ceb P e632c553ff4341328acf1cbaca678392 reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "521bb93035324cecbebed2f6f2a630f8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } }
I20240502 21:29:09.668372 4949 consensus_queue.cc:1035] T e44bf56d64b24e328803ebd0a3a54ceb P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:09.672489 1414 raft_consensus.cc:2932] T e44bf56d64b24e328803ebd0a3a54ceb P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } }
I20240502 21:29:09.676179 2227 raft_consensus.cc:1261] T e44bf56d64b24e328803ebd0a3a54ceb P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:29:09.675755 1550 raft_consensus.cc:2932] T e44bf56d64b24e328803ebd0a3a54ceb P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } }
I20240502 21:29:09.681529 4949 consensus_queue.cc:1035] T e44bf56d64b24e328803ebd0a3a54ceb P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:09.684535 2227 raft_consensus.cc:2932] T e44bf56d64b24e328803ebd0a3a54ceb P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } }
I20240502 21:29:09.696372 1818 raft_consensus.cc:2932] T e44bf56d64b24e328803ebd0a3a54ceb P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } }
I20240502 21:29:09.708369 2088 consensus_queue.cc:237] T e44bf56d64b24e328803ebd0a3a54ceb P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: NON_VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: true } }
I20240502 21:29:09.712080 1953 raft_consensus.cc:1261] T e44bf56d64b24e328803ebd0a3a54ceb P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:29:09.712757 5109 consensus_queue.cc:1035] T e44bf56d64b24e328803ebd0a3a54ceb P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:29:09.713367 1414 raft_consensus.cc:1261] T e44bf56d64b24e328803ebd0a3a54ceb P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240502 21:29:09.713508 2025 consensus_peers.cc:485] T e44bf56d64b24e328803ebd0a3a54ceb P 521bb93035324cecbebed2f6f2a630f8 -> Peer 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138:33375): Couldn't send request to peer 1fab5794cf1740288abd82f2cebb9b1a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e44bf56d64b24e328803ebd0a3a54ceb. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:29:09.713519 2227 raft_consensus.cc:1261] T e44bf56d64b24e328803ebd0a3a54ceb P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:29:09.714052 1550 raft_consensus.cc:1261] T e44bf56d64b24e328803ebd0a3a54ceb P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:29:09.714274 1818 raft_consensus.cc:1261] T e44bf56d64b24e328803ebd0a3a54ceb P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:29:09.714964 4504 consensus_queue.cc:1035] T e44bf56d64b24e328803ebd0a3a54ceb P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:29:09.715333 4949 consensus_queue.cc:1035] T e44bf56d64b24e328803ebd0a3a54ceb P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:29:09.715585 1686 raft_consensus.cc:1261] T e44bf56d64b24e328803ebd0a3a54ceb P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:29:09.715615 5123 consensus_queue.cc:1035] T e44bf56d64b24e328803ebd0a3a54ceb P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:29:09.715911 5113 consensus_queue.cc:1035] T e44bf56d64b24e328803ebd0a3a54ceb P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:29:09.716823 5087 consensus_queue.cc:1035] T e44bf56d64b24e328803ebd0a3a54ceb P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:29:09.719688 5109 raft_consensus.cc:2932] T e44bf56d64b24e328803ebd0a3a54ceb P 521bb93035324cecbebed2f6f2a630f8 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: NON_VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: true } } }
I20240502 21:29:09.720314 1686 raft_consensus.cc:2932] T e44bf56d64b24e328803ebd0a3a54ceb P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: NON_VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: true } } }
I20240502 21:29:09.723717 1254 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet e44bf56d64b24e328803ebd0a3a54ceb with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240502 21:29:09.723399 1414 raft_consensus.cc:2932] T e44bf56d64b24e328803ebd0a3a54ceb P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: NON_VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: true } } }
I20240502 21:29:09.724864 1953 raft_consensus.cc:2932] T e44bf56d64b24e328803ebd0a3a54ceb P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: NON_VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: true } } }
I20240502 21:29:09.725271 2227 raft_consensus.cc:2932] T e44bf56d64b24e328803ebd0a3a54ceb P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: NON_VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: true } } }
I20240502 21:29:09.726749 1818 raft_consensus.cc:2932] T e44bf56d64b24e328803ebd0a3a54ceb P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: NON_VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: true } } }
I20240502 21:29:09.730499 1266 catalog_manager.cc:5484] T e44bf56d64b24e328803ebd0a3a54ceb P e632c553ff4341328acf1cbaca678392 reported cstate change: config changed from index 4 to 5, NON_VOTER 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138) added. New cstate: current_term: 1 leader_uuid: "521bb93035324cecbebed2f6f2a630f8" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: NON_VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: true } } }
I20240502 21:29:09.731396 1550 raft_consensus.cc:2932] T e44bf56d64b24e328803ebd0a3a54ceb P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: NON_VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: true } } }
I20240502 21:29:09.765801 3987 raft_consensus.cc:1203] T 815178fbd07b4631b507927683a475c9 P 16a97c1394f346f0b987f97cce7371b6 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240502 21:29:10.119565 3987 raft_consensus.cc:1203] T 20a7af42e83c4639bba20f22dee69f2a P 16a97c1394f346f0b987f97cce7371b6 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240502 21:29:10.333513 5168 ts_tablet_manager.cc:902] T e44bf56d64b24e328803ebd0a3a54ceb P 1fab5794cf1740288abd82f2cebb9b1a: Initiating tablet copy from peer 521bb93035324cecbebed2f6f2a630f8 (127.15.255.134:40821)
I20240502 21:29:10.345218 5168 tablet_copy_client.cc:308] T e44bf56d64b24e328803ebd0a3a54ceb P 1fab5794cf1740288abd82f2cebb9b1a: tablet copy: Beginning tablet copy session from remote peer at address 127.15.255.134:40821
I20240502 21:29:10.346164 2113 tablet_copy_service.cc:140] P 521bb93035324cecbebed2f6f2a630f8: Received BeginTabletCopySession request for tablet e44bf56d64b24e328803ebd0a3a54ceb from peer 1fab5794cf1740288abd82f2cebb9b1a ({username='slave'} at 127.15.255.138:37811)
I20240502 21:29:10.346302 2113 tablet_copy_service.cc:161] P 521bb93035324cecbebed2f6f2a630f8: Beginning new tablet copy session on tablet e44bf56d64b24e328803ebd0a3a54ceb from peer 1fab5794cf1740288abd82f2cebb9b1a at {username='slave'} at 127.15.255.138:37811: session id = 1fab5794cf1740288abd82f2cebb9b1a-e44bf56d64b24e328803ebd0a3a54ceb
I20240502 21:29:10.348537 2113 tablet_copy_source_session.cc:205] T e44bf56d64b24e328803ebd0a3a54ceb P 521bb93035324cecbebed2f6f2a630f8: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:29:10.349777 5168 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e44bf56d64b24e328803ebd0a3a54ceb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:29:10.354086 5168 tablet_copy_client.cc:783] T e44bf56d64b24e328803ebd0a3a54ceb P 1fab5794cf1740288abd82f2cebb9b1a: tablet copy: Starting download of 0 data blocks...
I20240502 21:29:10.354308 5168 tablet_copy_client.cc:647] T e44bf56d64b24e328803ebd0a3a54ceb P 1fab5794cf1740288abd82f2cebb9b1a: tablet copy: Starting download of 1 WAL segments...
I20240502 21:29:10.355757 5168 tablet_copy_client.cc:523] T e44bf56d64b24e328803ebd0a3a54ceb P 1fab5794cf1740288abd82f2cebb9b1a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:29:10.358954 5168 tablet_bootstrap.cc:492] T e44bf56d64b24e328803ebd0a3a54ceb P 1fab5794cf1740288abd82f2cebb9b1a: Bootstrap starting.
I20240502 21:29:10.371874 5000 raft_consensus.cc:1050] T 815178fbd07b4631b507927683a475c9 P 7d1e5c1b047443cbb28d7b0a9744babf: attempting to promote NON_VOTER 16a97c1394f346f0b987f97cce7371b6 to VOTER
I20240502 21:29:10.373078 5000 consensus_queue.cc:237] T 815178fbd07b4631b507927683a475c9 P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } }
I20240502 21:29:10.376947 3987 raft_consensus.cc:1261] T 815178fbd07b4631b507927683a475c9 P 16a97c1394f346f0b987f97cce7371b6 [term 1 LEARNER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:29:10.376936 1686 raft_consensus.cc:1261] T 815178fbd07b4631b507927683a475c9 P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:29:10.377689 4889 consensus_queue.cc:1035] T 815178fbd07b4631b507927683a475c9 P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:29:10.378294 1550 raft_consensus.cc:1261] T 815178fbd07b4631b507927683a475c9 P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:29:10.378798 2226 raft_consensus.cc:1261] T 815178fbd07b4631b507927683a475c9 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:29:10.378870 4889 consensus_queue.cc:1035] T 815178fbd07b4631b507927683a475c9 P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:29:10.379343 5006 consensus_queue.cc:1035] T 815178fbd07b4631b507927683a475c9 P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:29:10.381117 1818 raft_consensus.cc:1261] T 815178fbd07b4631b507927683a475c9 P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:29:10.381127 1953 raft_consensus.cc:1261] T 815178fbd07b4631b507927683a475c9 P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:29:10.381688 5006 consensus_queue.cc:1035] T 815178fbd07b4631b507927683a475c9 P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:29:10.382236 5006 consensus_queue.cc:1035] T 815178fbd07b4631b507927683a475c9 P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:29:10.382298 2088 raft_consensus.cc:1261] T 815178fbd07b4631b507927683a475c9 P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:29:10.382786 4889 consensus_queue.cc:1035] T 815178fbd07b4631b507927683a475c9 P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:29:10.384479 5006 consensus_queue.cc:1035] T 815178fbd07b4631b507927683a475c9 P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:29:10.392086 5006 raft_consensus.cc:2932] T 815178fbd07b4631b507927683a475c9 P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 16a97c1394f346f0b987f97cce7371b6 (127.15.255.137) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } }
I20240502 21:29:10.399051 3987 raft_consensus.cc:2932] T 815178fbd07b4631b507927683a475c9 P 16a97c1394f346f0b987f97cce7371b6 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 16a97c1394f346f0b987f97cce7371b6 (127.15.255.137) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } }
I20240502 21:29:10.400383 1818 raft_consensus.cc:2932] T 815178fbd07b4631b507927683a475c9 P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 16a97c1394f346f0b987f97cce7371b6 (127.15.255.137) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } }
I20240502 21:29:10.403081 5168 tablet_bootstrap.cc:492] T e44bf56d64b24e328803ebd0a3a54ceb P 1fab5794cf1740288abd82f2cebb9b1a: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:29:10.403802 5168 tablet_bootstrap.cc:492] T e44bf56d64b24e328803ebd0a3a54ceb P 1fab5794cf1740288abd82f2cebb9b1a: Bootstrap complete.
I20240502 21:29:10.404088 5168 ts_tablet_manager.cc:1367] T e44bf56d64b24e328803ebd0a3a54ceb P 1fab5794cf1740288abd82f2cebb9b1a: Time spent bootstrapping tablet: real 0.045s user 0.035s sys 0.008s
I20240502 21:29:10.404886 5168 raft_consensus.cc:348] T e44bf56d64b24e328803ebd0a3a54ceb P 1fab5794cf1740288abd82f2cebb9b1a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: NON_VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: true } }
I20240502 21:29:10.405243 5168 raft_consensus.cc:729] T e44bf56d64b24e328803ebd0a3a54ceb P 1fab5794cf1740288abd82f2cebb9b1a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1fab5794cf1740288abd82f2cebb9b1a, State: Initialized, Role: LEARNER
I20240502 21:29:10.405450 5168 consensus_queue.cc:260] T e44bf56d64b24e328803ebd0a3a54ceb P 1fab5794cf1740288abd82f2cebb9b1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: NON_VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: true } }
I20240502 21:29:10.405928 1264 catalog_manager.cc:5484] T 815178fbd07b4631b507927683a475c9 P 7d1e5c1b047443cbb28d7b0a9744babf reported cstate change: config changed from index 5 to 6, 16a97c1394f346f0b987f97cce7371b6 (127.15.255.137) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:29:10.398222 1686 raft_consensus.cc:2932] T 815178fbd07b4631b507927683a475c9 P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 16a97c1394f346f0b987f97cce7371b6 (127.15.255.137) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } }
I20240502 21:29:10.399330 2088 raft_consensus.cc:2932] T 815178fbd07b4631b507927683a475c9 P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 16a97c1394f346f0b987f97cce7371b6 (127.15.255.137) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } }
I20240502 21:29:10.417076 2227 raft_consensus.cc:2932] T 815178fbd07b4631b507927683a475c9 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 16a97c1394f346f0b987f97cce7371b6 (127.15.255.137) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } }
I20240502 21:29:10.440404 1550 raft_consensus.cc:2932] T 815178fbd07b4631b507927683a475c9 P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 16a97c1394f346f0b987f97cce7371b6 (127.15.255.137) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } }
I20240502 21:29:10.446949 5168 ts_tablet_manager.cc:1398] T e44bf56d64b24e328803ebd0a3a54ceb P 1fab5794cf1740288abd82f2cebb9b1a: Time spent starting tablet: real 0.043s user 0.013s sys 0.026s
I20240502 21:29:10.448994 2113 tablet_copy_service.cc:342] P 521bb93035324cecbebed2f6f2a630f8: Request end of tablet copy session 1fab5794cf1740288abd82f2cebb9b1a-e44bf56d64b24e328803ebd0a3a54ceb received from {username='slave'} at 127.15.255.138:37811
I20240502 21:29:10.449121 2113 tablet_copy_service.cc:433] P 521bb93035324cecbebed2f6f2a630f8: ending tablet copy session 1fab5794cf1740288abd82f2cebb9b1a-e44bf56d64b24e328803ebd0a3a54ceb on tablet e44bf56d64b24e328803ebd0a3a54ceb with peer 1fab5794cf1740288abd82f2cebb9b1a
I20240502 21:29:10.455473 1953 raft_consensus.cc:2932] T 815178fbd07b4631b507927683a475c9 P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 16a97c1394f346f0b987f97cce7371b6 (127.15.255.137) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } }
I20240502 21:29:10.508735 5175 raft_consensus.cc:1050] T 20a7af42e83c4639bba20f22dee69f2a P 7d1e5c1b047443cbb28d7b0a9744babf: attempting to promote NON_VOTER 16a97c1394f346f0b987f97cce7371b6 to VOTER
I20240502 21:29:10.509614 5175 consensus_queue.cc:237] T 20a7af42e83c4639bba20f22dee69f2a P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } }
I20240502 21:29:10.512315 3987 raft_consensus.cc:1261] T 20a7af42e83c4639bba20f22dee69f2a P 16a97c1394f346f0b987f97cce7371b6 [term 1 LEARNER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:29:10.512506 2226 raft_consensus.cc:1261] T 20a7af42e83c4639bba20f22dee69f2a P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:29:10.512636 2088 raft_consensus.cc:1261] T 20a7af42e83c4639bba20f22dee69f2a P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:29:10.513312 1550 raft_consensus.cc:1261] T 20a7af42e83c4639bba20f22dee69f2a P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:29:10.513347 1818 raft_consensus.cc:1261] T 20a7af42e83c4639bba20f22dee69f2a P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:29:10.513350 1686 raft_consensus.cc:1261] T 20a7af42e83c4639bba20f22dee69f2a P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:29:10.513327 5000 consensus_queue.cc:1035] T 20a7af42e83c4639bba20f22dee69f2a P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:29:10.513875 5000 consensus_queue.cc:1035] T 20a7af42e83c4639bba20f22dee69f2a P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:29:10.514072 1953 raft_consensus.cc:1261] T 20a7af42e83c4639bba20f22dee69f2a P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:29:10.514150 5176 consensus_queue.cc:1035] T 20a7af42e83c4639bba20f22dee69f2a P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:29:10.514626 4889 consensus_queue.cc:1035] T 20a7af42e83c4639bba20f22dee69f2a P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:29:10.514863 5175 consensus_queue.cc:1035] T 20a7af42e83c4639bba20f22dee69f2a P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:29:10.515134 5006 consensus_queue.cc:1035] T 20a7af42e83c4639bba20f22dee69f2a P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:29:10.515723 5176 consensus_queue.cc:1035] T 20a7af42e83c4639bba20f22dee69f2a P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:29:10.533208 5000 raft_consensus.cc:2932] T 20a7af42e83c4639bba20f22dee69f2a P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 16a97c1394f346f0b987f97cce7371b6 (127.15.255.137) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } }
I20240502 21:29:10.536851 1818 raft_consensus.cc:2932] T 20a7af42e83c4639bba20f22dee69f2a P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 16a97c1394f346f0b987f97cce7371b6 (127.15.255.137) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } }
I20240502 21:29:10.537549 2088 raft_consensus.cc:2932] T 20a7af42e83c4639bba20f22dee69f2a P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 16a97c1394f346f0b987f97cce7371b6 (127.15.255.137) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } }
I20240502 21:29:10.538713 1550 raft_consensus.cc:2932] T 20a7af42e83c4639bba20f22dee69f2a P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 16a97c1394f346f0b987f97cce7371b6 (127.15.255.137) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } }
I20240502 21:29:10.541416 1269 catalog_manager.cc:5484] T 20a7af42e83c4639bba20f22dee69f2a P 7d1e5c1b047443cbb28d7b0a9744babf reported cstate change: config changed from index 5 to 6, 16a97c1394f346f0b987f97cce7371b6 (127.15.255.137) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:29:10.545012 3987 raft_consensus.cc:2932] T 20a7af42e83c4639bba20f22dee69f2a P 16a97c1394f346f0b987f97cce7371b6 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 16a97c1394f346f0b987f97cce7371b6 (127.15.255.137) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } }
I20240502 21:29:10.551218 2226 raft_consensus.cc:2932] T 20a7af42e83c4639bba20f22dee69f2a P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 16a97c1394f346f0b987f97cce7371b6 (127.15.255.137) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } }
I20240502 21:29:10.553429 1686 raft_consensus.cc:2932] T 20a7af42e83c4639bba20f22dee69f2a P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 16a97c1394f346f0b987f97cce7371b6 (127.15.255.137) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } }
I20240502 21:29:10.560523 1953 raft_consensus.cc:2932] T 20a7af42e83c4639bba20f22dee69f2a P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 16a97c1394f346f0b987f97cce7371b6 (127.15.255.137) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } }
I20240502 21:29:10.563367 1414 consensus_queue.cc:237] T 20a7af42e83c4639bba20f22dee69f2a P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } }
I20240502 21:29:10.565909 2088 raft_consensus.cc:1261] T 20a7af42e83c4639bba20f22dee69f2a P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:29:10.566098 1686 raft_consensus.cc:1261] T 20a7af42e83c4639bba20f22dee69f2a P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:29:10.566431 2226 raft_consensus.cc:1261] T 20a7af42e83c4639bba20f22dee69f2a P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:29:10.566437 3987 raft_consensus.cc:1261] T 20a7af42e83c4639bba20f22dee69f2a P 16a97c1394f346f0b987f97cce7371b6 [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:29:10.566589 5175 consensus_queue.cc:1035] T 20a7af42e83c4639bba20f22dee69f2a P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:29:10.566911 5176 consensus_queue.cc:1035] T 20a7af42e83c4639bba20f22dee69f2a P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:29:10.567596 1955 raft_consensus.cc:1261] T 20a7af42e83c4639bba20f22dee69f2a P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:29:10.567785 5000 consensus_queue.cc:1035] T 20a7af42e83c4639bba20f22dee69f2a P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:29:10.568830 5000 consensus_queue.cc:1035] T 20a7af42e83c4639bba20f22dee69f2a P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:29:10.573566 1550 raft_consensus.cc:1261] T 20a7af42e83c4639bba20f22dee69f2a P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:29:10.574303 5000 consensus_queue.cc:1035] T 20a7af42e83c4639bba20f22dee69f2a P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:29:10.574708 5176 consensus_queue.cc:1035] T 20a7af42e83c4639bba20f22dee69f2a P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:29:10.575968 5176 raft_consensus.cc:2932] T 20a7af42e83c4639bba20f22dee69f2a P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } }
I20240502 21:29:10.577872 1550 raft_consensus.cc:2932] T 20a7af42e83c4639bba20f22dee69f2a P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } }
I20240502 21:29:10.579262 1255 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 20a7af42e83c4639bba20f22dee69f2a with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:29:10.580344 1955 raft_consensus.cc:2932] T 20a7af42e83c4639bba20f22dee69f2a P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } }
I20240502 21:29:10.580896 3987 raft_consensus.cc:2932] T 20a7af42e83c4639bba20f22dee69f2a P 16a97c1394f346f0b987f97cce7371b6 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } }
I20240502 21:29:10.582273 2088 raft_consensus.cc:2932] T 20a7af42e83c4639bba20f22dee69f2a P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } }
I20240502 21:29:10.582580 2226 raft_consensus.cc:2932] T 20a7af42e83c4639bba20f22dee69f2a P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } }
I20240502 21:29:10.585170 1686 raft_consensus.cc:2932] T 20a7af42e83c4639bba20f22dee69f2a P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } }
I20240502 21:29:10.585947 1266 catalog_manager.cc:5484] T 20a7af42e83c4639bba20f22dee69f2a P e632c553ff4341328acf1cbaca678392 reported cstate change: config changed from index 6 to 7, VOTER 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132) evicted. New cstate: current_term: 1 leader_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } }
I20240502 21:29:10.595314 1798 tablet_service.cc:1514] Processing DeleteTablet for tablet 20a7af42e83c4639bba20f22dee69f2a with delete_type TABLET_DATA_TOMBSTONED (TS 153026dd7f8943cc93fedf6f48450fb4 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:60256
I20240502 21:29:10.596808 5213 tablet_replica.cc:330] T 20a7af42e83c4639bba20f22dee69f2a P 153026dd7f8943cc93fedf6f48450fb4: stopping tablet replica
I20240502 21:29:10.597092 5213 raft_consensus.cc:2229] T 20a7af42e83c4639bba20f22dee69f2a P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:29:10.597646 5213 raft_consensus.cc:2258] T 20a7af42e83c4639bba20f22dee69f2a P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:29:10.598706 5213 ts_tablet_manager.cc:1875] T 20a7af42e83c4639bba20f22dee69f2a P 153026dd7f8943cc93fedf6f48450fb4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:29:10.605079 5213 ts_tablet_manager.cc:1888] T 20a7af42e83c4639bba20f22dee69f2a P 153026dd7f8943cc93fedf6f48450fb4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240502 21:29:10.605195 5213 log.cc:1196] T 20a7af42e83c4639bba20f22dee69f2a P 153026dd7f8943cc93fedf6f48450fb4: Deleting WAL directory at /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-3/wal/wals/20a7af42e83c4639bba20f22dee69f2a
I20240502 21:29:10.605880 1256 catalog_manager.cc:4830] TS 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132:44947): tablet 20a7af42e83c4639bba20f22dee69f2a (table rebalance_test_table_1 [id=f2592b36bf814d51bd833a6ad315e513]) successfully deleted
I20240502 21:29:10.736479 4121 raft_consensus.cc:1203] T e44bf56d64b24e328803ebd0a3a54ceb P 1fab5794cf1740288abd82f2cebb9b1a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240502 21:29:10.924829 1550 consensus_queue.cc:237] T ef6c97c8da364cae96c34d33b20154fe P a62a935ff43d494ca35d0c417943133d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } }
I20240502 21:29:10.927659 2226 raft_consensus.cc:1261] T ef6c97c8da364cae96c34d33b20154fe P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Refusing update from remote peer a62a935ff43d494ca35d0c417943133d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:29:10.927675 2087 raft_consensus.cc:1261] T ef6c97c8da364cae96c34d33b20154fe P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Refusing update from remote peer a62a935ff43d494ca35d0c417943133d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:29:10.927929 1953 raft_consensus.cc:1261] T ef6c97c8da364cae96c34d33b20154fe P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Refusing update from remote peer a62a935ff43d494ca35d0c417943133d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:29:10.928733 5116 consensus_queue.cc:1035] T ef6c97c8da364cae96c34d33b20154fe P a62a935ff43d494ca35d0c417943133d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:10.929123 1818 raft_consensus.cc:1261] T ef6c97c8da364cae96c34d33b20154fe P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Refusing update from remote peer a62a935ff43d494ca35d0c417943133d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:29:10.929342 1414 raft_consensus.cc:1261] T ef6c97c8da364cae96c34d33b20154fe P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Refusing update from remote peer a62a935ff43d494ca35d0c417943133d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:29:10.929315 5116 consensus_queue.cc:1035] T ef6c97c8da364cae96c34d33b20154fe P a62a935ff43d494ca35d0c417943133d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:10.929813 5116 consensus_queue.cc:1035] T ef6c97c8da364cae96c34d33b20154fe P a62a935ff43d494ca35d0c417943133d [LEADER]: Connected to new peer: Peer: permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:10.930281 5214 consensus_queue.cc:1035] T ef6c97c8da364cae96c34d33b20154fe P a62a935ff43d494ca35d0c417943133d [LEADER]: Connected to new peer: Peer: permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:10.930670 1686 raft_consensus.cc:1261] T ef6c97c8da364cae96c34d33b20154fe P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Refusing update from remote peer a62a935ff43d494ca35d0c417943133d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:29:10.931890 5116 consensus_queue.cc:1035] T ef6c97c8da364cae96c34d33b20154fe P a62a935ff43d494ca35d0c417943133d [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:10.932474 5116 consensus_queue.cc:1035] T ef6c97c8da364cae96c34d33b20154fe P a62a935ff43d494ca35d0c417943133d [LEADER]: Connected to new peer: Peer: permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:10.943437 5214 raft_consensus.cc:2932] T ef6c97c8da364cae96c34d33b20154fe P a62a935ff43d494ca35d0c417943133d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } }
I20240502 21:29:10.944958 2087 raft_consensus.cc:2932] T ef6c97c8da364cae96c34d33b20154fe P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } }
I20240502 21:29:10.944311 2226 raft_consensus.cc:2932] T ef6c97c8da364cae96c34d33b20154fe P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } }
I20240502 21:29:10.951184 1269 catalog_manager.cc:5484] T ef6c97c8da364cae96c34d33b20154fe P 521bb93035324cecbebed2f6f2a630f8 reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "a62a935ff43d494ca35d0c417943133d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } }
I20240502 21:29:10.952572 1685 raft_consensus.cc:2932] T ef6c97c8da364cae96c34d33b20154fe P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } }
I20240502 21:29:10.956894 1953 raft_consensus.cc:2932] T ef6c97c8da364cae96c34d33b20154fe P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } }
I20240502 21:29:10.958302 1414 raft_consensus.cc:2932] T ef6c97c8da364cae96c34d33b20154fe P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } }
I20240502 21:29:10.961095 1818 raft_consensus.cc:2932] T ef6c97c8da364cae96c34d33b20154fe P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } }
I20240502 21:29:10.965668 1550 consensus_queue.cc:237] T ef6c97c8da364cae96c34d33b20154fe P a62a935ff43d494ca35d0c417943133d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: true } }
I20240502 21:29:10.970345 1818 raft_consensus.cc:1261] T ef6c97c8da364cae96c34d33b20154fe P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Refusing update from remote peer a62a935ff43d494ca35d0c417943133d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240502 21:29:10.970829 1482 consensus_peers.cc:485] T ef6c97c8da364cae96c34d33b20154fe P a62a935ff43d494ca35d0c417943133d -> Peer d8d7297186694f77bb98de9174e4279c (127.15.255.136:41459): Couldn't send request to peer d8d7297186694f77bb98de9174e4279c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ef6c97c8da364cae96c34d33b20154fe. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:29:10.972913 2226 raft_consensus.cc:1261] T ef6c97c8da364cae96c34d33b20154fe P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Refusing update from remote peer a62a935ff43d494ca35d0c417943133d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:29:10.973661 2087 raft_consensus.cc:1261] T ef6c97c8da364cae96c34d33b20154fe P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Refusing update from remote peer a62a935ff43d494ca35d0c417943133d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:29:10.973909 5247 consensus_queue.cc:1035] T ef6c97c8da364cae96c34d33b20154fe P a62a935ff43d494ca35d0c417943133d [LEADER]: Connected to new peer: Peer: permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:29:10.974356 1684 raft_consensus.cc:1261] T ef6c97c8da364cae96c34d33b20154fe P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Refusing update from remote peer a62a935ff43d494ca35d0c417943133d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:29:10.974325 5249 consensus_queue.cc:1035] T ef6c97c8da364cae96c34d33b20154fe P a62a935ff43d494ca35d0c417943133d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:29:10.975630 5249 consensus_queue.cc:1035] T ef6c97c8da364cae96c34d33b20154fe P a62a935ff43d494ca35d0c417943133d [LEADER]: Connected to new peer: Peer: permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:29:10.976866 5249 consensus_queue.cc:1035] T ef6c97c8da364cae96c34d33b20154fe P a62a935ff43d494ca35d0c417943133d [LEADER]: Connected to new peer: Peer: permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:29:10.978191 1414 raft_consensus.cc:1261] T ef6c97c8da364cae96c34d33b20154fe P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Refusing update from remote peer a62a935ff43d494ca35d0c417943133d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:29:10.978852 1953 raft_consensus.cc:1261] T ef6c97c8da364cae96c34d33b20154fe P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Refusing update from remote peer a62a935ff43d494ca35d0c417943133d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:29:10.981453 5250 consensus_queue.cc:1035] T ef6c97c8da364cae96c34d33b20154fe P a62a935ff43d494ca35d0c417943133d [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:29:10.982012 5247 consensus_queue.cc:1035] T ef6c97c8da364cae96c34d33b20154fe P a62a935ff43d494ca35d0c417943133d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:29:10.983279 5249 raft_consensus.cc:2932] T ef6c97c8da364cae96c34d33b20154fe P a62a935ff43d494ca35d0c417943133d [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER d8d7297186694f77bb98de9174e4279c (127.15.255.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: true } } }
I20240502 21:29:10.986045 1955 raft_consensus.cc:2932] T ef6c97c8da364cae96c34d33b20154fe P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER d8d7297186694f77bb98de9174e4279c (127.15.255.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: true } } }
I20240502 21:29:10.987438 1255 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet ef6c97c8da364cae96c34d33b20154fe with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240502 21:29:10.987557 1414 consensus_queue.cc:237] T 815178fbd07b4631b507927683a475c9 P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } }
I20240502 21:29:10.991816 1818 raft_consensus.cc:2932] T ef6c97c8da364cae96c34d33b20154fe P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER d8d7297186694f77bb98de9174e4279c (127.15.255.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: true } } }
I20240502 21:29:10.992192 1268 catalog_manager.cc:5484] T ef6c97c8da364cae96c34d33b20154fe P a62a935ff43d494ca35d0c417943133d reported cstate change: config changed from index 4 to 5, NON_VOTER d8d7297186694f77bb98de9174e4279c (127.15.255.136) added. New cstate: current_term: 1 leader_uuid: "a62a935ff43d494ca35d0c417943133d" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 21:29:10.994690 1415 raft_consensus.cc:2932] T ef6c97c8da364cae96c34d33b20154fe P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER d8d7297186694f77bb98de9174e4279c (127.15.255.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: true } } }
I20240502 21:29:10.994215 1686 raft_consensus.cc:2932] T ef6c97c8da364cae96c34d33b20154fe P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER d8d7297186694f77bb98de9174e4279c (127.15.255.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: true } } }
I20240502 21:29:10.995507 2226 raft_consensus.cc:2932] T ef6c97c8da364cae96c34d33b20154fe P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER d8d7297186694f77bb98de9174e4279c (127.15.255.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: true } } }
I20240502 21:29:11.000689 3987 raft_consensus.cc:1261] T 815178fbd07b4631b507927683a475c9 P 16a97c1394f346f0b987f97cce7371b6 [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:29:11.001329 5176 consensus_queue.cc:1035] T 815178fbd07b4631b507927683a475c9 P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:29:11.005463 1685 raft_consensus.cc:1261] T 815178fbd07b4631b507927683a475c9 P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:29:11.006068 5212 consensus_queue.cc:1035] T 815178fbd07b4631b507927683a475c9 P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:29:11.007102 1550 raft_consensus.cc:1261] T 815178fbd07b4631b507927683a475c9 P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:29:11.007591 5212 consensus_queue.cc:1035] T 815178fbd07b4631b507927683a475c9 P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:29:11.011487 1955 raft_consensus.cc:1261] T 815178fbd07b4631b507927683a475c9 P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:29:11.012248 2088 raft_consensus.cc:1261] T 815178fbd07b4631b507927683a475c9 P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:29:11.014317 2226 raft_consensus.cc:1261] T 815178fbd07b4631b507927683a475c9 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:29:11.014390 5176 raft_consensus.cc:2932] T 815178fbd07b4631b507927683a475c9 P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } }
I20240502 21:29:11.016559 5006 consensus_queue.cc:1035] T 815178fbd07b4631b507927683a475c9 P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:29:11.018675 1685 raft_consensus.cc:2932] T 815178fbd07b4631b507927683a475c9 P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } }
I20240502 21:29:11.019436 2226 raft_consensus.cc:2932] T 815178fbd07b4631b507927683a475c9 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } }
I20240502 21:29:11.024205 5006 consensus_queue.cc:1035] T 815178fbd07b4631b507927683a475c9 P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:29:11.024771 5006 consensus_queue.cc:1035] T 815178fbd07b4631b507927683a475c9 P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:29:11.027012 2087 raft_consensus.cc:2932] T ef6c97c8da364cae96c34d33b20154fe P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER d8d7297186694f77bb98de9174e4279c (127.15.255.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: true } } }
I20240502 21:29:11.028148 1955 raft_consensus.cc:2932] T 815178fbd07b4631b507927683a475c9 P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } }
I20240502 21:29:11.031826 2088 raft_consensus.cc:2932] T 815178fbd07b4631b507927683a475c9 P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } }
I20240502 21:29:11.035631 1255 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 815178fbd07b4631b507927683a475c9 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:29:11.037166 1550 raft_consensus.cc:2932] T 815178fbd07b4631b507927683a475c9 P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } }
I20240502 21:29:11.037400 3987 raft_consensus.cc:2932] T 815178fbd07b4631b507927683a475c9 P 16a97c1394f346f0b987f97cce7371b6 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } }
I20240502 21:29:11.045248 1265 catalog_manager.cc:5484] T 815178fbd07b4631b507927683a475c9 P 7d1e5c1b047443cbb28d7b0a9744babf reported cstate change: config changed from index 6 to 7, VOTER 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132) evicted. New cstate: current_term: 1 leader_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:29:11.050890 1798 tablet_service.cc:1514] Processing DeleteTablet for tablet 815178fbd07b4631b507927683a475c9 with delete_type TABLET_DATA_TOMBSTONED (TS 153026dd7f8943cc93fedf6f48450fb4 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:60256
I20240502 21:29:11.051299 5213 tablet_replica.cc:330] T 815178fbd07b4631b507927683a475c9 P 153026dd7f8943cc93fedf6f48450fb4: stopping tablet replica
I20240502 21:29:11.051524 5213 raft_consensus.cc:2229] T 815178fbd07b4631b507927683a475c9 P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:29:11.051903 5213 raft_consensus.cc:2258] T 815178fbd07b4631b507927683a475c9 P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:29:11.053078 5213 ts_tablet_manager.cc:1875] T 815178fbd07b4631b507927683a475c9 P 153026dd7f8943cc93fedf6f48450fb4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:29:11.063937 5213 ts_tablet_manager.cc:1888] T 815178fbd07b4631b507927683a475c9 P 153026dd7f8943cc93fedf6f48450fb4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240502 21:29:11.064071 5213 log.cc:1196] T 815178fbd07b4631b507927683a475c9 P 153026dd7f8943cc93fedf6f48450fb4: Deleting WAL directory at /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-3/wal/wals/815178fbd07b4631b507927683a475c9
I20240502 21:29:11.064810 1256 catalog_manager.cc:4830] TS 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132:44947): tablet 815178fbd07b4631b507927683a475c9 (table rebalance_test_table_3 [id=a205513324dd4db5a5ee4ed1928aefa6]) successfully deleted
I20240502 21:29:11.173557 5113 raft_consensus.cc:1050] T e44bf56d64b24e328803ebd0a3a54ceb P 521bb93035324cecbebed2f6f2a630f8: attempting to promote NON_VOTER 1fab5794cf1740288abd82f2cebb9b1a to VOTER
I20240502 21:29:11.174635 5113 consensus_queue.cc:237] T e44bf56d64b24e328803ebd0a3a54ceb P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } }
I20240502 21:29:11.177426 1686 raft_consensus.cc:1261] T e44bf56d64b24e328803ebd0a3a54ceb P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:29:11.178380 4121 raft_consensus.cc:1261] T e44bf56d64b24e328803ebd0a3a54ceb P 1fab5794cf1740288abd82f2cebb9b1a [term 1 LEARNER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:29:11.178992 2226 raft_consensus.cc:1261] T e44bf56d64b24e328803ebd0a3a54ceb P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:29:11.178999 1550 raft_consensus.cc:1261] T e44bf56d64b24e328803ebd0a3a54ceb P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:29:11.179255 1818 raft_consensus.cc:1261] T e44bf56d64b24e328803ebd0a3a54ceb P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:29:11.179644 1955 raft_consensus.cc:1261] T e44bf56d64b24e328803ebd0a3a54ceb P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:29:11.179922 5109 consensus_queue.cc:1035] T e44bf56d64b24e328803ebd0a3a54ceb P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:29:11.180552 5109 consensus_queue.cc:1035] T e44bf56d64b24e328803ebd0a3a54ceb P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:29:11.180809 5269 consensus_queue.cc:1035] T e44bf56d64b24e328803ebd0a3a54ceb P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:29:11.181149 5232 consensus_queue.cc:1035] T e44bf56d64b24e328803ebd0a3a54ceb P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:29:11.181726 1414 raft_consensus.cc:1261] T e44bf56d64b24e328803ebd0a3a54ceb P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:29:11.182246 5232 consensus_queue.cc:1035] T e44bf56d64b24e328803ebd0a3a54ceb P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:29:11.182780 5232 consensus_queue.cc:1035] T e44bf56d64b24e328803ebd0a3a54ceb P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:29:11.183252 5232 consensus_queue.cc:1035] T e44bf56d64b24e328803ebd0a3a54ceb P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:29:11.203927 5113 raft_consensus.cc:2932] T e44bf56d64b24e328803ebd0a3a54ceb P 521bb93035324cecbebed2f6f2a630f8 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } }
I20240502 21:29:11.204553 1818 raft_consensus.cc:2932] T e44bf56d64b24e328803ebd0a3a54ceb P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } }
I20240502 21:29:11.209661 1268 catalog_manager.cc:5484] T e44bf56d64b24e328803ebd0a3a54ceb P 153026dd7f8943cc93fedf6f48450fb4 reported cstate change: config changed from index 5 to 6, 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "521bb93035324cecbebed2f6f2a630f8" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } }
I20240502 21:29:11.213001 1686 raft_consensus.cc:2932] T e44bf56d64b24e328803ebd0a3a54ceb P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } }
I20240502 21:29:11.213239 4121 raft_consensus.cc:2932] T e44bf56d64b24e328803ebd0a3a54ceb P 1fab5794cf1740288abd82f2cebb9b1a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } }
I20240502 21:29:11.213622 1414 raft_consensus.cc:2932] T e44bf56d64b24e328803ebd0a3a54ceb P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } }
I20240502 21:29:11.229493 2226 raft_consensus.cc:2932] T e44bf56d64b24e328803ebd0a3a54ceb P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } }
I20240502 21:29:11.233317 1549 raft_consensus.cc:2932] T e44bf56d64b24e328803ebd0a3a54ceb P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } }
I20240502 21:29:11.242610 1953 raft_consensus.cc:2932] T e44bf56d64b24e328803ebd0a3a54ceb P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } }
I20240502 21:29:11.296300 1953 consensus_queue.cc:237] T 444c1cba2a6f4729aa01b75d06b6b517 P e632c553ff4341328acf1cbaca678392 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:29:11.299284 1414 raft_consensus.cc:1261] T 444c1cba2a6f4729aa01b75d06b6b517 P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Refusing update from remote peer e632c553ff4341328acf1cbaca678392: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:29:11.299830 1686 raft_consensus.cc:1261] T 444c1cba2a6f4729aa01b75d06b6b517 P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Refusing update from remote peer e632c553ff4341328acf1cbaca678392: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:29:11.300403 5215 consensus_queue.cc:1035] T 444c1cba2a6f4729aa01b75d06b6b517 P e632c553ff4341328acf1cbaca678392 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:11.300711 1549 raft_consensus.cc:1261] T 444c1cba2a6f4729aa01b75d06b6b517 P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Refusing update from remote peer e632c553ff4341328acf1cbaca678392: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:29:11.300881 5287 consensus_queue.cc:1035] T 444c1cba2a6f4729aa01b75d06b6b517 P e632c553ff4341328acf1cbaca678392 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:11.301586 2088 raft_consensus.cc:1261] T 444c1cba2a6f4729aa01b75d06b6b517 P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Refusing update from remote peer e632c553ff4341328acf1cbaca678392: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:29:11.302503 5287 consensus_queue.cc:1035] T 444c1cba2a6f4729aa01b75d06b6b517 P e632c553ff4341328acf1cbaca678392 [LEADER]: Connected to new peer: Peer: permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:11.303081 1818 raft_consensus.cc:1261] T 444c1cba2a6f4729aa01b75d06b6b517 P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Refusing update from remote peer e632c553ff4341328acf1cbaca678392: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:29:11.303638 5287 consensus_queue.cc:1035] T 444c1cba2a6f4729aa01b75d06b6b517 P e632c553ff4341328acf1cbaca678392 [LEADER]: Connected to new peer: Peer: permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:11.304270 5215 consensus_queue.cc:1035] T 444c1cba2a6f4729aa01b75d06b6b517 P e632c553ff4341328acf1cbaca678392 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:11.306622 2226 raft_consensus.cc:1261] T 444c1cba2a6f4729aa01b75d06b6b517 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Refusing update from remote peer e632c553ff4341328acf1cbaca678392: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:29:11.309832 5215 consensus_queue.cc:1035] T 444c1cba2a6f4729aa01b75d06b6b517 P e632c553ff4341328acf1cbaca678392 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:11.315340 5215 raft_consensus.cc:2932] T 444c1cba2a6f4729aa01b75d06b6b517 P e632c553ff4341328acf1cbaca678392 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } }
I20240502 21:29:11.317649 2087 raft_consensus.cc:2932] T 444c1cba2a6f4729aa01b75d06b6b517 P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } }
I20240502 21:29:11.318120 1686 raft_consensus.cc:2932] T 444c1cba2a6f4729aa01b75d06b6b517 P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } }
I20240502 21:29:11.319672 1820 raft_consensus.cc:2932] T 444c1cba2a6f4729aa01b75d06b6b517 P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } }
I20240502 21:29:11.321985 2227 raft_consensus.cc:2932] T 444c1cba2a6f4729aa01b75d06b6b517 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } }
I20240502 21:29:11.324313 1268 catalog_manager.cc:5484] T 444c1cba2a6f4729aa01b75d06b6b517 P 16b3a337bc5e462f83ba895ad8660deb reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "e632c553ff4341328acf1cbaca678392" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } }
I20240502 21:29:11.329182 1414 raft_consensus.cc:2932] T 444c1cba2a6f4729aa01b75d06b6b517 P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } }
I20240502 21:29:11.331007 1549 raft_consensus.cc:2932] T 444c1cba2a6f4729aa01b75d06b6b517 P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } }
I20240502 21:29:11.342113 1953 consensus_queue.cc:237] T 444c1cba2a6f4729aa01b75d06b6b517 P e632c553ff4341328acf1cbaca678392 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: true } }
I20240502 21:29:11.345396 1820 raft_consensus.cc:1261] T 444c1cba2a6f4729aa01b75d06b6b517 P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Refusing update from remote peer e632c553ff4341328acf1cbaca678392: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:29:11.346239 2087 raft_consensus.cc:1261] T 444c1cba2a6f4729aa01b75d06b6b517 P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Refusing update from remote peer e632c553ff4341328acf1cbaca678392: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:29:11.346869 5215 consensus_queue.cc:1035] T 444c1cba2a6f4729aa01b75d06b6b517 P e632c553ff4341328acf1cbaca678392 [LEADER]: Connected to new peer: Peer: permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:29:11.349673 1686 raft_consensus.cc:1261] T 444c1cba2a6f4729aa01b75d06b6b517 P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Refusing update from remote peer e632c553ff4341328acf1cbaca678392: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:29:11.350272 5215 consensus_queue.cc:1035] T 444c1cba2a6f4729aa01b75d06b6b517 P e632c553ff4341328acf1cbaca678392 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:29:11.353114 2226 raft_consensus.cc:1261] T 444c1cba2a6f4729aa01b75d06b6b517 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Refusing update from remote peer e632c553ff4341328acf1cbaca678392: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:29:11.353168 1414 raft_consensus.cc:1261] T 444c1cba2a6f4729aa01b75d06b6b517 P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Refusing update from remote peer e632c553ff4341328acf1cbaca678392: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:29:11.353288 1549 raft_consensus.cc:1261] T 444c1cba2a6f4729aa01b75d06b6b517 P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Refusing update from remote peer e632c553ff4341328acf1cbaca678392: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:29:11.353775 5288 consensus_queue.cc:1035] T 444c1cba2a6f4729aa01b75d06b6b517 P e632c553ff4341328acf1cbaca678392 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:29:11.355643 5287 consensus_queue.cc:1035] T 444c1cba2a6f4729aa01b75d06b6b517 P e632c553ff4341328acf1cbaca678392 [LEADER]: Connected to new peer: Peer: permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:29:11.356210 5288 consensus_queue.cc:1035] T 444c1cba2a6f4729aa01b75d06b6b517 P e632c553ff4341328acf1cbaca678392 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:29:11.357584 5286 raft_consensus.cc:2932] T 444c1cba2a6f4729aa01b75d06b6b517 P e632c553ff4341328acf1cbaca678392 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER d8d7297186694f77bb98de9174e4279c (127.15.255.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: true } } }
I20240502 21:29:11.358036 2226 raft_consensus.cc:2932] T 444c1cba2a6f4729aa01b75d06b6b517 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER d8d7297186694f77bb98de9174e4279c (127.15.255.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: true } } }
I20240502 21:29:11.363893 5215 consensus_queue.cc:1035] T 444c1cba2a6f4729aa01b75d06b6b517 P e632c553ff4341328acf1cbaca678392 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:29:11.363483 1414 raft_consensus.cc:2932] T 444c1cba2a6f4729aa01b75d06b6b517 P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER d8d7297186694f77bb98de9174e4279c (127.15.255.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: true } } }
I20240502 21:29:11.363034 1254 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 444c1cba2a6f4729aa01b75d06b6b517 with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240502 21:29:11.365540 1549 raft_consensus.cc:2932] T 444c1cba2a6f4729aa01b75d06b6b517 P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER d8d7297186694f77bb98de9174e4279c (127.15.255.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: true } } }
I20240502 21:29:11.366999 1270 catalog_manager.cc:5484] T 444c1cba2a6f4729aa01b75d06b6b517 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 reported cstate change: config changed from index 4 to 5, NON_VOTER d8d7297186694f77bb98de9174e4279c (127.15.255.136) added. New cstate: current_term: 1 leader_uuid: "e632c553ff4341328acf1cbaca678392" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: true } } }
I20240502 21:29:11.368283 2087 raft_consensus.cc:2932] T 444c1cba2a6f4729aa01b75d06b6b517 P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER d8d7297186694f77bb98de9174e4279c (127.15.255.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: true } } }
I20240502 21:29:11.368532 1686 raft_consensus.cc:2932] T 444c1cba2a6f4729aa01b75d06b6b517 P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER d8d7297186694f77bb98de9174e4279c (127.15.255.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: true } } }
W20240502 21:29:11.373965 1886 consensus_peers.cc:485] T 444c1cba2a6f4729aa01b75d06b6b517 P e632c553ff4341328acf1cbaca678392 -> Peer d8d7297186694f77bb98de9174e4279c (127.15.255.136:41459): Couldn't send request to peer d8d7297186694f77bb98de9174e4279c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 444c1cba2a6f4729aa01b75d06b6b517. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:29:11.375836 1820 raft_consensus.cc:2932] T 444c1cba2a6f4729aa01b75d06b6b517 P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER d8d7297186694f77bb98de9174e4279c (127.15.255.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: true } } }
I20240502 21:29:11.505712 5315 ts_tablet_manager.cc:902] T ef6c97c8da364cae96c34d33b20154fe P d8d7297186694f77bb98de9174e4279c: Initiating tablet copy from peer a62a935ff43d494ca35d0c417943133d (127.15.255.130:37991)
I20240502 21:29:11.506825 5315 tablet_copy_client.cc:308] T ef6c97c8da364cae96c34d33b20154fe P d8d7297186694f77bb98de9174e4279c: tablet copy: Beginning tablet copy session from remote peer at address 127.15.255.130:37991
I20240502 21:29:11.507443 1570 tablet_copy_service.cc:140] P a62a935ff43d494ca35d0c417943133d: Received BeginTabletCopySession request for tablet ef6c97c8da364cae96c34d33b20154fe from peer d8d7297186694f77bb98de9174e4279c ({username='slave'} at 127.15.255.136:51073)
I20240502 21:29:11.507579 1570 tablet_copy_service.cc:161] P a62a935ff43d494ca35d0c417943133d: Beginning new tablet copy session on tablet ef6c97c8da364cae96c34d33b20154fe from peer d8d7297186694f77bb98de9174e4279c at {username='slave'} at 127.15.255.136:51073: session id = d8d7297186694f77bb98de9174e4279c-ef6c97c8da364cae96c34d33b20154fe
I20240502 21:29:11.509603 1570 tablet_copy_source_session.cc:205] T ef6c97c8da364cae96c34d33b20154fe P a62a935ff43d494ca35d0c417943133d: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:29:11.510943 5315 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef6c97c8da364cae96c34d33b20154fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:29:11.515365 5315 tablet_copy_client.cc:783] T ef6c97c8da364cae96c34d33b20154fe P d8d7297186694f77bb98de9174e4279c: tablet copy: Starting download of 0 data blocks...
I20240502 21:29:11.515588 5315 tablet_copy_client.cc:647] T ef6c97c8da364cae96c34d33b20154fe P d8d7297186694f77bb98de9174e4279c: tablet copy: Starting download of 1 WAL segments...
I20240502 21:29:11.517069 5315 tablet_copy_client.cc:523] T ef6c97c8da364cae96c34d33b20154fe P d8d7297186694f77bb98de9174e4279c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:29:11.519549 5315 tablet_bootstrap.cc:492] T ef6c97c8da364cae96c34d33b20154fe P d8d7297186694f77bb98de9174e4279c: Bootstrap starting.
I20240502 21:29:11.554845 5315 tablet_bootstrap.cc:492] T ef6c97c8da364cae96c34d33b20154fe P d8d7297186694f77bb98de9174e4279c: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:29:11.555240 5315 tablet_bootstrap.cc:492] T ef6c97c8da364cae96c34d33b20154fe P d8d7297186694f77bb98de9174e4279c: Bootstrap complete.
I20240502 21:29:11.555508 5315 ts_tablet_manager.cc:1367] T ef6c97c8da364cae96c34d33b20154fe P d8d7297186694f77bb98de9174e4279c: Time spent bootstrapping tablet: real 0.036s user 0.035s sys 0.000s
I20240502 21:29:11.556141 5315 raft_consensus.cc:348] T ef6c97c8da364cae96c34d33b20154fe P d8d7297186694f77bb98de9174e4279c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: true } }
I20240502 21:29:11.556521 5315 raft_consensus.cc:729] T ef6c97c8da364cae96c34d33b20154fe P d8d7297186694f77bb98de9174e4279c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d8d7297186694f77bb98de9174e4279c, State: Initialized, Role: LEARNER
I20240502 21:29:11.556758 5315 consensus_queue.cc:260] T ef6c97c8da364cae96c34d33b20154fe P d8d7297186694f77bb98de9174e4279c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: true } }
I20240502 21:29:11.558306 5315 ts_tablet_manager.cc:1398] T ef6c97c8da364cae96c34d33b20154fe P d8d7297186694f77bb98de9174e4279c: Time spent starting tablet: real 0.003s user 0.005s sys 0.000s
I20240502 21:29:11.559746 1570 tablet_copy_service.cc:342] P a62a935ff43d494ca35d0c417943133d: Request end of tablet copy session d8d7297186694f77bb98de9174e4279c-ef6c97c8da364cae96c34d33b20154fe received from {username='slave'} at 127.15.255.136:51073
I20240502 21:29:11.559885 1570 tablet_copy_service.cc:433] P a62a935ff43d494ca35d0c417943133d: ending tablet copy session d8d7297186694f77bb98de9174e4279c-ef6c97c8da364cae96c34d33b20154fe on tablet ef6c97c8da364cae96c34d33b20154fe with peer d8d7297186694f77bb98de9174e4279c
I20240502 21:29:11.823087 2087 consensus_queue.cc:237] T e44bf56d64b24e328803ebd0a3a54ceb P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } }
I20240502 21:29:11.825470 1686 raft_consensus.cc:1261] T e44bf56d64b24e328803ebd0a3a54ceb P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:29:11.825493 4121 raft_consensus.cc:1261] T e44bf56d64b24e328803ebd0a3a54ceb P 1fab5794cf1740288abd82f2cebb9b1a [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:29:11.825516 1414 raft_consensus.cc:1261] T e44bf56d64b24e328803ebd0a3a54ceb P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:29:11.826175 5339 consensus_queue.cc:1035] T e44bf56d64b24e328803ebd0a3a54ceb P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:29:11.827199 2226 raft_consensus.cc:1261] T e44bf56d64b24e328803ebd0a3a54ceb P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:29:11.827368 5339 consensus_queue.cc:1035] T e44bf56d64b24e328803ebd0a3a54ceb P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:29:11.828126 5339 consensus_queue.cc:1035] T e44bf56d64b24e328803ebd0a3a54ceb P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:29:11.829346 5113 consensus_queue.cc:1035] T e44bf56d64b24e328803ebd0a3a54ceb P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:29:11.831348 5270 raft_consensus.cc:2932] T e44bf56d64b24e328803ebd0a3a54ceb P 521bb93035324cecbebed2f6f2a630f8 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } }
I20240502 21:29:11.832937 2226 raft_consensus.cc:2932] T e44bf56d64b24e328803ebd0a3a54ceb P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } }
I20240502 21:29:11.834570 1955 raft_consensus.cc:1261] T e44bf56d64b24e328803ebd0a3a54ceb P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:29:11.835121 5340 consensus_queue.cc:1035] T e44bf56d64b24e328803ebd0a3a54ceb P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:29:11.837033 1955 raft_consensus.cc:2932] T e44bf56d64b24e328803ebd0a3a54ceb P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } }
I20240502 21:29:11.839628 1549 raft_consensus.cc:1261] T e44bf56d64b24e328803ebd0a3a54ceb P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:29:11.839893 1254 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet e44bf56d64b24e328803ebd0a3a54ceb with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:29:11.840591 4121 raft_consensus.cc:2932] T e44bf56d64b24e328803ebd0a3a54ceb P 1fab5794cf1740288abd82f2cebb9b1a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } }
I20240502 21:29:11.840698 1686 raft_consensus.cc:2932] T e44bf56d64b24e328803ebd0a3a54ceb P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } }
I20240502 21:29:11.849486 1263 catalog_manager.cc:5484] T e44bf56d64b24e328803ebd0a3a54ceb P e632c553ff4341328acf1cbaca678392 reported cstate change: config changed from index 6 to 7, VOTER 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132) evicted. New cstate: current_term: 1 leader_uuid: "521bb93035324cecbebed2f6f2a630f8" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } }
I20240502 21:29:11.854214 1798 tablet_service.cc:1514] Processing DeleteTablet for tablet e44bf56d64b24e328803ebd0a3a54ceb with delete_type TABLET_DATA_TOMBSTONED (TS 153026dd7f8943cc93fedf6f48450fb4 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:60256
I20240502 21:29:11.871057 5339 consensus_queue.cc:1035] T e44bf56d64b24e328803ebd0a3a54ceb P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:29:11.871933 1414 raft_consensus.cc:2932] T e44bf56d64b24e328803ebd0a3a54ceb P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } }
I20240502 21:29:11.873399 1549 raft_consensus.cc:2932] T e44bf56d64b24e328803ebd0a3a54ceb P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } }
I20240502 21:29:11.882814 5347 tablet_replica.cc:330] T e44bf56d64b24e328803ebd0a3a54ceb P 153026dd7f8943cc93fedf6f48450fb4: stopping tablet replica
I20240502 21:29:11.883081 5347 raft_consensus.cc:2229] T e44bf56d64b24e328803ebd0a3a54ceb P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:29:11.883498 5347 raft_consensus.cc:2258] T e44bf56d64b24e328803ebd0a3a54ceb P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:29:11.884538 5347 ts_tablet_manager.cc:1875] T e44bf56d64b24e328803ebd0a3a54ceb P 153026dd7f8943cc93fedf6f48450fb4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:29:11.894078 5347 ts_tablet_manager.cc:1888] T e44bf56d64b24e328803ebd0a3a54ceb P 153026dd7f8943cc93fedf6f48450fb4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240502 21:29:11.894204 5347 log.cc:1196] T e44bf56d64b24e328803ebd0a3a54ceb P 153026dd7f8943cc93fedf6f48450fb4: Deleting WAL directory at /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-3/wal/wals/e44bf56d64b24e328803ebd0a3a54ceb
I20240502 21:29:11.895007 1256 catalog_manager.cc:4830] TS 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132:44947): tablet e44bf56d64b24e328803ebd0a3a54ceb (table rebalance_test_table_1 [id=f2592b36bf814d51bd833a6ad315e513]) successfully deleted
I20240502 21:29:11.947098 5315 ts_tablet_manager.cc:902] T 444c1cba2a6f4729aa01b75d06b6b517 P d8d7297186694f77bb98de9174e4279c: Initiating tablet copy from peer e632c553ff4341328acf1cbaca678392 (127.15.255.133:37439)
I20240502 21:29:11.948275 5315 tablet_copy_client.cc:308] T 444c1cba2a6f4729aa01b75d06b6b517 P d8d7297186694f77bb98de9174e4279c: tablet copy: Beginning tablet copy session from remote peer at address 127.15.255.133:37439
I20240502 21:29:11.954001 1975 tablet_copy_service.cc:140] P e632c553ff4341328acf1cbaca678392: Received BeginTabletCopySession request for tablet 444c1cba2a6f4729aa01b75d06b6b517 from peer d8d7297186694f77bb98de9174e4279c ({username='slave'} at 127.15.255.136:48551)
I20240502 21:29:11.954151 1975 tablet_copy_service.cc:161] P e632c553ff4341328acf1cbaca678392: Beginning new tablet copy session on tablet 444c1cba2a6f4729aa01b75d06b6b517 from peer d8d7297186694f77bb98de9174e4279c at {username='slave'} at 127.15.255.136:48551: session id = d8d7297186694f77bb98de9174e4279c-444c1cba2a6f4729aa01b75d06b6b517
I20240502 21:29:11.956185 1975 tablet_copy_source_session.cc:205] T 444c1cba2a6f4729aa01b75d06b6b517 P e632c553ff4341328acf1cbaca678392: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:29:11.957397 5315 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 444c1cba2a6f4729aa01b75d06b6b517. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:29:11.963716 5315 tablet_copy_client.cc:783] T 444c1cba2a6f4729aa01b75d06b6b517 P d8d7297186694f77bb98de9174e4279c: tablet copy: Starting download of 0 data blocks...
I20240502 21:29:11.964035 5315 tablet_copy_client.cc:647] T 444c1cba2a6f4729aa01b75d06b6b517 P d8d7297186694f77bb98de9174e4279c: tablet copy: Starting download of 1 WAL segments...
I20240502 21:29:11.965755 5315 tablet_copy_client.cc:523] T 444c1cba2a6f4729aa01b75d06b6b517 P d8d7297186694f77bb98de9174e4279c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:29:11.969241 5315 tablet_bootstrap.cc:492] T 444c1cba2a6f4729aa01b75d06b6b517 P d8d7297186694f77bb98de9174e4279c: Bootstrap starting.
I20240502 21:29:11.999511 3856 raft_consensus.cc:1203] T ef6c97c8da364cae96c34d33b20154fe P d8d7297186694f77bb98de9174e4279c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240502 21:29:12.019932 5315 tablet_bootstrap.cc:492] T 444c1cba2a6f4729aa01b75d06b6b517 P d8d7297186694f77bb98de9174e4279c: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:29:12.020562 5315 tablet_bootstrap.cc:492] T 444c1cba2a6f4729aa01b75d06b6b517 P d8d7297186694f77bb98de9174e4279c: Bootstrap complete.
I20240502 21:29:12.020888 5315 ts_tablet_manager.cc:1367] T 444c1cba2a6f4729aa01b75d06b6b517 P d8d7297186694f77bb98de9174e4279c: Time spent bootstrapping tablet: real 0.052s user 0.046s sys 0.004s
I20240502 21:29:12.021950 5315 raft_consensus.cc:348] T 444c1cba2a6f4729aa01b75d06b6b517 P d8d7297186694f77bb98de9174e4279c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: true } }
I20240502 21:29:12.022327 5315 raft_consensus.cc:729] T 444c1cba2a6f4729aa01b75d06b6b517 P d8d7297186694f77bb98de9174e4279c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d8d7297186694f77bb98de9174e4279c, State: Initialized, Role: LEARNER
I20240502 21:29:12.022610 5315 consensus_queue.cc:260] T 444c1cba2a6f4729aa01b75d06b6b517 P d8d7297186694f77bb98de9174e4279c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: true } }
I20240502 21:29:12.023820 5315 ts_tablet_manager.cc:1398] T 444c1cba2a6f4729aa01b75d06b6b517 P d8d7297186694f77bb98de9174e4279c: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 21:29:12.024574 1975 tablet_copy_service.cc:342] P e632c553ff4341328acf1cbaca678392: Request end of tablet copy session d8d7297186694f77bb98de9174e4279c-444c1cba2a6f4729aa01b75d06b6b517 received from {username='slave'} at 127.15.255.136:48551
I20240502 21:29:12.024714 1975 tablet_copy_service.cc:433] P e632c553ff4341328acf1cbaca678392: ending tablet copy session d8d7297186694f77bb98de9174e4279c-444c1cba2a6f4729aa01b75d06b6b517 on tablet 444c1cba2a6f4729aa01b75d06b6b517 with peer d8d7297186694f77bb98de9174e4279c
I20240502 21:29:12.136534 2087 consensus_queue.cc:237] T c46e36b0bceb45b4bafbf1ffaf8e039e P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } }
I20240502 21:29:12.139139 1414 raft_consensus.cc:1261] T c46e36b0bceb45b4bafbf1ffaf8e039e P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:29:12.139139 1686 raft_consensus.cc:1261] T c46e36b0bceb45b4bafbf1ffaf8e039e P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:29:12.139678 1549 raft_consensus.cc:1261] T c46e36b0bceb45b4bafbf1ffaf8e039e P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:29:12.139834 5340 consensus_queue.cc:1035] T c46e36b0bceb45b4bafbf1ffaf8e039e P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:12.139988 1955 raft_consensus.cc:1261] T c46e36b0bceb45b4bafbf1ffaf8e039e P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:29:12.140898 1818 raft_consensus.cc:1261] T c46e36b0bceb45b4bafbf1ffaf8e039e P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:29:12.140893 5270 consensus_queue.cc:1035] T c46e36b0bceb45b4bafbf1ffaf8e039e P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:12.141096 2226 raft_consensus.cc:1261] T c46e36b0bceb45b4bafbf1ffaf8e039e P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:29:12.141405 5270 consensus_queue.cc:1035] T c46e36b0bceb45b4bafbf1ffaf8e039e P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:12.141661 5113 consensus_queue.cc:1035] T c46e36b0bceb45b4bafbf1ffaf8e039e P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:12.143855 5340 consensus_queue.cc:1035] T c46e36b0bceb45b4bafbf1ffaf8e039e P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:12.145080 5340 consensus_queue.cc:1035] T c46e36b0bceb45b4bafbf1ffaf8e039e P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:12.154531 5270 raft_consensus.cc:2932] T c46e36b0bceb45b4bafbf1ffaf8e039e P 521bb93035324cecbebed2f6f2a630f8 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } }
I20240502 21:29:12.155247 1818 raft_consensus.cc:2932] T c46e36b0bceb45b4bafbf1ffaf8e039e P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } }
I20240502 21:29:12.160404 1686 raft_consensus.cc:2932] T c46e36b0bceb45b4bafbf1ffaf8e039e P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } }
I20240502 21:29:12.165515 1549 raft_consensus.cc:2932] T c46e36b0bceb45b4bafbf1ffaf8e039e P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } }
I20240502 21:29:12.168983 1414 raft_consensus.cc:2932] T c46e36b0bceb45b4bafbf1ffaf8e039e P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } }
I20240502 21:29:12.169729 1269 catalog_manager.cc:5484] T c46e36b0bceb45b4bafbf1ffaf8e039e P 153026dd7f8943cc93fedf6f48450fb4 reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "521bb93035324cecbebed2f6f2a630f8" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } }
I20240502 21:29:12.174424 1955 raft_consensus.cc:2932] T c46e36b0bceb45b4bafbf1ffaf8e039e P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } }
I20240502 21:29:12.179606 2226 raft_consensus.cc:2932] T c46e36b0bceb45b4bafbf1ffaf8e039e P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } }
I20240502 21:29:12.186883 2087 consensus_queue.cc:237] T c46e36b0bceb45b4bafbf1ffaf8e039e P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: NON_VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: true } }
I20240502 21:29:12.190711 1686 raft_consensus.cc:1261] T c46e36b0bceb45b4bafbf1ffaf8e039e P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240502 21:29:12.191704 2025 consensus_peers.cc:485] T c46e36b0bceb45b4bafbf1ffaf8e039e P 521bb93035324cecbebed2f6f2a630f8 -> Peer 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138:33375): Couldn't send request to peer 1fab5794cf1740288abd82f2cebb9b1a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c46e36b0bceb45b4bafbf1ffaf8e039e. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:29:12.192020 5339 consensus_queue.cc:1035] T c46e36b0bceb45b4bafbf1ffaf8e039e P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:29:12.193141 1818 raft_consensus.cc:1261] T c46e36b0bceb45b4bafbf1ffaf8e039e P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:29:12.194242 5339 consensus_queue.cc:1035] T c46e36b0bceb45b4bafbf1ffaf8e039e P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:29:12.195760 1414 raft_consensus.cc:1261] T c46e36b0bceb45b4bafbf1ffaf8e039e P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:29:12.196630 2226 raft_consensus.cc:1261] T c46e36b0bceb45b4bafbf1ffaf8e039e P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:29:12.196738 5270 consensus_queue.cc:1035] T c46e36b0bceb45b4bafbf1ffaf8e039e P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:29:12.197338 5113 consensus_queue.cc:1035] T c46e36b0bceb45b4bafbf1ffaf8e039e P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:29:12.197623 1955 raft_consensus.cc:1261] T c46e36b0bceb45b4bafbf1ffaf8e039e P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:29:12.198175 5339 consensus_queue.cc:1035] T c46e36b0bceb45b4bafbf1ffaf8e039e P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:29:12.198482 1549 raft_consensus.cc:1261] T c46e36b0bceb45b4bafbf1ffaf8e039e P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:29:12.199744 5339 consensus_queue.cc:1035] T c46e36b0bceb45b4bafbf1ffaf8e039e P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:29:12.200357 5339 raft_consensus.cc:2932] T c46e36b0bceb45b4bafbf1ffaf8e039e P 521bb93035324cecbebed2f6f2a630f8 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: NON_VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: true } } }
I20240502 21:29:12.202114 1953 raft_consensus.cc:2932] T c46e36b0bceb45b4bafbf1ffaf8e039e P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: NON_VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: true } } }
I20240502 21:29:12.206440 1414 raft_consensus.cc:2932] T c46e36b0bceb45b4bafbf1ffaf8e039e P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: NON_VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: true } } }
I20240502 21:29:12.208850 2226 raft_consensus.cc:2932] T c46e36b0bceb45b4bafbf1ffaf8e039e P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: NON_VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: true } } }
I20240502 21:29:12.208940 1550 raft_consensus.cc:2932] T c46e36b0bceb45b4bafbf1ffaf8e039e P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: NON_VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: true } } }
I20240502 21:29:12.213743 1263 catalog_manager.cc:5484] T c46e36b0bceb45b4bafbf1ffaf8e039e P e632c553ff4341328acf1cbaca678392 reported cstate change: config changed from index 4 to 5, NON_VOTER 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138) added. New cstate: current_term: 1 leader_uuid: "521bb93035324cecbebed2f6f2a630f8" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: NON_VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: true } } }
I20240502 21:29:12.219564 1254 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet c46e36b0bceb45b4bafbf1ffaf8e039e with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240502 21:29:12.220506 1818 raft_consensus.cc:2932] T c46e36b0bceb45b4bafbf1ffaf8e039e P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: NON_VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: true } } }
I20240502 21:29:12.221782 1686 raft_consensus.cc:2932] T c46e36b0bceb45b4bafbf1ffaf8e039e P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: NON_VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: true } } }
I20240502 21:29:12.356061 3856 raft_consensus.cc:1203] T 444c1cba2a6f4729aa01b75d06b6b517 P d8d7297186694f77bb98de9174e4279c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240502 21:29:12.616618 5323 raft_consensus.cc:1050] T ef6c97c8da364cae96c34d33b20154fe P a62a935ff43d494ca35d0c417943133d: attempting to promote NON_VOTER d8d7297186694f77bb98de9174e4279c to VOTER
I20240502 21:29:12.617476 5323 consensus_queue.cc:237] T ef6c97c8da364cae96c34d33b20154fe P a62a935ff43d494ca35d0c417943133d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } }
I20240502 21:29:12.620327 2226 raft_consensus.cc:1261] T ef6c97c8da364cae96c34d33b20154fe P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Refusing update from remote peer a62a935ff43d494ca35d0c417943133d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:29:12.620651 2087 raft_consensus.cc:1261] T ef6c97c8da364cae96c34d33b20154fe P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Refusing update from remote peer a62a935ff43d494ca35d0c417943133d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:29:12.620923 3856 raft_consensus.cc:1261] T ef6c97c8da364cae96c34d33b20154fe P d8d7297186694f77bb98de9174e4279c [term 1 LEARNER]: Refusing update from remote peer a62a935ff43d494ca35d0c417943133d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:29:12.621439 5323 consensus_queue.cc:1035] T ef6c97c8da364cae96c34d33b20154fe P a62a935ff43d494ca35d0c417943133d [LEADER]: Connected to new peer: Peer: permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:29:12.621574 1818 raft_consensus.cc:1261] T ef6c97c8da364cae96c34d33b20154fe P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Refusing update from remote peer a62a935ff43d494ca35d0c417943133d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:29:12.621958 5323 consensus_queue.cc:1035] T ef6c97c8da364cae96c34d33b20154fe P a62a935ff43d494ca35d0c417943133d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:29:12.622076 1414 raft_consensus.cc:1261] T ef6c97c8da364cae96c34d33b20154fe P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Refusing update from remote peer a62a935ff43d494ca35d0c417943133d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:29:12.622340 1685 raft_consensus.cc:1261] T ef6c97c8da364cae96c34d33b20154fe P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Refusing update from remote peer a62a935ff43d494ca35d0c417943133d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:29:12.622655 5250 consensus_queue.cc:1035] T ef6c97c8da364cae96c34d33b20154fe P a62a935ff43d494ca35d0c417943133d [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:29:12.622926 5415 consensus_queue.cc:1035] T ef6c97c8da364cae96c34d33b20154fe P a62a935ff43d494ca35d0c417943133d [LEADER]: Connected to new peer: Peer: permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:29:12.623443 5250 consensus_queue.cc:1035] T ef6c97c8da364cae96c34d33b20154fe P a62a935ff43d494ca35d0c417943133d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:29:12.623891 5250 consensus_queue.cc:1035] T ef6c97c8da364cae96c34d33b20154fe P a62a935ff43d494ca35d0c417943133d [LEADER]: Connected to new peer: Peer: permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:29:12.631310 1953 raft_consensus.cc:1261] T ef6c97c8da364cae96c34d33b20154fe P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Refusing update from remote peer a62a935ff43d494ca35d0c417943133d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:29:12.632473 5416 consensus_queue.cc:1035] T ef6c97c8da364cae96c34d33b20154fe P a62a935ff43d494ca35d0c417943133d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:29:12.637996 5416 raft_consensus.cc:2932] T ef6c97c8da364cae96c34d33b20154fe P a62a935ff43d494ca35d0c417943133d [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, d8d7297186694f77bb98de9174e4279c (127.15.255.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } }
I20240502 21:29:12.638513 2227 raft_consensus.cc:2932] T ef6c97c8da364cae96c34d33b20154fe P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, d8d7297186694f77bb98de9174e4279c (127.15.255.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } }
I20240502 21:29:12.644678 2087 raft_consensus.cc:2932] T ef6c97c8da364cae96c34d33b20154fe P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, d8d7297186694f77bb98de9174e4279c (127.15.255.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } }
I20240502 21:29:12.645102 1263 catalog_manager.cc:5484] T ef6c97c8da364cae96c34d33b20154fe P 6ab8df3c796f4b6cb3e29d5045ad3ea0 reported cstate change: config changed from index 5 to 6, d8d7297186694f77bb98de9174e4279c (127.15.255.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a62a935ff43d494ca35d0c417943133d" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } }
I20240502 21:29:12.646379 1953 raft_consensus.cc:2932] T ef6c97c8da364cae96c34d33b20154fe P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, d8d7297186694f77bb98de9174e4279c (127.15.255.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } }
I20240502 21:29:12.648931 1414 raft_consensus.cc:2932] T ef6c97c8da364cae96c34d33b20154fe P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, d8d7297186694f77bb98de9174e4279c (127.15.255.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } }
I20240502 21:29:12.649696 1686 raft_consensus.cc:2932] T ef6c97c8da364cae96c34d33b20154fe P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, d8d7297186694f77bb98de9174e4279c (127.15.255.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } }
I20240502 21:29:12.658828 3856 raft_consensus.cc:2932] T ef6c97c8da364cae96c34d33b20154fe P d8d7297186694f77bb98de9174e4279c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, d8d7297186694f77bb98de9174e4279c (127.15.255.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } }
I20240502 21:29:12.661190 1818 raft_consensus.cc:2932] T ef6c97c8da364cae96c34d33b20154fe P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, d8d7297186694f77bb98de9174e4279c (127.15.255.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } }
I20240502 21:29:12.666944 1550 consensus_queue.cc:237] T ef6c97c8da364cae96c34d33b20154fe P a62a935ff43d494ca35d0c417943133d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } }
I20240502 21:29:12.669342 3856 raft_consensus.cc:1261] T ef6c97c8da364cae96c34d33b20154fe P d8d7297186694f77bb98de9174e4279c [term 1 FOLLOWER]: Refusing update from remote peer a62a935ff43d494ca35d0c417943133d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:29:12.669517 2226 raft_consensus.cc:1261] T ef6c97c8da364cae96c34d33b20154fe P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Refusing update from remote peer a62a935ff43d494ca35d0c417943133d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:29:12.669958 1953 raft_consensus.cc:1261] T ef6c97c8da364cae96c34d33b20154fe P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Refusing update from remote peer a62a935ff43d494ca35d0c417943133d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:29:12.670264 2087 raft_consensus.cc:1261] T ef6c97c8da364cae96c34d33b20154fe P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Refusing update from remote peer a62a935ff43d494ca35d0c417943133d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:29:12.670334 1414 raft_consensus.cc:1261] T ef6c97c8da364cae96c34d33b20154fe P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Refusing update from remote peer a62a935ff43d494ca35d0c417943133d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:29:12.670898 5415 consensus_queue.cc:1035] T ef6c97c8da364cae96c34d33b20154fe P a62a935ff43d494ca35d0c417943133d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:29:12.671157 5250 consensus_queue.cc:1035] T ef6c97c8da364cae96c34d33b20154fe P a62a935ff43d494ca35d0c417943133d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:29:12.671623 5250 consensus_queue.cc:1035] T ef6c97c8da364cae96c34d33b20154fe P a62a935ff43d494ca35d0c417943133d [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:29:12.671905 5417 consensus_queue.cc:1035] T ef6c97c8da364cae96c34d33b20154fe P a62a935ff43d494ca35d0c417943133d [LEADER]: Connected to new peer: Peer: permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:29:12.673250 1686 raft_consensus.cc:1261] T ef6c97c8da364cae96c34d33b20154fe P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Refusing update from remote peer a62a935ff43d494ca35d0c417943133d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:29:12.675304 5250 consensus_queue.cc:1035] T ef6c97c8da364cae96c34d33b20154fe P a62a935ff43d494ca35d0c417943133d [LEADER]: Connected to new peer: Peer: permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:29:12.676779 5416 raft_consensus.cc:2932] T ef6c97c8da364cae96c34d33b20154fe P a62a935ff43d494ca35d0c417943133d [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } }
I20240502 21:29:12.677352 1414 raft_consensus.cc:2932] T ef6c97c8da364cae96c34d33b20154fe P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } }
I20240502 21:29:12.681767 5415 consensus_queue.cc:1035] T ef6c97c8da364cae96c34d33b20154fe P a62a935ff43d494ca35d0c417943133d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:29:12.683115 1266 catalog_manager.cc:5484] T ef6c97c8da364cae96c34d33b20154fe P 7d1e5c1b047443cbb28d7b0a9744babf reported cstate change: config changed from index 6 to 7, VOTER 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132) evicted. New cstate: current_term: 1 leader_uuid: "a62a935ff43d494ca35d0c417943133d" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } }
I20240502 21:29:12.683034 1686 raft_consensus.cc:2932] T ef6c97c8da364cae96c34d33b20154fe P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } }
I20240502 21:29:12.683521 2226 raft_consensus.cc:2932] T ef6c97c8da364cae96c34d33b20154fe P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } }
I20240502 21:29:12.684000 3856 raft_consensus.cc:2932] T ef6c97c8da364cae96c34d33b20154fe P d8d7297186694f77bb98de9174e4279c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } }
I20240502 21:29:12.684736 1255 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet ef6c97c8da364cae96c34d33b20154fe with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:29:12.690006 1798 tablet_service.cc:1514] Processing DeleteTablet for tablet ef6c97c8da364cae96c34d33b20154fe with delete_type TABLET_DATA_TOMBSTONED (TS 153026dd7f8943cc93fedf6f48450fb4 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:60256
I20240502 21:29:12.693425 2087 raft_consensus.cc:2932] T ef6c97c8da364cae96c34d33b20154fe P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } }
I20240502 21:29:12.699064 1953 raft_consensus.cc:2932] T ef6c97c8da364cae96c34d33b20154fe P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } }
I20240502 21:29:12.700253 5427 tablet_replica.cc:330] T ef6c97c8da364cae96c34d33b20154fe P 153026dd7f8943cc93fedf6f48450fb4: stopping tablet replica
I20240502 21:29:12.700564 5427 raft_consensus.cc:2229] T ef6c97c8da364cae96c34d33b20154fe P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:29:12.700906 5427 raft_consensus.cc:2258] T ef6c97c8da364cae96c34d33b20154fe P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:29:12.701398 5426 ts_tablet_manager.cc:902] T c46e36b0bceb45b4bafbf1ffaf8e039e P 1fab5794cf1740288abd82f2cebb9b1a: Initiating tablet copy from peer 521bb93035324cecbebed2f6f2a630f8 (127.15.255.134:40821)
I20240502 21:29:12.710225 5426 tablet_copy_client.cc:308] T c46e36b0bceb45b4bafbf1ffaf8e039e P 1fab5794cf1740288abd82f2cebb9b1a: tablet copy: Beginning tablet copy session from remote peer at address 127.15.255.134:40821
I20240502 21:29:12.710915 2113 tablet_copy_service.cc:140] P 521bb93035324cecbebed2f6f2a630f8: Received BeginTabletCopySession request for tablet c46e36b0bceb45b4bafbf1ffaf8e039e from peer 1fab5794cf1740288abd82f2cebb9b1a ({username='slave'} at 127.15.255.138:37811)
I20240502 21:29:12.711045 2113 tablet_copy_service.cc:161] P 521bb93035324cecbebed2f6f2a630f8: Beginning new tablet copy session on tablet c46e36b0bceb45b4bafbf1ffaf8e039e from peer 1fab5794cf1740288abd82f2cebb9b1a at {username='slave'} at 127.15.255.138:37811: session id = 1fab5794cf1740288abd82f2cebb9b1a-c46e36b0bceb45b4bafbf1ffaf8e039e
I20240502 21:29:12.713006 2113 tablet_copy_source_session.cc:205] T c46e36b0bceb45b4bafbf1ffaf8e039e P 521bb93035324cecbebed2f6f2a630f8: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:29:12.714198 5426 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c46e36b0bceb45b4bafbf1ffaf8e039e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:29:12.717180 5427 ts_tablet_manager.cc:1875] T ef6c97c8da364cae96c34d33b20154fe P 153026dd7f8943cc93fedf6f48450fb4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:29:12.720644 5426 tablet_copy_client.cc:783] T c46e36b0bceb45b4bafbf1ffaf8e039e P 1fab5794cf1740288abd82f2cebb9b1a: tablet copy: Starting download of 0 data blocks...
I20240502 21:29:12.722057 5427 ts_tablet_manager.cc:1888] T ef6c97c8da364cae96c34d33b20154fe P 153026dd7f8943cc93fedf6f48450fb4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240502 21:29:12.722056 5426 tablet_copy_client.cc:647] T c46e36b0bceb45b4bafbf1ffaf8e039e P 1fab5794cf1740288abd82f2cebb9b1a: tablet copy: Starting download of 1 WAL segments...
I20240502 21:29:12.722139 5427 log.cc:1196] T ef6c97c8da364cae96c34d33b20154fe P 153026dd7f8943cc93fedf6f48450fb4: Deleting WAL directory at /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-3/wal/wals/ef6c97c8da364cae96c34d33b20154fe
I20240502 21:29:12.722956 1256 catalog_manager.cc:4830] TS 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132:44947): tablet ef6c97c8da364cae96c34d33b20154fe (table rebalance_test_table_6 [id=552d53f5242c4a2eb8e89b63e8eb6a7a]) successfully deleted
I20240502 21:29:12.724589 5426 tablet_copy_client.cc:523] T c46e36b0bceb45b4bafbf1ffaf8e039e P 1fab5794cf1740288abd82f2cebb9b1a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:29:12.727770 5426 tablet_bootstrap.cc:492] T c46e36b0bceb45b4bafbf1ffaf8e039e P 1fab5794cf1740288abd82f2cebb9b1a: Bootstrap starting.
I20240502 21:29:12.767381 5426 tablet_bootstrap.cc:492] T c46e36b0bceb45b4bafbf1ffaf8e039e P 1fab5794cf1740288abd82f2cebb9b1a: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:29:12.767957 5426 tablet_bootstrap.cc:492] T c46e36b0bceb45b4bafbf1ffaf8e039e P 1fab5794cf1740288abd82f2cebb9b1a: Bootstrap complete.
I20240502 21:29:12.768209 5426 ts_tablet_manager.cc:1367] T c46e36b0bceb45b4bafbf1ffaf8e039e P 1fab5794cf1740288abd82f2cebb9b1a: Time spent bootstrapping tablet: real 0.041s user 0.038s sys 0.000s
I20240502 21:29:12.768893 5426 raft_consensus.cc:348] T c46e36b0bceb45b4bafbf1ffaf8e039e P 1fab5794cf1740288abd82f2cebb9b1a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: NON_VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: true } }
I20240502 21:29:12.769138 5426 raft_consensus.cc:729] T c46e36b0bceb45b4bafbf1ffaf8e039e P 1fab5794cf1740288abd82f2cebb9b1a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1fab5794cf1740288abd82f2cebb9b1a, State: Initialized, Role: LEARNER
I20240502 21:29:12.769358 5426 consensus_queue.cc:260] T c46e36b0bceb45b4bafbf1ffaf8e039e P 1fab5794cf1740288abd82f2cebb9b1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: NON_VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: true } }
I20240502 21:29:12.770067 5426 ts_tablet_manager.cc:1398] T c46e36b0bceb45b4bafbf1ffaf8e039e P 1fab5794cf1740288abd82f2cebb9b1a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:29:12.770706 2113 tablet_copy_service.cc:342] P 521bb93035324cecbebed2f6f2a630f8: Request end of tablet copy session 1fab5794cf1740288abd82f2cebb9b1a-c46e36b0bceb45b4bafbf1ffaf8e039e received from {username='slave'} at 127.15.255.138:37811
I20240502 21:29:12.770812 2113 tablet_copy_service.cc:433] P 521bb93035324cecbebed2f6f2a630f8: ending tablet copy session 1fab5794cf1740288abd82f2cebb9b1a-c46e36b0bceb45b4bafbf1ffaf8e039e on tablet c46e36b0bceb45b4bafbf1ffaf8e039e with peer 1fab5794cf1740288abd82f2cebb9b1a
I20240502 21:29:12.934538 5357 raft_consensus.cc:1050] T 444c1cba2a6f4729aa01b75d06b6b517 P e632c553ff4341328acf1cbaca678392: attempting to promote NON_VOTER d8d7297186694f77bb98de9174e4279c to VOTER
I20240502 21:29:12.935372 5357 consensus_queue.cc:237] T 444c1cba2a6f4729aa01b75d06b6b517 P e632c553ff4341328acf1cbaca678392 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } }
I20240502 21:29:12.938174 3856 raft_consensus.cc:1261] T 444c1cba2a6f4729aa01b75d06b6b517 P d8d7297186694f77bb98de9174e4279c [term 1 LEARNER]: Refusing update from remote peer e632c553ff4341328acf1cbaca678392: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:29:12.938325 1685 raft_consensus.cc:1261] T 444c1cba2a6f4729aa01b75d06b6b517 P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Refusing update from remote peer e632c553ff4341328acf1cbaca678392: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:29:12.939045 1414 raft_consensus.cc:1261] T 444c1cba2a6f4729aa01b75d06b6b517 P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Refusing update from remote peer e632c553ff4341328acf1cbaca678392: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:29:12.938947 5356 consensus_queue.cc:1035] T 444c1cba2a6f4729aa01b75d06b6b517 P e632c553ff4341328acf1cbaca678392 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:29:12.939442 1550 raft_consensus.cc:1261] T 444c1cba2a6f4729aa01b75d06b6b517 P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Refusing update from remote peer e632c553ff4341328acf1cbaca678392: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:29:12.939375 5452 consensus_queue.cc:1035] T 444c1cba2a6f4729aa01b75d06b6b517 P e632c553ff4341328acf1cbaca678392 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:29:12.939980 5452 consensus_queue.cc:1035] T 444c1cba2a6f4729aa01b75d06b6b517 P e632c553ff4341328acf1cbaca678392 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:29:12.940232 5356 consensus_queue.cc:1035] T 444c1cba2a6f4729aa01b75d06b6b517 P e632c553ff4341328acf1cbaca678392 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:29:12.941768 2226 raft_consensus.cc:1261] T 444c1cba2a6f4729aa01b75d06b6b517 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Refusing update from remote peer e632c553ff4341328acf1cbaca678392: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:29:12.942008 1818 raft_consensus.cc:1261] T 444c1cba2a6f4729aa01b75d06b6b517 P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Refusing update from remote peer e632c553ff4341328acf1cbaca678392: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:29:12.942330 5356 consensus_queue.cc:1035] T 444c1cba2a6f4729aa01b75d06b6b517 P e632c553ff4341328acf1cbaca678392 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:29:12.943938 5356 consensus_queue.cc:1035] T 444c1cba2a6f4729aa01b75d06b6b517 P e632c553ff4341328acf1cbaca678392 [LEADER]: Connected to new peer: Peer: permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:29:12.944891 2087 raft_consensus.cc:1261] T 444c1cba2a6f4729aa01b75d06b6b517 P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Refusing update from remote peer e632c553ff4341328acf1cbaca678392: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:29:12.952664 5356 consensus_queue.cc:1035] T 444c1cba2a6f4729aa01b75d06b6b517 P e632c553ff4341328acf1cbaca678392 [LEADER]: Connected to new peer: Peer: permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:29:12.960462 5357 raft_consensus.cc:2932] T 444c1cba2a6f4729aa01b75d06b6b517 P e632c553ff4341328acf1cbaca678392 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, d8d7297186694f77bb98de9174e4279c (127.15.255.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } }
I20240502 21:29:12.961398 1818 raft_consensus.cc:2932] T 444c1cba2a6f4729aa01b75d06b6b517 P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, d8d7297186694f77bb98de9174e4279c (127.15.255.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } }
I20240502 21:29:12.963630 1550 raft_consensus.cc:2932] T 444c1cba2a6f4729aa01b75d06b6b517 P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, d8d7297186694f77bb98de9174e4279c (127.15.255.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } }
I20240502 21:29:12.963930 1686 raft_consensus.cc:2932] T 444c1cba2a6f4729aa01b75d06b6b517 P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, d8d7297186694f77bb98de9174e4279c (127.15.255.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } }
I20240502 21:29:12.965883 3856 raft_consensus.cc:2932] T 444c1cba2a6f4729aa01b75d06b6b517 P d8d7297186694f77bb98de9174e4279c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, d8d7297186694f77bb98de9174e4279c (127.15.255.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } }
I20240502 21:29:12.969444 1266 catalog_manager.cc:5484] T 444c1cba2a6f4729aa01b75d06b6b517 P 16b3a337bc5e462f83ba895ad8660deb reported cstate change: config changed from index 5 to 6, d8d7297186694f77bb98de9174e4279c (127.15.255.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e632c553ff4341328acf1cbaca678392" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } }
I20240502 21:29:12.972661 1414 raft_consensus.cc:2932] T 444c1cba2a6f4729aa01b75d06b6b517 P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, d8d7297186694f77bb98de9174e4279c (127.15.255.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } }
I20240502 21:29:12.979483 2087 raft_consensus.cc:2932] T 444c1cba2a6f4729aa01b75d06b6b517 P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, d8d7297186694f77bb98de9174e4279c (127.15.255.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } }
I20240502 21:29:12.979719 2227 raft_consensus.cc:2932] T 444c1cba2a6f4729aa01b75d06b6b517 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, d8d7297186694f77bb98de9174e4279c (127.15.255.136) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } }
I20240502 21:29:12.991214 1953 consensus_queue.cc:237] T 444c1cba2a6f4729aa01b75d06b6b517 P e632c553ff4341328acf1cbaca678392 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } }
I20240502 21:29:12.994953 1686 raft_consensus.cc:1261] T 444c1cba2a6f4729aa01b75d06b6b517 P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Refusing update from remote peer e632c553ff4341328acf1cbaca678392: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:29:12.995280 2225 raft_consensus.cc:1261] T 444c1cba2a6f4729aa01b75d06b6b517 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Refusing update from remote peer e632c553ff4341328acf1cbaca678392: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:29:12.995523 5452 consensus_queue.cc:1035] T 444c1cba2a6f4729aa01b75d06b6b517 P e632c553ff4341328acf1cbaca678392 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:29:12.996392 5452 consensus_queue.cc:1035] T 444c1cba2a6f4729aa01b75d06b6b517 P e632c553ff4341328acf1cbaca678392 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:29:12.996522 1415 raft_consensus.cc:1261] T 444c1cba2a6f4729aa01b75d06b6b517 P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Refusing update from remote peer e632c553ff4341328acf1cbaca678392: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:29:12.997521 5454 consensus_queue.cc:1035] T 444c1cba2a6f4729aa01b75d06b6b517 P e632c553ff4341328acf1cbaca678392 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:29:12.998279 3856 raft_consensus.cc:1261] T 444c1cba2a6f4729aa01b75d06b6b517 P d8d7297186694f77bb98de9174e4279c [term 1 FOLLOWER]: Refusing update from remote peer e632c553ff4341328acf1cbaca678392: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:29:12.998862 5454 consensus_queue.cc:1035] T 444c1cba2a6f4729aa01b75d06b6b517 P e632c553ff4341328acf1cbaca678392 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:29:12.999442 1550 raft_consensus.cc:1261] T 444c1cba2a6f4729aa01b75d06b6b517 P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Refusing update from remote peer e632c553ff4341328acf1cbaca678392: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:29:13.000108 5463 consensus_queue.cc:1035] T 444c1cba2a6f4729aa01b75d06b6b517 P e632c553ff4341328acf1cbaca678392 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:29:13.003559 5452 raft_consensus.cc:2932] T 444c1cba2a6f4729aa01b75d06b6b517 P e632c553ff4341328acf1cbaca678392 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } }
I20240502 21:29:13.006919 2087 raft_consensus.cc:1261] T 444c1cba2a6f4729aa01b75d06b6b517 P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Refusing update from remote peer e632c553ff4341328acf1cbaca678392: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:29:13.007515 1254 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 444c1cba2a6f4729aa01b75d06b6b517 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:29:13.008188 5454 consensus_queue.cc:1035] T 444c1cba2a6f4729aa01b75d06b6b517 P e632c553ff4341328acf1cbaca678392 [LEADER]: Connected to new peer: Peer: permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:29:13.009449 3856 raft_consensus.cc:2932] T 444c1cba2a6f4729aa01b75d06b6b517 P d8d7297186694f77bb98de9174e4279c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } }
I20240502 21:29:13.011431 2088 raft_consensus.cc:2932] T 444c1cba2a6f4729aa01b75d06b6b517 P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } }
I20240502 21:29:13.015792 1686 raft_consensus.cc:2932] T 444c1cba2a6f4729aa01b75d06b6b517 P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } }
I20240502 21:29:13.016034 1268 catalog_manager.cc:5484] T 444c1cba2a6f4729aa01b75d06b6b517 P d8d7297186694f77bb98de9174e4279c reported cstate change: config changed from index 6 to 7, VOTER 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132) evicted. New cstate: current_term: 1 leader_uuid: "e632c553ff4341328acf1cbaca678392" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } }
I20240502 21:29:13.022372 1415 raft_consensus.cc:2932] T 444c1cba2a6f4729aa01b75d06b6b517 P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } }
I20240502 21:29:13.022539 2225 raft_consensus.cc:2932] T 444c1cba2a6f4729aa01b75d06b6b517 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } }
I20240502 21:29:13.036069 1798 tablet_service.cc:1514] Processing DeleteTablet for tablet 444c1cba2a6f4729aa01b75d06b6b517 with delete_type TABLET_DATA_TOMBSTONED (TS 153026dd7f8943cc93fedf6f48450fb4 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:60256
I20240502 21:29:13.036486 5427 tablet_replica.cc:330] T 444c1cba2a6f4729aa01b75d06b6b517 P 153026dd7f8943cc93fedf6f48450fb4: stopping tablet replica
I20240502 21:29:13.030838 1550 raft_consensus.cc:2932] T 444c1cba2a6f4729aa01b75d06b6b517 P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } }
I20240502 21:29:13.036677 5427 raft_consensus.cc:2229] T 444c1cba2a6f4729aa01b75d06b6b517 P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:29:13.037101 5427 raft_consensus.cc:2258] T 444c1cba2a6f4729aa01b75d06b6b517 P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:29:13.038152 5427 ts_tablet_manager.cc:1875] T 444c1cba2a6f4729aa01b75d06b6b517 P 153026dd7f8943cc93fedf6f48450fb4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:29:13.043115 5427 ts_tablet_manager.cc:1888] T 444c1cba2a6f4729aa01b75d06b6b517 P 153026dd7f8943cc93fedf6f48450fb4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240502 21:29:13.043211 5427 log.cc:1196] T 444c1cba2a6f4729aa01b75d06b6b517 P 153026dd7f8943cc93fedf6f48450fb4: Deleting WAL directory at /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-3/wal/wals/444c1cba2a6f4729aa01b75d06b6b517
I20240502 21:29:13.043862 1256 catalog_manager.cc:4830] TS 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132:44947): tablet 444c1cba2a6f4729aa01b75d06b6b517 (table rebalance_test_table_1 [id=f2592b36bf814d51bd833a6ad315e513]) successfully deleted
I20240502 21:29:13.064986 2225 consensus_queue.cc:237] T 05e885ae741c43808c6dfbcdc5769942 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } }
I20240502 21:29:13.067459 1953 raft_consensus.cc:1261] T 05e885ae741c43808c6dfbcdc5769942 P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Refusing update from remote peer 6ab8df3c796f4b6cb3e29d5045ad3ea0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:29:13.067580 2088 raft_consensus.cc:1261] T 05e885ae741c43808c6dfbcdc5769942 P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Refusing update from remote peer 6ab8df3c796f4b6cb3e29d5045ad3ea0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:29:13.068343 1818 raft_consensus.cc:1261] T 05e885ae741c43808c6dfbcdc5769942 P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Refusing update from remote peer 6ab8df3c796f4b6cb3e29d5045ad3ea0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:29:13.068544 5449 consensus_queue.cc:1035] T 05e885ae741c43808c6dfbcdc5769942 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:13.069139 5449 consensus_queue.cc:1035] T 05e885ae741c43808c6dfbcdc5769942 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:13.069602 5449 consensus_queue.cc:1035] T 05e885ae741c43808c6dfbcdc5769942 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:13.074283 1686 raft_consensus.cc:1261] T 05e885ae741c43808c6dfbcdc5769942 P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Refusing update from remote peer 6ab8df3c796f4b6cb3e29d5045ad3ea0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:29:13.074970 5152 consensus_queue.cc:1035] T 05e885ae741c43808c6dfbcdc5769942 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:13.079600 1550 raft_consensus.cc:1261] T 05e885ae741c43808c6dfbcdc5769942 P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Refusing update from remote peer 6ab8df3c796f4b6cb3e29d5045ad3ea0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:29:13.080197 1415 raft_consensus.cc:1261] T 05e885ae741c43808c6dfbcdc5769942 P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Refusing update from remote peer 6ab8df3c796f4b6cb3e29d5045ad3ea0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:29:13.080881 5152 consensus_queue.cc:1035] T 05e885ae741c43808c6dfbcdc5769942 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:13.081385 5152 consensus_queue.cc:1035] T 05e885ae741c43808c6dfbcdc5769942 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:13.081986 5152 raft_consensus.cc:2932] T 05e885ae741c43808c6dfbcdc5769942 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } }
I20240502 21:29:13.082490 1955 raft_consensus.cc:2932] T 05e885ae741c43808c6dfbcdc5769942 P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } }
I20240502 21:29:13.086117 1818 raft_consensus.cc:2932] T 05e885ae741c43808c6dfbcdc5769942 P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } }
I20240502 21:29:13.086230 1686 raft_consensus.cc:2932] T 05e885ae741c43808c6dfbcdc5769942 P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } }
I20240502 21:29:13.088301 1268 catalog_manager.cc:5484] T 05e885ae741c43808c6dfbcdc5769942 P e632c553ff4341328acf1cbaca678392 reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } }
I20240502 21:29:13.093886 1415 raft_consensus.cc:2932] T 05e885ae741c43808c6dfbcdc5769942 P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } }
I20240502 21:29:13.109551 1549 raft_consensus.cc:2932] T 05e885ae741c43808c6dfbcdc5769942 P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } }
W20240502 21:29:13.115404 5477 consensus_queue.cc:1159] T 05e885ae741c43808c6dfbcdc5769942 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "a62a935ff43d494ca35d0c417943133d" responder_term: 1 status { last_received { term: 1 index: 4 } last_committed_idx: 4 last_received_current_leader { term: 1 index: 4 } }
I20240502 21:29:13.116024 2225 consensus_queue.cc:237] T 05e885ae741c43808c6dfbcdc5769942 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: NON_VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: true } }
I20240502 21:29:13.119938 1818 raft_consensus.cc:1261] T 05e885ae741c43808c6dfbcdc5769942 P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Refusing update from remote peer 6ab8df3c796f4b6cb3e29d5045ad3ea0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:29:13.119952 1415 raft_consensus.cc:1261] T 05e885ae741c43808c6dfbcdc5769942 P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Refusing update from remote peer 6ab8df3c796f4b6cb3e29d5045ad3ea0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:29:13.120468 5449 consensus_queue.cc:1035] T 05e885ae741c43808c6dfbcdc5769942 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:29:13.121043 1686 raft_consensus.cc:1261] T 05e885ae741c43808c6dfbcdc5769942 P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Refusing update from remote peer 6ab8df3c796f4b6cb3e29d5045ad3ea0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:29:13.121529 1549 raft_consensus.cc:1261] T 05e885ae741c43808c6dfbcdc5769942 P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Refusing update from remote peer 6ab8df3c796f4b6cb3e29d5045ad3ea0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:29:13.122198 5152 consensus_queue.cc:1035] T 05e885ae741c43808c6dfbcdc5769942 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:29:13.122489 5477 consensus_queue.cc:1035] T 05e885ae741c43808c6dfbcdc5769942 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:29:13.122921 5449 consensus_queue.cc:1035] T 05e885ae741c43808c6dfbcdc5769942 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:29:13.123512 1953 raft_consensus.cc:1261] T 05e885ae741c43808c6dfbcdc5769942 P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Refusing update from remote peer 6ab8df3c796f4b6cb3e29d5045ad3ea0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:29:13.124392 2088 raft_consensus.cc:1261] T 05e885ae741c43808c6dfbcdc5769942 P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Refusing update from remote peer 6ab8df3c796f4b6cb3e29d5045ad3ea0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:29:13.126315 5449 consensus_queue.cc:1035] T 05e885ae741c43808c6dfbcdc5769942 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:13.126590 5476 consensus_queue.cc:1035] T 05e885ae741c43808c6dfbcdc5769942 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:29:13.126282 5477 raft_consensus.cc:2932] T 05e885ae741c43808c6dfbcdc5769942 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 16a97c1394f346f0b987f97cce7371b6 (127.15.255.137) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: NON_VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: true } } }
W20240502 21:29:13.127305 2158 consensus_peers.cc:485] T 05e885ae741c43808c6dfbcdc5769942 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 -> Peer 16a97c1394f346f0b987f97cce7371b6 (127.15.255.137:43717): Couldn't send request to peer 16a97c1394f346f0b987f97cce7371b6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 05e885ae741c43808c6dfbcdc5769942. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:29:13.127313 1549 raft_consensus.cc:2932] T 05e885ae741c43808c6dfbcdc5769942 P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 16a97c1394f346f0b987f97cce7371b6 (127.15.255.137) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: NON_VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: true } } }
I20240502 21:29:13.128847 2088 raft_consensus.cc:2932] T 05e885ae741c43808c6dfbcdc5769942 P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } }
I20240502 21:29:13.131083 1255 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 05e885ae741c43808c6dfbcdc5769942 with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240502 21:29:13.132462 1686 raft_consensus.cc:2932] T 05e885ae741c43808c6dfbcdc5769942 P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 16a97c1394f346f0b987f97cce7371b6 (127.15.255.137) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: NON_VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: true } } }
I20240502 21:29:13.133572 1266 catalog_manager.cc:5484] T 05e885ae741c43808c6dfbcdc5769942 P a62a935ff43d494ca35d0c417943133d reported cstate change: config changed from index 4 to 5, NON_VOTER 16a97c1394f346f0b987f97cce7371b6 (127.15.255.137) added. New cstate: current_term: 1 leader_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: NON_VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: true } } }
I20240502 21:29:13.133790 2088 raft_consensus.cc:2932] T 05e885ae741c43808c6dfbcdc5769942 P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 16a97c1394f346f0b987f97cce7371b6 (127.15.255.137) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: NON_VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: true } } }
I20240502 21:29:13.138706 1415 raft_consensus.cc:2932] T 05e885ae741c43808c6dfbcdc5769942 P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 16a97c1394f346f0b987f97cce7371b6 (127.15.255.137) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: NON_VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: true } } }
I20240502 21:29:13.140149 1953 raft_consensus.cc:2932] T 05e885ae741c43808c6dfbcdc5769942 P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 16a97c1394f346f0b987f97cce7371b6 (127.15.255.137) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: NON_VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: true } } }
I20240502 21:29:13.146031 1820 raft_consensus.cc:2932] T 05e885ae741c43808c6dfbcdc5769942 P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 16a97c1394f346f0b987f97cce7371b6 (127.15.255.137) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: NON_VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: true } } }
I20240502 21:29:13.279054 4121 raft_consensus.cc:1203] T c46e36b0bceb45b4bafbf1ffaf8e039e P 1fab5794cf1740288abd82f2cebb9b1a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240502 21:29:13.381731 1414 consensus_queue.cc:237] T aae7b5a01a0543afb1b5a7f44a9a864c P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } }
I20240502 21:29:13.384620 2088 raft_consensus.cc:1261] T aae7b5a01a0543afb1b5a7f44a9a864c P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:29:13.384713 1953 raft_consensus.cc:1261] T aae7b5a01a0543afb1b5a7f44a9a864c P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:29:13.385144 5507 consensus_queue.cc:1035] T aae7b5a01a0543afb1b5a7f44a9a864c P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:13.385342 2225 raft_consensus.cc:1261] T aae7b5a01a0543afb1b5a7f44a9a864c P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:29:13.385376 1686 raft_consensus.cc:1261] T aae7b5a01a0543afb1b5a7f44a9a864c P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:29:13.385733 1818 raft_consensus.cc:1261] T aae7b5a01a0543afb1b5a7f44a9a864c P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:29:13.385813 1549 raft_consensus.cc:1261] T aae7b5a01a0543afb1b5a7f44a9a864c P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 21:29:13.386477 5466 consensus_queue.cc:1035] T aae7b5a01a0543afb1b5a7f44a9a864c P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:13.386749 4889 consensus_queue.cc:1035] T aae7b5a01a0543afb1b5a7f44a9a864c P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:13.387346 4889 consensus_queue.cc:1035] T aae7b5a01a0543afb1b5a7f44a9a864c P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:13.387643 5466 consensus_queue.cc:1035] T aae7b5a01a0543afb1b5a7f44a9a864c P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:13.388224 5466 consensus_queue.cc:1035] T aae7b5a01a0543afb1b5a7f44a9a864c P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:13.392086 4889 raft_consensus.cc:2932] T aae7b5a01a0543afb1b5a7f44a9a864c P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } }
I20240502 21:29:13.392859 1818 raft_consensus.cc:2932] T aae7b5a01a0543afb1b5a7f44a9a864c P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } }
I20240502 21:29:13.393297 1549 raft_consensus.cc:2932] T aae7b5a01a0543afb1b5a7f44a9a864c P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } }
I20240502 21:29:13.399163 2088 raft_consensus.cc:2932] T aae7b5a01a0543afb1b5a7f44a9a864c P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } }
I20240502 21:29:13.399632 1266 catalog_manager.cc:5484] T aae7b5a01a0543afb1b5a7f44a9a864c P 7d1e5c1b047443cbb28d7b0a9744babf reported cstate change: config changed from index -1 to 4. New cstate: current_term: 1 leader_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } health_report { overall_health: UNKNOWN } } }
I20240502 21:29:13.405417 1686 raft_consensus.cc:2932] T aae7b5a01a0543afb1b5a7f44a9a864c P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } }
I20240502 21:29:13.414314 1414 consensus_queue.cc:237] T aae7b5a01a0543afb1b5a7f44a9a864c P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: NON_VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: true } }
I20240502 21:29:13.417209 1686 raft_consensus.cc:1261] T aae7b5a01a0543afb1b5a7f44a9a864c P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:29:13.422215 1818 raft_consensus.cc:1261] T aae7b5a01a0543afb1b5a7f44a9a864c P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:29:13.422811 5508 consensus_queue.cc:1035] T aae7b5a01a0543afb1b5a7f44a9a864c P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:29:13.423574 1549 raft_consensus.cc:1261] T aae7b5a01a0543afb1b5a7f44a9a864c P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:29:13.424147 5507 consensus_queue.cc:1035] T aae7b5a01a0543afb1b5a7f44a9a864c P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:29:13.432627 5507 consensus_queue.cc:1035] T aae7b5a01a0543afb1b5a7f44a9a864c P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 21:29:13.433991 2227 raft_consensus.cc:1261] T aae7b5a01a0543afb1b5a7f44a9a864c P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:29:13.434710 5507 consensus_queue.cc:1035] T aae7b5a01a0543afb1b5a7f44a9a864c P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:13.435895 2227 raft_consensus.cc:2932] T aae7b5a01a0543afb1b5a7f44a9a864c P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } }
I20240502 21:29:13.437479 1955 raft_consensus.cc:1261] T aae7b5a01a0543afb1b5a7f44a9a864c P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:29:13.438442 5464 consensus_queue.cc:1035] T aae7b5a01a0543afb1b5a7f44a9a864c P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 21:29:13.438866 5464 raft_consensus.cc:2932] T aae7b5a01a0543afb1b5a7f44a9a864c P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: NON_VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: true } } }
I20240502 21:29:13.441355 2225 raft_consensus.cc:2932] T aae7b5a01a0543afb1b5a7f44a9a864c P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: NON_VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: true } } }
I20240502 21:29:13.443006 1255 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet aae7b5a01a0543afb1b5a7f44a9a864c with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240502 21:29:13.444188 2088 raft_consensus.cc:1261] T aae7b5a01a0543afb1b5a7f44a9a864c P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 21:29:13.444391 1955 raft_consensus.cc:2932] T aae7b5a01a0543afb1b5a7f44a9a864c P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } }
I20240502 21:29:13.447434 4889 consensus_queue.cc:1035] T aae7b5a01a0543afb1b5a7f44a9a864c P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240502 21:29:13.449275 1350 consensus_peers.cc:485] T aae7b5a01a0543afb1b5a7f44a9a864c P 7d1e5c1b047443cbb28d7b0a9744babf -> Peer 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138:33375): Couldn't send request to peer 1fab5794cf1740288abd82f2cebb9b1a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: aae7b5a01a0543afb1b5a7f44a9a864c. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:29:13.449026 1266 catalog_manager.cc:5484] T aae7b5a01a0543afb1b5a7f44a9a864c P 6ab8df3c796f4b6cb3e29d5045ad3ea0 reported cstate change: config changed from index 4 to 5, NON_VOTER 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138) added. New cstate: current_term: 1 leader_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: NON_VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: true } } }
I20240502 21:29:13.449343 2088 raft_consensus.cc:2932] T aae7b5a01a0543afb1b5a7f44a9a864c P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: NON_VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: true } } }
I20240502 21:29:13.454818 1820 raft_consensus.cc:2932] T aae7b5a01a0543afb1b5a7f44a9a864c P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: NON_VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: true } } }
I20240502 21:29:13.455621 1686 raft_consensus.cc:2932] T aae7b5a01a0543afb1b5a7f44a9a864c P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: NON_VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: true } } }
I20240502 21:29:13.461079 1955 raft_consensus.cc:2932] T aae7b5a01a0543afb1b5a7f44a9a864c P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: NON_VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: true } } }
I20240502 21:29:13.463514 1550 raft_consensus.cc:2932] T aae7b5a01a0543afb1b5a7f44a9a864c P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: NON_VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: true } } }
I20240502 21:29:13.548838 5537 ts_tablet_manager.cc:902] T 05e885ae741c43808c6dfbcdc5769942 P 16a97c1394f346f0b987f97cce7371b6: Initiating tablet copy from peer 6ab8df3c796f4b6cb3e29d5045ad3ea0 (127.15.255.135:34701)
I20240502 21:29:13.551167 5537 tablet_copy_client.cc:308] T 05e885ae741c43808c6dfbcdc5769942 P 16a97c1394f346f0b987f97cce7371b6: tablet copy: Beginning tablet copy session from remote peer at address 127.15.255.135:34701
I20240502 21:29:13.551990 2249 tablet_copy_service.cc:140] P 6ab8df3c796f4b6cb3e29d5045ad3ea0: Received BeginTabletCopySession request for tablet 05e885ae741c43808c6dfbcdc5769942 from peer 16a97c1394f346f0b987f97cce7371b6 ({username='slave'} at 127.15.255.137:35291)
I20240502 21:29:13.552166 2249 tablet_copy_service.cc:161] P 6ab8df3c796f4b6cb3e29d5045ad3ea0: Beginning new tablet copy session on tablet 05e885ae741c43808c6dfbcdc5769942 from peer 16a97c1394f346f0b987f97cce7371b6 at {username='slave'} at 127.15.255.137:35291: session id = 16a97c1394f346f0b987f97cce7371b6-05e885ae741c43808c6dfbcdc5769942
I20240502 21:29:13.555234 2249 tablet_copy_source_session.cc:205] T 05e885ae741c43808c6dfbcdc5769942 P 6ab8df3c796f4b6cb3e29d5045ad3ea0: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:29:13.556979 5537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05e885ae741c43808c6dfbcdc5769942. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:29:13.561827 5537 tablet_copy_client.cc:783] T 05e885ae741c43808c6dfbcdc5769942 P 16a97c1394f346f0b987f97cce7371b6: tablet copy: Starting download of 0 data blocks...
I20240502 21:29:13.562075 5537 tablet_copy_client.cc:647] T 05e885ae741c43808c6dfbcdc5769942 P 16a97c1394f346f0b987f97cce7371b6: tablet copy: Starting download of 1 WAL segments...
I20240502 21:29:13.563773 5537 tablet_copy_client.cc:523] T 05e885ae741c43808c6dfbcdc5769942 P 16a97c1394f346f0b987f97cce7371b6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:29:13.566335 5537 tablet_bootstrap.cc:492] T 05e885ae741c43808c6dfbcdc5769942 P 16a97c1394f346f0b987f97cce7371b6: Bootstrap starting.
I20240502 21:29:13.603183 5537 tablet_bootstrap.cc:492] T 05e885ae741c43808c6dfbcdc5769942 P 16a97c1394f346f0b987f97cce7371b6: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:29:13.603616 5537 tablet_bootstrap.cc:492] T 05e885ae741c43808c6dfbcdc5769942 P 16a97c1394f346f0b987f97cce7371b6: Bootstrap complete.
I20240502 21:29:13.603868 5537 ts_tablet_manager.cc:1367] T 05e885ae741c43808c6dfbcdc5769942 P 16a97c1394f346f0b987f97cce7371b6: Time spent bootstrapping tablet: real 0.038s user 0.030s sys 0.004s
I20240502 21:29:13.604887 5537 raft_consensus.cc:348] T 05e885ae741c43808c6dfbcdc5769942 P 16a97c1394f346f0b987f97cce7371b6 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: NON_VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: true } }
I20240502 21:29:13.605180 5537 raft_consensus.cc:729] T 05e885ae741c43808c6dfbcdc5769942 P 16a97c1394f346f0b987f97cce7371b6 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 16a97c1394f346f0b987f97cce7371b6, State: Initialized, Role: LEARNER
I20240502 21:29:13.605474 5537 consensus_queue.cc:260] T 05e885ae741c43808c6dfbcdc5769942 P 16a97c1394f346f0b987f97cce7371b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: NON_VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: true } }
I20240502 21:29:13.606262 5537 ts_tablet_manager.cc:1398] T 05e885ae741c43808c6dfbcdc5769942 P 16a97c1394f346f0b987f97cce7371b6: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 21:29:13.607039 2249 tablet_copy_service.cc:342] P 6ab8df3c796f4b6cb3e29d5045ad3ea0: Request end of tablet copy session 16a97c1394f346f0b987f97cce7371b6-05e885ae741c43808c6dfbcdc5769942 received from {username='slave'} at 127.15.255.137:35291
I20240502 21:29:13.607195 2249 tablet_copy_service.cc:433] P 6ab8df3c796f4b6cb3e29d5045ad3ea0: ending tablet copy session 16a97c1394f346f0b987f97cce7371b6-05e885ae741c43808c6dfbcdc5769942 on tablet 05e885ae741c43808c6dfbcdc5769942 with peer 16a97c1394f346f0b987f97cce7371b6
I20240502 21:29:13.824190 5389 raft_consensus.cc:1050] T c46e36b0bceb45b4bafbf1ffaf8e039e P 521bb93035324cecbebed2f6f2a630f8: attempting to promote NON_VOTER 1fab5794cf1740288abd82f2cebb9b1a to VOTER
I20240502 21:29:13.825148 5389 consensus_queue.cc:237] T c46e36b0bceb45b4bafbf1ffaf8e039e P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } }
I20240502 21:29:13.828230 1414 raft_consensus.cc:1261] T c46e36b0bceb45b4bafbf1ffaf8e039e P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:29:13.828285 4121 raft_consensus.cc:1261] T c46e36b0bceb45b4bafbf1ffaf8e039e P 1fab5794cf1740288abd82f2cebb9b1a [term 1 LEARNER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:29:13.828346 1686 raft_consensus.cc:1261] T c46e36b0bceb45b4bafbf1ffaf8e039e P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:29:13.829331 1955 raft_consensus.cc:1261] T c46e36b0bceb45b4bafbf1ffaf8e039e P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:29:13.829394 1820 raft_consensus.cc:1261] T c46e36b0bceb45b4bafbf1ffaf8e039e P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:29:13.829421 2225 raft_consensus.cc:1261] T c46e36b0bceb45b4bafbf1ffaf8e039e P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:29:13.830482 5389 consensus_queue.cc:1035] T c46e36b0bceb45b4bafbf1ffaf8e039e P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20240502 21:29:13.830852 5498 consensus_queue.cc:1035] T c46e36b0bceb45b4bafbf1ffaf8e039e P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:29:13.831023 1550 raft_consensus.cc:1261] T c46e36b0bceb45b4bafbf1ffaf8e039e P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:29:13.831748 5561 consensus_queue.cc:1035] T c46e36b0bceb45b4bafbf1ffaf8e039e P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:29:13.832137 5339 consensus_queue.cc:1035] T c46e36b0bceb45b4bafbf1ffaf8e039e P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:29:13.832885 5339 consensus_queue.cc:1035] T c46e36b0bceb45b4bafbf1ffaf8e039e P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:29:13.833334 5498 consensus_queue.cc:1035] T c46e36b0bceb45b4bafbf1ffaf8e039e P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:29:13.833938 5497 consensus_queue.cc:1035] T c46e36b0bceb45b4bafbf1ffaf8e039e P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:29:13.866204 5497 raft_consensus.cc:2932] T c46e36b0bceb45b4bafbf1ffaf8e039e P 521bb93035324cecbebed2f6f2a630f8 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } }
I20240502 21:29:13.866751 4121 raft_consensus.cc:2932] T c46e36b0bceb45b4bafbf1ffaf8e039e P 1fab5794cf1740288abd82f2cebb9b1a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } }
I20240502 21:29:13.871958 1686 raft_consensus.cc:2932] T c46e36b0bceb45b4bafbf1ffaf8e039e P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } }
I20240502 21:29:13.873306 1820 raft_consensus.cc:2932] T c46e36b0bceb45b4bafbf1ffaf8e039e P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } }
I20240502 21:29:13.875706 1955 raft_consensus.cc:2932] T c46e36b0bceb45b4bafbf1ffaf8e039e P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } }
I20240502 21:29:13.874771 1550 raft_consensus.cc:2932] T c46e36b0bceb45b4bafbf1ffaf8e039e P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } }
I20240502 21:29:13.879173 2225 raft_consensus.cc:2932] T c46e36b0bceb45b4bafbf1ffaf8e039e P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } }
I20240502 21:29:13.875149 1268 catalog_manager.cc:5484] T c46e36b0bceb45b4bafbf1ffaf8e039e P 521bb93035324cecbebed2f6f2a630f8 reported cstate change: config changed from index 5 to 6, 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "521bb93035324cecbebed2f6f2a630f8" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:29:13.889958 1415 raft_consensus.cc:2932] T c46e36b0bceb45b4bafbf1ffaf8e039e P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } }
I20240502 21:29:13.898517 2088 consensus_queue.cc:237] T c46e36b0bceb45b4bafbf1ffaf8e039e P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } }
I20240502 21:29:13.906172 1685 raft_consensus.cc:1261] T c46e36b0bceb45b4bafbf1ffaf8e039e P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:29:13.907584 4121 raft_consensus.cc:1261] T c46e36b0bceb45b4bafbf1ffaf8e039e P 1fab5794cf1740288abd82f2cebb9b1a [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:29:13.908264 5339 consensus_queue.cc:1035] T c46e36b0bceb45b4bafbf1ffaf8e039e P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:29:13.909600 1549 raft_consensus.cc:1261] T c46e36b0bceb45b4bafbf1ffaf8e039e P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:29:13.910622 2227 raft_consensus.cc:1261] T c46e36b0bceb45b4bafbf1ffaf8e039e P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:29:13.910766 1415 raft_consensus.cc:1261] T c46e36b0bceb45b4bafbf1ffaf8e039e P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:29:13.911041 5389 consensus_queue.cc:1035] T c46e36b0bceb45b4bafbf1ffaf8e039e P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:29:13.911885 5339 consensus_queue.cc:1035] T c46e36b0bceb45b4bafbf1ffaf8e039e P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:29:13.912132 5389 consensus_queue.cc:1035] T c46e36b0bceb45b4bafbf1ffaf8e039e P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:29:13.912375 5562 consensus_queue.cc:1035] T c46e36b0bceb45b4bafbf1ffaf8e039e P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:29:13.913479 1955 raft_consensus.cc:1261] T c46e36b0bceb45b4bafbf1ffaf8e039e P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:29:13.914000 5389 consensus_queue.cc:1035] T c46e36b0bceb45b4bafbf1ffaf8e039e P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:29:13.917189 5562 raft_consensus.cc:2932] T c46e36b0bceb45b4bafbf1ffaf8e039e P 521bb93035324cecbebed2f6f2a630f8 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } }
I20240502 21:29:13.917802 2225 raft_consensus.cc:2932] T c46e36b0bceb45b4bafbf1ffaf8e039e P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } }
I20240502 21:29:13.918259 1953 raft_consensus.cc:2932] T c46e36b0bceb45b4bafbf1ffaf8e039e P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } }
I20240502 21:29:13.923807 1685 raft_consensus.cc:2932] T c46e36b0bceb45b4bafbf1ffaf8e039e P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } }
I20240502 21:29:13.924278 4120 raft_consensus.cc:2932] T c46e36b0bceb45b4bafbf1ffaf8e039e P 1fab5794cf1740288abd82f2cebb9b1a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } }
I20240502 21:29:13.928371 1268 catalog_manager.cc:5484] T c46e36b0bceb45b4bafbf1ffaf8e039e P 6ab8df3c796f4b6cb3e29d5045ad3ea0 reported cstate change: config changed from index 6 to 7, VOTER 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132) evicted. New cstate: current_term: 1 leader_uuid: "521bb93035324cecbebed2f6f2a630f8" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } }
I20240502 21:29:13.932387 1798 tablet_service.cc:1514] Processing DeleteTablet for tablet c46e36b0bceb45b4bafbf1ffaf8e039e with delete_type TABLET_DATA_TOMBSTONED (TS 153026dd7f8943cc93fedf6f48450fb4 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:60256
I20240502 21:29:13.933261 5572 tablet_replica.cc:330] T c46e36b0bceb45b4bafbf1ffaf8e039e P 153026dd7f8943cc93fedf6f48450fb4: stopping tablet replica
I20240502 21:29:13.933584 5572 raft_consensus.cc:2229] T c46e36b0bceb45b4bafbf1ffaf8e039e P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:29:13.933856 5572 raft_consensus.cc:2258] T c46e36b0bceb45b4bafbf1ffaf8e039e P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:29:13.934800 5572 ts_tablet_manager.cc:1875] T c46e36b0bceb45b4bafbf1ffaf8e039e P 153026dd7f8943cc93fedf6f48450fb4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:29:13.934861 1254 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c46e36b0bceb45b4bafbf1ffaf8e039e with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:29:13.939080 1550 raft_consensus.cc:2932] T c46e36b0bceb45b4bafbf1ffaf8e039e P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } }
I20240502 21:29:13.938251 1415 raft_consensus.cc:2932] T c46e36b0bceb45b4bafbf1ffaf8e039e P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } }
I20240502 21:29:13.941502 5574 ts_tablet_manager.cc:902] T aae7b5a01a0543afb1b5a7f44a9a864c P 1fab5794cf1740288abd82f2cebb9b1a: Initiating tablet copy from peer 7d1e5c1b047443cbb28d7b0a9744babf (127.15.255.129:37897)
I20240502 21:29:13.948087 5572 ts_tablet_manager.cc:1888] T c46e36b0bceb45b4bafbf1ffaf8e039e P 153026dd7f8943cc93fedf6f48450fb4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240502 21:29:13.948239 5572 log.cc:1196] T c46e36b0bceb45b4bafbf1ffaf8e039e P 153026dd7f8943cc93fedf6f48450fb4: Deleting WAL directory at /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-3/wal/wals/c46e36b0bceb45b4bafbf1ffaf8e039e
I20240502 21:29:13.949528 1256 catalog_manager.cc:4830] TS 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132:44947): tablet c46e36b0bceb45b4bafbf1ffaf8e039e (table rebalance_test_table_4 [id=d523de924ba14cb493e559296c083a34]) successfully deleted
I20240502 21:29:13.950943 5574 tablet_copy_client.cc:308] T aae7b5a01a0543afb1b5a7f44a9a864c P 1fab5794cf1740288abd82f2cebb9b1a: tablet copy: Beginning tablet copy session from remote peer at address 127.15.255.129:37897
I20240502 21:29:13.957923 1435 tablet_copy_service.cc:140] P 7d1e5c1b047443cbb28d7b0a9744babf: Received BeginTabletCopySession request for tablet aae7b5a01a0543afb1b5a7f44a9a864c from peer 1fab5794cf1740288abd82f2cebb9b1a ({username='slave'} at 127.15.255.138:36825)
I20240502 21:29:13.958079 1435 tablet_copy_service.cc:161] P 7d1e5c1b047443cbb28d7b0a9744babf: Beginning new tablet copy session on tablet aae7b5a01a0543afb1b5a7f44a9a864c from peer 1fab5794cf1740288abd82f2cebb9b1a at {username='slave'} at 127.15.255.138:36825: session id = 1fab5794cf1740288abd82f2cebb9b1a-aae7b5a01a0543afb1b5a7f44a9a864c
I20240502 21:29:13.961223 1435 tablet_copy_source_session.cc:205] T aae7b5a01a0543afb1b5a7f44a9a864c P 7d1e5c1b047443cbb28d7b0a9744babf: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:29:13.964967 5574 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aae7b5a01a0543afb1b5a7f44a9a864c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:29:13.971562 5574 tablet_copy_client.cc:783] T aae7b5a01a0543afb1b5a7f44a9a864c P 1fab5794cf1740288abd82f2cebb9b1a: tablet copy: Starting download of 0 data blocks...
I20240502 21:29:13.971912 5574 tablet_copy_client.cc:647] T aae7b5a01a0543afb1b5a7f44a9a864c P 1fab5794cf1740288abd82f2cebb9b1a: tablet copy: Starting download of 1 WAL segments...
I20240502 21:29:13.973520 5574 tablet_copy_client.cc:523] T aae7b5a01a0543afb1b5a7f44a9a864c P 1fab5794cf1740288abd82f2cebb9b1a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:29:13.976843 5574 tablet_bootstrap.cc:492] T aae7b5a01a0543afb1b5a7f44a9a864c P 1fab5794cf1740288abd82f2cebb9b1a: Bootstrap starting.
I20240502 21:29:14.009644 3987 raft_consensus.cc:1203] T 05e885ae741c43808c6dfbcdc5769942 P 16a97c1394f346f0b987f97cce7371b6 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240502 21:29:14.033098 5574 tablet_bootstrap.cc:492] T aae7b5a01a0543afb1b5a7f44a9a864c P 1fab5794cf1740288abd82f2cebb9b1a: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:29:14.033531 5574 tablet_bootstrap.cc:492] T aae7b5a01a0543afb1b5a7f44a9a864c P 1fab5794cf1740288abd82f2cebb9b1a: Bootstrap complete.
I20240502 21:29:14.033821 5574 ts_tablet_manager.cc:1367] T aae7b5a01a0543afb1b5a7f44a9a864c P 1fab5794cf1740288abd82f2cebb9b1a: Time spent bootstrapping tablet: real 0.057s user 0.043s sys 0.008s
I20240502 21:29:14.034603 5574 raft_consensus.cc:348] T aae7b5a01a0543afb1b5a7f44a9a864c P 1fab5794cf1740288abd82f2cebb9b1a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: NON_VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: true } }
I20240502 21:29:14.034935 5574 raft_consensus.cc:729] T aae7b5a01a0543afb1b5a7f44a9a864c P 1fab5794cf1740288abd82f2cebb9b1a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1fab5794cf1740288abd82f2cebb9b1a, State: Initialized, Role: LEARNER
I20240502 21:29:14.035197 5574 consensus_queue.cc:260] T aae7b5a01a0543afb1b5a7f44a9a864c P 1fab5794cf1740288abd82f2cebb9b1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: NON_VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: true } }
I20240502 21:29:14.035883 5574 ts_tablet_manager.cc:1398] T aae7b5a01a0543afb1b5a7f44a9a864c P 1fab5794cf1740288abd82f2cebb9b1a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:29:14.038339 1435 tablet_copy_service.cc:342] P 7d1e5c1b047443cbb28d7b0a9744babf: Request end of tablet copy session 1fab5794cf1740288abd82f2cebb9b1a-aae7b5a01a0543afb1b5a7f44a9a864c received from {username='slave'} at 127.15.255.138:36825
I20240502 21:29:14.038465 1435 tablet_copy_service.cc:433] P 7d1e5c1b047443cbb28d7b0a9744babf: ending tablet copy session 1fab5794cf1740288abd82f2cebb9b1a-aae7b5a01a0543afb1b5a7f44a9a864c on tablet aae7b5a01a0543afb1b5a7f44a9a864c with peer 1fab5794cf1740288abd82f2cebb9b1a
I20240502 21:29:14.397444 5477 raft_consensus.cc:1050] T 05e885ae741c43808c6dfbcdc5769942 P 6ab8df3c796f4b6cb3e29d5045ad3ea0: attempting to promote NON_VOTER 16a97c1394f346f0b987f97cce7371b6 to VOTER
I20240502 21:29:14.398342 5477 consensus_queue.cc:237] T 05e885ae741c43808c6dfbcdc5769942 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } }
I20240502 21:29:14.400990 3987 raft_consensus.cc:1261] T 05e885ae741c43808c6dfbcdc5769942 P 16a97c1394f346f0b987f97cce7371b6 [term 1 LEARNER]: Refusing update from remote peer 6ab8df3c796f4b6cb3e29d5045ad3ea0: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:29:14.401559 1820 raft_consensus.cc:1261] T 05e885ae741c43808c6dfbcdc5769942 P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Refusing update from remote peer 6ab8df3c796f4b6cb3e29d5045ad3ea0: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:29:14.401517 5485 consensus_queue.cc:1035] T 05e885ae741c43808c6dfbcdc5769942 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:29:14.401901 1955 raft_consensus.cc:1261] T 05e885ae741c43808c6dfbcdc5769942 P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Refusing update from remote peer 6ab8df3c796f4b6cb3e29d5045ad3ea0: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:29:14.402444 5485 consensus_queue.cc:1035] T 05e885ae741c43808c6dfbcdc5769942 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:29:14.402649 1550 raft_consensus.cc:1261] T 05e885ae741c43808c6dfbcdc5769942 P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Refusing update from remote peer 6ab8df3c796f4b6cb3e29d5045ad3ea0: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:29:14.403295 2088 raft_consensus.cc:1261] T 05e885ae741c43808c6dfbcdc5769942 P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Refusing update from remote peer 6ab8df3c796f4b6cb3e29d5045ad3ea0: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:29:14.403358 5607 consensus_queue.cc:1035] T 05e885ae741c43808c6dfbcdc5769942 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:29:14.403820 5152 consensus_queue.cc:1035] T 05e885ae741c43808c6dfbcdc5769942 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:29:14.404143 1686 raft_consensus.cc:1261] T 05e885ae741c43808c6dfbcdc5769942 P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Refusing update from remote peer 6ab8df3c796f4b6cb3e29d5045ad3ea0: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:29:14.404124 5485 consensus_queue.cc:1035] T 05e885ae741c43808c6dfbcdc5769942 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:29:14.405200 5485 consensus_queue.cc:1035] T 05e885ae741c43808c6dfbcdc5769942 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:29:14.407653 1414 raft_consensus.cc:1261] T 05e885ae741c43808c6dfbcdc5769942 P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Refusing update from remote peer 6ab8df3c796f4b6cb3e29d5045ad3ea0: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:29:14.409832 5610 consensus_queue.cc:1035] T 05e885ae741c43808c6dfbcdc5769942 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:29:14.423194 5607 raft_consensus.cc:2932] T 05e885ae741c43808c6dfbcdc5769942 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 16a97c1394f346f0b987f97cce7371b6 (127.15.255.137) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } }
I20240502 21:29:14.424364 1818 raft_consensus.cc:2932] T 05e885ae741c43808c6dfbcdc5769942 P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 16a97c1394f346f0b987f97cce7371b6 (127.15.255.137) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } }
I20240502 21:29:14.424777 1414 raft_consensus.cc:2932] T 05e885ae741c43808c6dfbcdc5769942 P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 16a97c1394f346f0b987f97cce7371b6 (127.15.255.137) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } }
I20240502 21:29:14.425254 1550 raft_consensus.cc:2932] T 05e885ae741c43808c6dfbcdc5769942 P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 16a97c1394f346f0b987f97cce7371b6 (127.15.255.137) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } }
I20240502 21:29:14.428205 3987 raft_consensus.cc:2932] T 05e885ae741c43808c6dfbcdc5769942 P 16a97c1394f346f0b987f97cce7371b6 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 16a97c1394f346f0b987f97cce7371b6 (127.15.255.137) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } }
I20240502 21:29:14.430507 2088 raft_consensus.cc:2932] T 05e885ae741c43808c6dfbcdc5769942 P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 16a97c1394f346f0b987f97cce7371b6 (127.15.255.137) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } }
I20240502 21:29:14.431190 1686 raft_consensus.cc:2932] T 05e885ae741c43808c6dfbcdc5769942 P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 16a97c1394f346f0b987f97cce7371b6 (127.15.255.137) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } }
I20240502 21:29:14.433917 1266 catalog_manager.cc:5484] T 05e885ae741c43808c6dfbcdc5769942 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 reported cstate change: config changed from index 5 to 6, 16a97c1394f346f0b987f97cce7371b6 (127.15.255.137) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:29:14.435947 1955 raft_consensus.cc:2932] T 05e885ae741c43808c6dfbcdc5769942 P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 16a97c1394f346f0b987f97cce7371b6 (127.15.255.137) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } }
I20240502 21:29:14.451054 4119 raft_consensus.cc:1203] T aae7b5a01a0543afb1b5a7f44a9a864c P 1fab5794cf1740288abd82f2cebb9b1a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240502 21:29:14.455571 2225 consensus_queue.cc:237] T 05e885ae741c43808c6dfbcdc5769942 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } }
I20240502 21:29:14.458426 1414 raft_consensus.cc:1261] T 05e885ae741c43808c6dfbcdc5769942 P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Refusing update from remote peer 6ab8df3c796f4b6cb3e29d5045ad3ea0: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:29:14.458446 3987 raft_consensus.cc:1261] T 05e885ae741c43808c6dfbcdc5769942 P 16a97c1394f346f0b987f97cce7371b6 [term 1 FOLLOWER]: Refusing update from remote peer 6ab8df3c796f4b6cb3e29d5045ad3ea0: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:29:14.458976 2088 raft_consensus.cc:1261] T 05e885ae741c43808c6dfbcdc5769942 P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Refusing update from remote peer 6ab8df3c796f4b6cb3e29d5045ad3ea0: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:29:14.459133 1955 raft_consensus.cc:1261] T 05e885ae741c43808c6dfbcdc5769942 P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Refusing update from remote peer 6ab8df3c796f4b6cb3e29d5045ad3ea0: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:29:14.459244 1686 raft_consensus.cc:1261] T 05e885ae741c43808c6dfbcdc5769942 P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Refusing update from remote peer 6ab8df3c796f4b6cb3e29d5045ad3ea0: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:29:14.459662 5610 consensus_queue.cc:1035] T 05e885ae741c43808c6dfbcdc5769942 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:29:14.459990 1550 raft_consensus.cc:1261] T 05e885ae741c43808c6dfbcdc5769942 P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Refusing update from remote peer 6ab8df3c796f4b6cb3e29d5045ad3ea0: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:29:14.460259 5610 consensus_queue.cc:1035] T 05e885ae741c43808c6dfbcdc5769942 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:29:14.460808 5610 consensus_queue.cc:1035] T 05e885ae741c43808c6dfbcdc5769942 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:29:14.461202 5607 consensus_queue.cc:1035] T 05e885ae741c43808c6dfbcdc5769942 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:29:14.463362 5152 consensus_queue.cc:1035] T 05e885ae741c43808c6dfbcdc5769942 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:29:14.463871 5152 consensus_queue.cc:1035] T 05e885ae741c43808c6dfbcdc5769942 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:29:14.465778 5607 raft_consensus.cc:2932] T 05e885ae741c43808c6dfbcdc5769942 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } }
I20240502 21:29:14.467564 1550 raft_consensus.cc:2932] T 05e885ae741c43808c6dfbcdc5769942 P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } }
I20240502 21:29:14.468946 1255 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 05e885ae741c43808c6dfbcdc5769942 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:29:14.469910 2088 raft_consensus.cc:2932] T 05e885ae741c43808c6dfbcdc5769942 P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } }
I20240502 21:29:14.471201 1955 raft_consensus.cc:2932] T 05e885ae741c43808c6dfbcdc5769942 P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } }
I20240502 21:29:14.472110 1415 raft_consensus.cc:2932] T 05e885ae741c43808c6dfbcdc5769942 P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } }
I20240502 21:29:14.476577 1266 catalog_manager.cc:5484] T 05e885ae741c43808c6dfbcdc5769942 P a62a935ff43d494ca35d0c417943133d reported cstate change: config changed from index 6 to 7, VOTER 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132) evicted. New cstate: current_term: 1 leader_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } }
I20240502 21:29:14.481153 1798 tablet_service.cc:1514] Processing DeleteTablet for tablet 05e885ae741c43808c6dfbcdc5769942 with delete_type TABLET_DATA_TOMBSTONED (TS 153026dd7f8943cc93fedf6f48450fb4 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:60256
I20240502 21:29:14.484351 1685 raft_consensus.cc:2932] T 05e885ae741c43808c6dfbcdc5769942 P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } }
I20240502 21:29:14.484333 3987 raft_consensus.cc:2932] T 05e885ae741c43808c6dfbcdc5769942 P 16a97c1394f346f0b987f97cce7371b6 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } }
I20240502 21:29:14.496111 5632 tablet_replica.cc:330] T 05e885ae741c43808c6dfbcdc5769942 P 153026dd7f8943cc93fedf6f48450fb4: stopping tablet replica
I20240502 21:29:14.496390 5632 raft_consensus.cc:2229] T 05e885ae741c43808c6dfbcdc5769942 P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:29:14.496906 5632 raft_consensus.cc:2258] T 05e885ae741c43808c6dfbcdc5769942 P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:29:14.498122 5632 ts_tablet_manager.cc:1875] T 05e885ae741c43808c6dfbcdc5769942 P 153026dd7f8943cc93fedf6f48450fb4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:29:14.503437 5632 ts_tablet_manager.cc:1888] T 05e885ae741c43808c6dfbcdc5769942 P 153026dd7f8943cc93fedf6f48450fb4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240502 21:29:14.503545 5632 log.cc:1196] T 05e885ae741c43808c6dfbcdc5769942 P 153026dd7f8943cc93fedf6f48450fb4: Deleting WAL directory at /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-3/wal/wals/05e885ae741c43808c6dfbcdc5769942
I20240502 21:29:14.504304 1256 catalog_manager.cc:4830] TS 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132:44947): tablet 05e885ae741c43808c6dfbcdc5769942 (table rebalance_test_table_4 [id=d523de924ba14cb493e559296c083a34]) successfully deleted
I20240502 21:29:14.979734 5626 raft_consensus.cc:1050] T aae7b5a01a0543afb1b5a7f44a9a864c P 7d1e5c1b047443cbb28d7b0a9744babf: attempting to promote NON_VOTER 1fab5794cf1740288abd82f2cebb9b1a to VOTER
I20240502 21:29:14.980660 5626 consensus_queue.cc:237] T aae7b5a01a0543afb1b5a7f44a9a864c P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } }
I20240502 21:29:14.983456 4119 raft_consensus.cc:1261] T aae7b5a01a0543afb1b5a7f44a9a864c P 1fab5794cf1740288abd82f2cebb9b1a [term 1 LEARNER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:29:14.983484 2088 raft_consensus.cc:1261] T aae7b5a01a0543afb1b5a7f44a9a864c P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:29:14.984328 5647 consensus_queue.cc:1035] T aae7b5a01a0543afb1b5a7f44a9a864c P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:29:14.984565 1818 raft_consensus.cc:1261] T aae7b5a01a0543afb1b5a7f44a9a864c P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:29:14.985199 5646 consensus_queue.cc:1035] T aae7b5a01a0543afb1b5a7f44a9a864c P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:29:14.985843 2225 raft_consensus.cc:1261] T aae7b5a01a0543afb1b5a7f44a9a864c P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:29:14.986487 1550 raft_consensus.cc:1261] T aae7b5a01a0543afb1b5a7f44a9a864c P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:29:14.987083 5626 consensus_queue.cc:1035] T aae7b5a01a0543afb1b5a7f44a9a864c P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:29:14.987444 1953 raft_consensus.cc:1261] T aae7b5a01a0543afb1b5a7f44a9a864c P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:29:14.988046 5626 consensus_queue.cc:1035] T aae7b5a01a0543afb1b5a7f44a9a864c P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:29:14.989759 5626 consensus_queue.cc:1035] T aae7b5a01a0543afb1b5a7f44a9a864c P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:29:14.990319 5646 consensus_queue.cc:1035] T aae7b5a01a0543afb1b5a7f44a9a864c P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:29:14.991163 1685 raft_consensus.cc:1261] T aae7b5a01a0543afb1b5a7f44a9a864c P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 21:29:14.993176 5646 consensus_queue.cc:1035] T aae7b5a01a0543afb1b5a7f44a9a864c P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 21:29:14.998193 5626 raft_consensus.cc:2932] T aae7b5a01a0543afb1b5a7f44a9a864c P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } }
I20240502 21:29:14.998705 1953 raft_consensus.cc:2932] T aae7b5a01a0543afb1b5a7f44a9a864c P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } }
I20240502 21:29:14.999128 4119 raft_consensus.cc:2932] T aae7b5a01a0543afb1b5a7f44a9a864c P 1fab5794cf1740288abd82f2cebb9b1a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } }
I20240502 21:29:15.005703 1266 catalog_manager.cc:5484] T aae7b5a01a0543afb1b5a7f44a9a864c P e632c553ff4341328acf1cbaca678392 reported cstate change: config changed from index 5 to 6, 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } }
I20240502 21:29:15.009699 1818 raft_consensus.cc:2932] T aae7b5a01a0543afb1b5a7f44a9a864c P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } }
I20240502 21:29:15.015127 2088 raft_consensus.cc:2932] T aae7b5a01a0543afb1b5a7f44a9a864c P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } }
I20240502 21:29:15.022182 2227 raft_consensus.cc:2932] T aae7b5a01a0543afb1b5a7f44a9a864c P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } }
I20240502 21:29:15.022714 1685 raft_consensus.cc:2932] T aae7b5a01a0543afb1b5a7f44a9a864c P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } }
I20240502 21:29:15.045572 1549 raft_consensus.cc:2932] T aae7b5a01a0543afb1b5a7f44a9a864c P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } }
I20240502 21:29:15.050819 1415 consensus_queue.cc:237] T aae7b5a01a0543afb1b5a7f44a9a864c P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } }
I20240502 21:29:15.054122 1953 raft_consensus.cc:1261] T aae7b5a01a0543afb1b5a7f44a9a864c P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:29:15.054647 2227 raft_consensus.cc:1261] T aae7b5a01a0543afb1b5a7f44a9a864c P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:29:15.054693 1549 raft_consensus.cc:1261] T aae7b5a01a0543afb1b5a7f44a9a864c P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:29:15.054721 2088 raft_consensus.cc:1261] T aae7b5a01a0543afb1b5a7f44a9a864c P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:29:15.055269 5523 consensus_queue.cc:1035] T aae7b5a01a0543afb1b5a7f44a9a864c P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:29:15.056015 5646 consensus_queue.cc:1035] T aae7b5a01a0543afb1b5a7f44a9a864c P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:29:15.056375 5523 consensus_queue.cc:1035] T aae7b5a01a0543afb1b5a7f44a9a864c P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:29:15.058279 5626 consensus_queue.cc:1035] T aae7b5a01a0543afb1b5a7f44a9a864c P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:29:15.059424 1685 raft_consensus.cc:1261] T aae7b5a01a0543afb1b5a7f44a9a864c P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:29:15.059967 5626 consensus_queue.cc:1035] T aae7b5a01a0543afb1b5a7f44a9a864c P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:29:15.061347 5626 raft_consensus.cc:2932] T aae7b5a01a0543afb1b5a7f44a9a864c P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } }
I20240502 21:29:15.062238 1549 raft_consensus.cc:2932] T aae7b5a01a0543afb1b5a7f44a9a864c P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } }
I20240502 21:29:15.063180 4119 raft_consensus.cc:1261] T aae7b5a01a0543afb1b5a7f44a9a864c P 1fab5794cf1740288abd82f2cebb9b1a [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 21:29:15.063493 2088 raft_consensus.cc:2932] T aae7b5a01a0543afb1b5a7f44a9a864c P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } }
I20240502 21:29:15.065116 1255 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet aae7b5a01a0543afb1b5a7f44a9a864c with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:29:15.065143 5523 consensus_queue.cc:1035] T aae7b5a01a0543afb1b5a7f44a9a864c P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 21:29:15.066524 2225 raft_consensus.cc:2932] T aae7b5a01a0543afb1b5a7f44a9a864c P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } }
I20240502 21:29:15.067143 1953 raft_consensus.cc:2932] T aae7b5a01a0543afb1b5a7f44a9a864c P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } }
I20240502 21:29:15.070492 4119 raft_consensus.cc:2932] T aae7b5a01a0543afb1b5a7f44a9a864c P 1fab5794cf1740288abd82f2cebb9b1a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } }
I20240502 21:29:15.071676 1264 catalog_manager.cc:5484] T aae7b5a01a0543afb1b5a7f44a9a864c P 521bb93035324cecbebed2f6f2a630f8 reported cstate change: config changed from index 6 to 7, VOTER 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132) evicted. New cstate: current_term: 1 leader_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } }
I20240502 21:29:15.075649 1685 raft_consensus.cc:2932] T aae7b5a01a0543afb1b5a7f44a9a864c P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } }
I20240502 21:29:15.076390 1798 tablet_service.cc:1514] Processing DeleteTablet for tablet aae7b5a01a0543afb1b5a7f44a9a864c with delete_type TABLET_DATA_TOMBSTONED (TS 153026dd7f8943cc93fedf6f48450fb4 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:60256
I20240502 21:29:15.077112 5662 tablet_replica.cc:330] T aae7b5a01a0543afb1b5a7f44a9a864c P 153026dd7f8943cc93fedf6f48450fb4: stopping tablet replica
I20240502 21:29:15.077374 5662 raft_consensus.cc:2229] T aae7b5a01a0543afb1b5a7f44a9a864c P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:29:15.077740 5662 raft_consensus.cc:2258] T aae7b5a01a0543afb1b5a7f44a9a864c P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:29:15.081115 5662 ts_tablet_manager.cc:1875] T aae7b5a01a0543afb1b5a7f44a9a864c P 153026dd7f8943cc93fedf6f48450fb4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:29:15.086397 5662 ts_tablet_manager.cc:1888] T aae7b5a01a0543afb1b5a7f44a9a864c P 153026dd7f8943cc93fedf6f48450fb4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240502 21:29:15.086494 5662 log.cc:1196] T aae7b5a01a0543afb1b5a7f44a9a864c P 153026dd7f8943cc93fedf6f48450fb4: Deleting WAL directory at /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-3/wal/wals/aae7b5a01a0543afb1b5a7f44a9a864c
I20240502 21:29:15.087162 1256 catalog_manager.cc:4830] TS 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132:44947): tablet aae7b5a01a0543afb1b5a7f44a9a864c (table rebalance_test_table_4 [id=d523de924ba14cb493e559296c083a34]) successfully deleted
I20240502 21:29:15.470562 1798 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:29:15.482219 1664 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:15.494536 3968 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:15.503154 4101 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:15.524844 2068 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 21:29:15.543931 1394 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:29:15.584568 2205 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:29:15.641294 3836 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:15.642558 1528 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:29:15.674283 1931 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:29:15.866760 2068 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 21:29:15.874034 1798 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:29:15.877554 2205 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:29:15.877969 1394 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:29:15.885830 3968 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:15.898947 1528 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:29:15.941118 1664 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:15.973635 3836 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:15.979823 4101 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:16.009819 1931 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:29:16.145853 1415 consensus_queue.cc:237] T f8bee9bdcbff404280f000bdaeadf465 P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: NON_VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: true } }
I20240502 21:29:16.148563 3856 raft_consensus.cc:1261] T f8bee9bdcbff404280f000bdaeadf465 P d8d7297186694f77bb98de9174e4279c [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
W20240502 21:29:16.148854 1347 consensus_peers.cc:485] T f8bee9bdcbff404280f000bdaeadf465 P 7d1e5c1b047443cbb28d7b0a9744babf -> Peer 16a97c1394f346f0b987f97cce7371b6 (127.15.255.137:43717): Couldn't send request to peer 16a97c1394f346f0b987f97cce7371b6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f8bee9bdcbff404280f000bdaeadf465. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:29:16.148855 1685 raft_consensus.cc:1261] T f8bee9bdcbff404280f000bdaeadf465 P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:29:16.149185 5761 consensus_queue.cc:1035] T f8bee9bdcbff404280f000bdaeadf465 P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:29:16.149937 5762 consensus_queue.cc:1035] T f8bee9bdcbff404280f000bdaeadf465 P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:29:16.150683 2225 raft_consensus.cc:1261] T f8bee9bdcbff404280f000bdaeadf465 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:29:16.151901 5661 consensus_queue.cc:1035] T f8bee9bdcbff404280f000bdaeadf465 P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:29:16.153424 1953 raft_consensus.cc:1261] T f8bee9bdcbff404280f000bdaeadf465 P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:29:16.153561 1549 raft_consensus.cc:1261] T f8bee9bdcbff404280f000bdaeadf465 P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:29:16.154487 5657 consensus_queue.cc:1035] T f8bee9bdcbff404280f000bdaeadf465 P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:29:16.154893 5661 consensus_queue.cc:1035] T f8bee9bdcbff404280f000bdaeadf465 P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:29:16.156877 2087 raft_consensus.cc:1261] T f8bee9bdcbff404280f000bdaeadf465 P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:29:16.157500 5761 consensus_queue.cc:1035] T f8bee9bdcbff404280f000bdaeadf465 P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:29:16.158031 5661 raft_consensus.cc:2932] T f8bee9bdcbff404280f000bdaeadf465 P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 16a97c1394f346f0b987f97cce7371b6 (127.15.255.137) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: NON_VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: true } } }
I20240502 21:29:16.161727 1953 raft_consensus.cc:2932] T f8bee9bdcbff404280f000bdaeadf465 P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 16a97c1394f346f0b987f97cce7371b6 (127.15.255.137) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: NON_VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: true } } }
I20240502 21:29:16.165220 2225 raft_consensus.cc:2932] T f8bee9bdcbff404280f000bdaeadf465 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 16a97c1394f346f0b987f97cce7371b6 (127.15.255.137) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: NON_VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: true } } }
I20240502 21:29:16.166040 3856 raft_consensus.cc:2932] T f8bee9bdcbff404280f000bdaeadf465 P d8d7297186694f77bb98de9174e4279c [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 16a97c1394f346f0b987f97cce7371b6 (127.15.255.137) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: NON_VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: true } } }
I20240502 21:29:16.168622 1549 raft_consensus.cc:2932] T f8bee9bdcbff404280f000bdaeadf465 P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 16a97c1394f346f0b987f97cce7371b6 (127.15.255.137) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: NON_VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: true } } }
I20240502 21:29:16.171263 1263 catalog_manager.cc:5484] T f8bee9bdcbff404280f000bdaeadf465 P 7d1e5c1b047443cbb28d7b0a9744babf reported cstate change: config changed from index 7 to 8, NON_VOTER 16a97c1394f346f0b987f97cce7371b6 (127.15.255.137) added. New cstate: current_term: 1 leader_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: NON_VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 21:29:16.182194 1685 raft_consensus.cc:2932] T f8bee9bdcbff404280f000bdaeadf465 P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 16a97c1394f346f0b987f97cce7371b6 (127.15.255.137) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: NON_VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: true } } }
I20240502 21:29:16.195613 2087 raft_consensus.cc:2932] T f8bee9bdcbff404280f000bdaeadf465 P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 16a97c1394f346f0b987f97cce7371b6 (127.15.255.137) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: NON_VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: true } } }
I20240502 21:29:16.212301 1415 consensus_queue.cc:237] T 72355dd734ea4aeba031171e8c39696d P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { replace: true } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: NON_VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: true } }
I20240502 21:29:16.215590 2225 raft_consensus.cc:1261] T 72355dd734ea4aeba031171e8c39696d P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:29:16.215765 1685 raft_consensus.cc:1261] T 72355dd734ea4aeba031171e8c39696d P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
W20240502 21:29:16.215978 1350 consensus_peers.cc:485] T 72355dd734ea4aeba031171e8c39696d P 7d1e5c1b047443cbb28d7b0a9744babf -> Peer 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138:33375): Couldn't send request to peer 1fab5794cf1740288abd82f2cebb9b1a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 72355dd734ea4aeba031171e8c39696d. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:29:16.216254 5762 consensus_queue.cc:1035] T 72355dd734ea4aeba031171e8c39696d P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:29:16.216661 1549 raft_consensus.cc:1261] T 72355dd734ea4aeba031171e8c39696d P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:29:16.216959 3856 raft_consensus.cc:1261] T 72355dd734ea4aeba031171e8c39696d P d8d7297186694f77bb98de9174e4279c [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:29:16.217206 1953 raft_consensus.cc:1261] T 72355dd734ea4aeba031171e8c39696d P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:29:16.217171 5762 consensus_queue.cc:1035] T 72355dd734ea4aeba031171e8c39696d P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:29:16.217494 5763 consensus_queue.cc:1035] T 72355dd734ea4aeba031171e8c39696d P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:29:16.218416 5763 consensus_queue.cc:1035] T 72355dd734ea4aeba031171e8c39696d P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:29:16.221405 5762 consensus_queue.cc:1035] T 72355dd734ea4aeba031171e8c39696d P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:29:16.226508 1818 raft_consensus.cc:1261] T 72355dd734ea4aeba031171e8c39696d P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:29:16.227173 5762 consensus_queue.cc:1035] T 72355dd734ea4aeba031171e8c39696d P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:29:16.247800 5762 raft_consensus.cc:2932] T 72355dd734ea4aeba031171e8c39696d P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { replace: true } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: NON_VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: true } } }
I20240502 21:29:16.252745 1549 raft_consensus.cc:2932] T 72355dd734ea4aeba031171e8c39696d P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { replace: true } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: NON_VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: true } } }
I20240502 21:29:16.254009 3856 raft_consensus.cc:2932] T 72355dd734ea4aeba031171e8c39696d P d8d7297186694f77bb98de9174e4279c [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { replace: true } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: NON_VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: true } } }
I20240502 21:29:16.256273 1268 catalog_manager.cc:5484] T 72355dd734ea4aeba031171e8c39696d P 7d1e5c1b047443cbb28d7b0a9744babf reported cstate change: config changed from index 7 to 8, NON_VOTER 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138) added. New cstate: current_term: 1 leader_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: NON_VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 21:29:16.270586 2227 raft_consensus.cc:2932] T 72355dd734ea4aeba031171e8c39696d P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { replace: true } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: NON_VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: true } } }
I20240502 21:29:16.271608 1685 raft_consensus.cc:2932] T 72355dd734ea4aeba031171e8c39696d P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { replace: true } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: NON_VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: true } } }
I20240502 21:29:16.273730 1817 raft_consensus.cc:2932] T 72355dd734ea4aeba031171e8c39696d P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { replace: true } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: NON_VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: true } } }
I20240502 21:29:16.274757 1955 raft_consensus.cc:2932] T 72355dd734ea4aeba031171e8c39696d P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { replace: true } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: NON_VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: true } } }
I20240502 21:29:16.293311 2087 consensus_queue.cc:237] T 8f86d1517df74ead9e397ca9bed44c3f P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: true } }
I20240502 21:29:16.296684 4119 raft_consensus.cc:1261] T 8f86d1517df74ead9e397ca9bed44c3f P 1fab5794cf1740288abd82f2cebb9b1a [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:29:16.297120 2225 raft_consensus.cc:1261] T 8f86d1517df74ead9e397ca9bed44c3f P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:29:16.297217 1685 raft_consensus.cc:1261] T 8f86d1517df74ead9e397ca9bed44c3f P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:29:16.297546 1954 raft_consensus.cc:1261] T 8f86d1517df74ead9e397ca9bed44c3f P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:29:16.297786 5389 consensus_queue.cc:1035] T 8f86d1517df74ead9e397ca9bed44c3f P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240502 21:29:16.298228 2024 consensus_peers.cc:485] T 8f86d1517df74ead9e397ca9bed44c3f P 521bb93035324cecbebed2f6f2a630f8 -> Peer d8d7297186694f77bb98de9174e4279c (127.15.255.136:41459): Couldn't send request to peer d8d7297186694f77bb98de9174e4279c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8f86d1517df74ead9e397ca9bed44c3f. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:29:16.298316 5389 consensus_queue.cc:1035] T 8f86d1517df74ead9e397ca9bed44c3f P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:29:16.298614 5795 consensus_queue.cc:1035] T 8f86d1517df74ead9e397ca9bed44c3f P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:29:16.298930 5562 consensus_queue.cc:1035] T 8f86d1517df74ead9e397ca9bed44c3f P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:29:16.300242 1549 raft_consensus.cc:1261] T 8f86d1517df74ead9e397ca9bed44c3f P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:29:16.300899 5797 consensus_queue.cc:1035] T 8f86d1517df74ead9e397ca9bed44c3f P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:29:16.309846 1415 raft_consensus.cc:1261] T 8f86d1517df74ead9e397ca9bed44c3f P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:29:16.310453 5389 consensus_queue.cc:1035] T 8f86d1517df74ead9e397ca9bed44c3f P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:29:16.315665 5389 raft_consensus.cc:2932] T 8f86d1517df74ead9e397ca9bed44c3f P 521bb93035324cecbebed2f6f2a630f8 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER d8d7297186694f77bb98de9174e4279c (127.15.255.136) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: true } } }
I20240502 21:29:16.319043 4119 raft_consensus.cc:2932] T 8f86d1517df74ead9e397ca9bed44c3f P 1fab5794cf1740288abd82f2cebb9b1a [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER d8d7297186694f77bb98de9174e4279c (127.15.255.136) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: true } } }
I20240502 21:29:16.319056 1954 raft_consensus.cc:2932] T 8f86d1517df74ead9e397ca9bed44c3f P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER d8d7297186694f77bb98de9174e4279c (127.15.255.136) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: true } } }
I20240502 21:29:16.320475 1549 raft_consensus.cc:2932] T 8f86d1517df74ead9e397ca9bed44c3f P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER d8d7297186694f77bb98de9174e4279c (127.15.255.136) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: true } } }
I20240502 21:29:16.325080 1262 catalog_manager.cc:5484] T 8f86d1517df74ead9e397ca9bed44c3f P e632c553ff4341328acf1cbaca678392 reported cstate change: config changed from index 7 to 8, NON_VOTER d8d7297186694f77bb98de9174e4279c (127.15.255.136) added. New cstate: current_term: 1 leader_uuid: "521bb93035324cecbebed2f6f2a630f8" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: true } } }
I20240502 21:29:16.326027 1415 raft_consensus.cc:2932] T 8f86d1517df74ead9e397ca9bed44c3f P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER d8d7297186694f77bb98de9174e4279c (127.15.255.136) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: true } } }
I20240502 21:29:16.326416 2225 raft_consensus.cc:2932] T 8f86d1517df74ead9e397ca9bed44c3f P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER d8d7297186694f77bb98de9174e4279c (127.15.255.136) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: true } } }
I20240502 21:29:16.335837 1685 raft_consensus.cc:2932] T 8f86d1517df74ead9e397ca9bed44c3f P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER d8d7297186694f77bb98de9174e4279c (127.15.255.136) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: true } } }
I20240502 21:29:16.365618 2087 consensus_queue.cc:237] T f8de1fd0869348e6b0aa131b958ce822 P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { replace: true } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: NON_VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: true } }
I20240502 21:29:16.368271 1685 raft_consensus.cc:1261] T f8de1fd0869348e6b0aa131b958ce822 P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
W20240502 21:29:16.368489 2022 consensus_peers.cc:485] T f8de1fd0869348e6b0aa131b958ce822 P 521bb93035324cecbebed2f6f2a630f8 -> Peer 16a97c1394f346f0b987f97cce7371b6 (127.15.255.137:43717): Couldn't send request to peer 16a97c1394f346f0b987f97cce7371b6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f8de1fd0869348e6b0aa131b958ce822. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:29:16.368971 5562 consensus_queue.cc:1035] T f8de1fd0869348e6b0aa131b958ce822 P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:29:16.369560 1549 raft_consensus.cc:1261] T f8de1fd0869348e6b0aa131b958ce822 P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:29:16.369985 1415 raft_consensus.cc:1261] T f8de1fd0869348e6b0aa131b958ce822 P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:29:16.370062 5797 consensus_queue.cc:1035] T f8de1fd0869348e6b0aa131b958ce822 P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:29:16.370608 5797 consensus_queue.cc:1035] T f8de1fd0869348e6b0aa131b958ce822 P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:29:16.372045 4119 raft_consensus.cc:1261] T f8de1fd0869348e6b0aa131b958ce822 P 1fab5794cf1740288abd82f2cebb9b1a [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:29:16.372545 1817 raft_consensus.cc:1261] T f8de1fd0869348e6b0aa131b958ce822 P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:29:16.372550 5562 consensus_queue.cc:1035] T f8de1fd0869348e6b0aa131b958ce822 P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:29:16.373395 2225 raft_consensus.cc:1261] T f8de1fd0869348e6b0aa131b958ce822 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:29:16.373548 5797 consensus_queue.cc:1035] T f8de1fd0869348e6b0aa131b958ce822 P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:29:16.374078 5796 consensus_queue.cc:1035] T f8de1fd0869348e6b0aa131b958ce822 P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:29:16.386543 5797 raft_consensus.cc:2932] T f8de1fd0869348e6b0aa131b958ce822 P 521bb93035324cecbebed2f6f2a630f8 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 16a97c1394f346f0b987f97cce7371b6 (127.15.255.137) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { replace: true } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: NON_VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: true } } }
I20240502 21:29:16.387121 1685 raft_consensus.cc:2932] T f8de1fd0869348e6b0aa131b958ce822 P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 16a97c1394f346f0b987f97cce7371b6 (127.15.255.137) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { replace: true } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: NON_VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: true } } }
I20240502 21:29:16.393060 1415 raft_consensus.cc:2932] T f8de1fd0869348e6b0aa131b958ce822 P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 16a97c1394f346f0b987f97cce7371b6 (127.15.255.137) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { replace: true } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: NON_VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: true } } }
I20240502 21:29:16.393215 1549 raft_consensus.cc:2932] T f8de1fd0869348e6b0aa131b958ce822 P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 16a97c1394f346f0b987f97cce7371b6 (127.15.255.137) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { replace: true } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: NON_VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: true } } }
I20240502 21:29:16.392606 1268 catalog_manager.cc:5484] T f8de1fd0869348e6b0aa131b958ce822 P 16b3a337bc5e462f83ba895ad8660deb reported cstate change: config changed from index 7 to 8, NON_VOTER 16a97c1394f346f0b987f97cce7371b6 (127.15.255.137) added. New cstate: current_term: 1 leader_uuid: "521bb93035324cecbebed2f6f2a630f8" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { replace: true } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: NON_VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: true } } }
I20240502 21:29:16.404114 4121 raft_consensus.cc:2932] T f8de1fd0869348e6b0aa131b958ce822 P 1fab5794cf1740288abd82f2cebb9b1a [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 16a97c1394f346f0b987f97cce7371b6 (127.15.255.137) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { replace: true } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: NON_VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: true } } }
I20240502 21:29:16.415714 1817 raft_consensus.cc:2932] T f8de1fd0869348e6b0aa131b958ce822 P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 16a97c1394f346f0b987f97cce7371b6 (127.15.255.137) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { replace: true } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: NON_VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: true } } }
I20240502 21:29:16.417407 2226 raft_consensus.cc:2932] T f8de1fd0869348e6b0aa131b958ce822 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 16a97c1394f346f0b987f97cce7371b6 (127.15.255.137) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { replace: true } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: NON_VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: true } } }
I20240502 21:29:16.433459 2225 consensus_queue.cc:237] T 05e885ae741c43808c6dfbcdc5769942 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: NON_VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: true } }
I20240502 21:29:16.436149 3987 raft_consensus.cc:1261] T 05e885ae741c43808c6dfbcdc5769942 P 16a97c1394f346f0b987f97cce7371b6 [term 1 FOLLOWER]: Refusing update from remote peer 6ab8df3c796f4b6cb3e29d5045ad3ea0: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:29:16.436165 2087 raft_consensus.cc:1261] T 05e885ae741c43808c6dfbcdc5769942 P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Refusing update from remote peer 6ab8df3c796f4b6cb3e29d5045ad3ea0: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:29:16.436815 5824 consensus_queue.cc:1035] T 05e885ae741c43808c6dfbcdc5769942 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240502 21:29:16.437312 2161 consensus_peers.cc:485] T 05e885ae741c43808c6dfbcdc5769942 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 -> Peer 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138:33375): Couldn't send request to peer 1fab5794cf1740288abd82f2cebb9b1a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 05e885ae741c43808c6dfbcdc5769942. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:29:16.439205 1954 raft_consensus.cc:1261] T 05e885ae741c43808c6dfbcdc5769942 P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Refusing update from remote peer 6ab8df3c796f4b6cb3e29d5045ad3ea0: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:29:16.439253 1549 raft_consensus.cc:1261] T 05e885ae741c43808c6dfbcdc5769942 P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Refusing update from remote peer 6ab8df3c796f4b6cb3e29d5045ad3ea0: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:29:16.439774 1415 raft_consensus.cc:1261] T 05e885ae741c43808c6dfbcdc5769942 P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Refusing update from remote peer 6ab8df3c796f4b6cb3e29d5045ad3ea0: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:29:16.440008 5788 consensus_queue.cc:1035] T 05e885ae741c43808c6dfbcdc5769942 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:29:16.440691 5788 consensus_queue.cc:1035] T 05e885ae741c43808c6dfbcdc5769942 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:29:16.441614 5824 consensus_queue.cc:1035] T 05e885ae741c43808c6dfbcdc5769942 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:29:16.442122 5824 consensus_queue.cc:1035] T 05e885ae741c43808c6dfbcdc5769942 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:29:16.442976 1685 raft_consensus.cc:1261] T 05e885ae741c43808c6dfbcdc5769942 P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Refusing update from remote peer 6ab8df3c796f4b6cb3e29d5045ad3ea0: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:29:16.445744 5627 consensus_queue.cc:1035] T 05e885ae741c43808c6dfbcdc5769942 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:29:16.455700 5825 raft_consensus.cc:2932] T 05e885ae741c43808c6dfbcdc5769942 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: NON_VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: true } } }
I20240502 21:29:16.456058 1415 raft_consensus.cc:2932] T 05e885ae741c43808c6dfbcdc5769942 P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: NON_VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: true } } }
I20240502 21:29:16.460808 1953 raft_consensus.cc:2932] T 05e885ae741c43808c6dfbcdc5769942 P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: NON_VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: true } } }
I20240502 21:29:16.466917 1266 catalog_manager.cc:5484] T 05e885ae741c43808c6dfbcdc5769942 P 7d1e5c1b047443cbb28d7b0a9744babf reported cstate change: config changed from index 7 to 8, NON_VOTER 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138) added. New cstate: current_term: 1 leader_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: NON_VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: true } } }
I20240502 21:29:16.470520 3987 raft_consensus.cc:2932] T 05e885ae741c43808c6dfbcdc5769942 P 16a97c1394f346f0b987f97cce7371b6 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: NON_VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: true } } }
I20240502 21:29:16.477734 2087 raft_consensus.cc:2932] T 05e885ae741c43808c6dfbcdc5769942 P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: NON_VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: true } } }
I20240502 21:29:16.478049 1685 raft_consensus.cc:2932] T 05e885ae741c43808c6dfbcdc5769942 P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: NON_VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: true } } }
I20240502 21:29:16.484148 1549 raft_consensus.cc:2932] T 05e885ae741c43808c6dfbcdc5769942 P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: NON_VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: true } } }
I20240502 21:29:16.497965 2087 consensus_queue.cc:237] T e44bf56d64b24e328803ebd0a3a54ceb P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { replace: true } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: true } }
I20240502 21:29:16.500975 1685 raft_consensus.cc:1261] T e44bf56d64b24e328803ebd0a3a54ceb P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:29:16.501005 1415 raft_consensus.cc:1261] T e44bf56d64b24e328803ebd0a3a54ceb P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:29:16.501535 5797 consensus_queue.cc:1035] T e44bf56d64b24e328803ebd0a3a54ceb P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240502 21:29:16.501832 2024 consensus_peers.cc:485] T e44bf56d64b24e328803ebd0a3a54ceb P 521bb93035324cecbebed2f6f2a630f8 -> Peer d8d7297186694f77bb98de9174e4279c (127.15.255.136:41459): Couldn't send request to peer d8d7297186694f77bb98de9174e4279c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e44bf56d64b24e328803ebd0a3a54ceb. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:29:16.501890 4119 raft_consensus.cc:1261] T e44bf56d64b24e328803ebd0a3a54ceb P 1fab5794cf1740288abd82f2cebb9b1a [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:29:16.501924 5562 consensus_queue.cc:1035] T e44bf56d64b24e328803ebd0a3a54ceb P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:29:16.502496 5562 consensus_queue.cc:1035] T e44bf56d64b24e328803ebd0a3a54ceb P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:29:16.503724 2225 raft_consensus.cc:1261] T e44bf56d64b24e328803ebd0a3a54ceb P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:29:16.504555 5562 consensus_queue.cc:1035] T e44bf56d64b24e328803ebd0a3a54ceb P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:29:16.510869 1549 raft_consensus.cc:1261] T e44bf56d64b24e328803ebd0a3a54ceb P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:29:16.511404 5562 consensus_queue.cc:1035] T e44bf56d64b24e328803ebd0a3a54ceb P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:29:16.514775 1954 raft_consensus.cc:1261] T e44bf56d64b24e328803ebd0a3a54ceb P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:29:16.515573 5797 consensus_queue.cc:1035] T e44bf56d64b24e328803ebd0a3a54ceb P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:29:16.516683 5562 raft_consensus.cc:2932] T e44bf56d64b24e328803ebd0a3a54ceb P 521bb93035324cecbebed2f6f2a630f8 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER d8d7297186694f77bb98de9174e4279c (127.15.255.136) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { replace: true } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: true } } }
I20240502 21:29:16.518997 1954 raft_consensus.cc:2932] T e44bf56d64b24e328803ebd0a3a54ceb P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER d8d7297186694f77bb98de9174e4279c (127.15.255.136) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { replace: true } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: true } } }
I20240502 21:29:16.521091 2225 raft_consensus.cc:2932] T e44bf56d64b24e328803ebd0a3a54ceb P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER d8d7297186694f77bb98de9174e4279c (127.15.255.136) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { replace: true } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: true } } }
I20240502 21:29:16.521565 1685 raft_consensus.cc:2932] T e44bf56d64b24e328803ebd0a3a54ceb P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER d8d7297186694f77bb98de9174e4279c (127.15.255.136) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { replace: true } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: true } } }
I20240502 21:29:16.523728 1550 raft_consensus.cc:2932] T e44bf56d64b24e328803ebd0a3a54ceb P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER d8d7297186694f77bb98de9174e4279c (127.15.255.136) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { replace: true } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: true } } }
I20240502 21:29:16.529469 1268 catalog_manager.cc:5484] T e44bf56d64b24e328803ebd0a3a54ceb P 16b3a337bc5e462f83ba895ad8660deb reported cstate change: config changed from index 7 to 8, NON_VOTER d8d7297186694f77bb98de9174e4279c (127.15.255.136) added. New cstate: current_term: 1 leader_uuid: "521bb93035324cecbebed2f6f2a630f8" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { replace: true } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: true } } }
I20240502 21:29:16.529467 4121 raft_consensus.cc:2932] T e44bf56d64b24e328803ebd0a3a54ceb P 1fab5794cf1740288abd82f2cebb9b1a [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER d8d7297186694f77bb98de9174e4279c (127.15.255.136) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { replace: true } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: true } } }
I20240502 21:29:16.538640 1415 raft_consensus.cc:2932] T e44bf56d64b24e328803ebd0a3a54ceb P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER d8d7297186694f77bb98de9174e4279c (127.15.255.136) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { replace: true } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: true } } }
I20240502 21:29:16.557464 2087 consensus_queue.cc:237] T 1aa57620996e42b6adf941e9d606f7e2 P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: NON_VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: true } }
I20240502 21:29:16.560199 3856 raft_consensus.cc:1261] T 1aa57620996e42b6adf941e9d606f7e2 P d8d7297186694f77bb98de9174e4279c [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:29:16.560843 1817 raft_consensus.cc:1261] T 1aa57620996e42b6adf941e9d606f7e2 P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:29:16.560798 5562 consensus_queue.cc:1035] T 1aa57620996e42b6adf941e9d606f7e2 P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:29:16.560989 1415 raft_consensus.cc:1261] T 1aa57620996e42b6adf941e9d606f7e2 P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:29:16.561491 1685 raft_consensus.cc:1261] T 1aa57620996e42b6adf941e9d606f7e2 P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:29:16.561553 1550 raft_consensus.cc:1261] T 1aa57620996e42b6adf941e9d606f7e2 P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:29:16.562031 5797 consensus_queue.cc:1035] T 1aa57620996e42b6adf941e9d606f7e2 P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:29:16.562304 5795 consensus_queue.cc:1035] T 1aa57620996e42b6adf941e9d606f7e2 P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:29:16.562842 5795 consensus_queue.cc:1035] T 1aa57620996e42b6adf941e9d606f7e2 P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:29:16.563118 2225 raft_consensus.cc:1261] T 1aa57620996e42b6adf941e9d606f7e2 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:29:16.563339 5389 consensus_queue.cc:1035] T 1aa57620996e42b6adf941e9d606f7e2 P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240502 21:29:16.563848 2022 consensus_peers.cc:485] T 1aa57620996e42b6adf941e9d606f7e2 P 521bb93035324cecbebed2f6f2a630f8 -> Peer 16a97c1394f346f0b987f97cce7371b6 (127.15.255.137:43717): Couldn't send request to peer 16a97c1394f346f0b987f97cce7371b6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1aa57620996e42b6adf941e9d606f7e2. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:29:16.563979 5389 consensus_queue.cc:1035] T 1aa57620996e42b6adf941e9d606f7e2 P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:29:16.577009 1686 raft_consensus.cc:2932] T 1aa57620996e42b6adf941e9d606f7e2 P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 16a97c1394f346f0b987f97cce7371b6 (127.15.255.137) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: NON_VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: true } } }
I20240502 21:29:16.580439 5796 raft_consensus.cc:2932] T 1aa57620996e42b6adf941e9d606f7e2 P 521bb93035324cecbebed2f6f2a630f8 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 16a97c1394f346f0b987f97cce7371b6 (127.15.255.137) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: NON_VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: true } } }
I20240502 21:29:16.587284 1268 catalog_manager.cc:5484] T 1aa57620996e42b6adf941e9d606f7e2 P 16b3a337bc5e462f83ba895ad8660deb reported cstate change: config changed from index 7 to 8, NON_VOTER 16a97c1394f346f0b987f97cce7371b6 (127.15.255.137) added. New cstate: current_term: 1 leader_uuid: "521bb93035324cecbebed2f6f2a630f8" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: NON_VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: true } } }
I20240502 21:29:16.588358 3856 raft_consensus.cc:2932] T 1aa57620996e42b6adf941e9d606f7e2 P d8d7297186694f77bb98de9174e4279c [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 16a97c1394f346f0b987f97cce7371b6 (127.15.255.137) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: NON_VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: true } } }
I20240502 21:29:16.589107 1550 raft_consensus.cc:2932] T 1aa57620996e42b6adf941e9d606f7e2 P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 16a97c1394f346f0b987f97cce7371b6 (127.15.255.137) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: NON_VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: true } } }
I20240502 21:29:16.598719 2226 raft_consensus.cc:2932] T 1aa57620996e42b6adf941e9d606f7e2 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 16a97c1394f346f0b987f97cce7371b6 (127.15.255.137) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: NON_VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: true } } }
I20240502 21:29:16.614437 1817 raft_consensus.cc:2932] T 1aa57620996e42b6adf941e9d606f7e2 P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 16a97c1394f346f0b987f97cce7371b6 (127.15.255.137) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: NON_VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: true } } }
I20240502 21:29:16.619107 1414 raft_consensus.cc:2932] T 1aa57620996e42b6adf941e9d606f7e2 P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 16a97c1394f346f0b987f97cce7371b6 (127.15.255.137) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: NON_VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: true } } }
I20240502 21:29:16.630590 1954 consensus_queue.cc:237] T 444c1cba2a6f4729aa01b75d06b6b517 P e632c553ff4341328acf1cbaca678392 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { replace: true } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: NON_VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: true } }
I20240502 21:29:16.633749 1685 raft_consensus.cc:1261] T 444c1cba2a6f4729aa01b75d06b6b517 P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Refusing update from remote peer e632c553ff4341328acf1cbaca678392: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:29:16.633878 3856 raft_consensus.cc:1261] T 444c1cba2a6f4729aa01b75d06b6b517 P d8d7297186694f77bb98de9174e4279c [term 1 FOLLOWER]: Refusing update from remote peer e632c553ff4341328acf1cbaca678392: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:29:16.634349 5735 consensus_queue.cc:1035] T 444c1cba2a6f4729aa01b75d06b6b517 P e632c553ff4341328acf1cbaca678392 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:29:16.635083 5735 consensus_queue.cc:1035] T 444c1cba2a6f4729aa01b75d06b6b517 P e632c553ff4341328acf1cbaca678392 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:29:16.639014 1550 raft_consensus.cc:1261] T 444c1cba2a6f4729aa01b75d06b6b517 P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Refusing update from remote peer e632c553ff4341328acf1cbaca678392: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:29:16.639784 5463 consensus_queue.cc:1035] T 444c1cba2a6f4729aa01b75d06b6b517 P e632c553ff4341328acf1cbaca678392 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:29:16.640013 1414 raft_consensus.cc:1261] T 444c1cba2a6f4729aa01b75d06b6b517 P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Refusing update from remote peer e632c553ff4341328acf1cbaca678392: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:29:16.640777 5463 consensus_queue.cc:1035] T 444c1cba2a6f4729aa01b75d06b6b517 P e632c553ff4341328acf1cbaca678392 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:29:16.642776 2225 raft_consensus.cc:1261] T 444c1cba2a6f4729aa01b75d06b6b517 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Refusing update from remote peer e632c553ff4341328acf1cbaca678392: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:29:16.643848 5735 consensus_queue.cc:1035] T 444c1cba2a6f4729aa01b75d06b6b517 P e632c553ff4341328acf1cbaca678392 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240502 21:29:16.645785 1887 consensus_peers.cc:485] T 444c1cba2a6f4729aa01b75d06b6b517 P e632c553ff4341328acf1cbaca678392 -> Peer 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138:33375): Couldn't send request to peer 1fab5794cf1740288abd82f2cebb9b1a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 444c1cba2a6f4729aa01b75d06b6b517. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:29:16.649947 5463 raft_consensus.cc:2932] T 444c1cba2a6f4729aa01b75d06b6b517 P e632c553ff4341328acf1cbaca678392 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { replace: true } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: NON_VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: true } } }
I20240502 21:29:16.652294 5866 ts_tablet_manager.cc:902] T f8bee9bdcbff404280f000bdaeadf465 P 16a97c1394f346f0b987f97cce7371b6: Initiating tablet copy from peer 7d1e5c1b047443cbb28d7b0a9744babf (127.15.255.129:37897)
I20240502 21:29:16.651757 2225 raft_consensus.cc:2932] T 444c1cba2a6f4729aa01b75d06b6b517 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { replace: true } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: NON_VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: true } } }
I20240502 21:29:16.657828 1266 catalog_manager.cc:5484] T 444c1cba2a6f4729aa01b75d06b6b517 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 reported cstate change: config changed from index 7 to 8, NON_VOTER 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138) added. New cstate: current_term: 1 leader_uuid: "e632c553ff4341328acf1cbaca678392" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { replace: true } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: NON_VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: true } } }
I20240502 21:29:16.658169 1414 raft_consensus.cc:2932] T 444c1cba2a6f4729aa01b75d06b6b517 P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { replace: true } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: NON_VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: true } } }
I20240502 21:29:16.667686 1549 raft_consensus.cc:2932] T 444c1cba2a6f4729aa01b75d06b6b517 P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { replace: true } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: NON_VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: true } } }
I20240502 21:29:16.669337 2087 raft_consensus.cc:1261] T 444c1cba2a6f4729aa01b75d06b6b517 P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Refusing update from remote peer e632c553ff4341328acf1cbaca678392: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:29:16.669656 3856 raft_consensus.cc:2932] T 444c1cba2a6f4729aa01b75d06b6b517 P d8d7297186694f77bb98de9174e4279c [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { replace: true } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: NON_VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: true } } }
I20240502 21:29:16.669797 1685 raft_consensus.cc:2932] T 444c1cba2a6f4729aa01b75d06b6b517 P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { replace: true } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: NON_VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: true } } }
I20240502 21:29:16.677565 5869 consensus_queue.cc:1035] T 444c1cba2a6f4729aa01b75d06b6b517 P e632c553ff4341328acf1cbaca678392 [LEADER]: Connected to new peer: Peer: permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:29:16.679816 2087 raft_consensus.cc:2932] T 444c1cba2a6f4729aa01b75d06b6b517 P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { replace: true } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: NON_VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: true } } }
I20240502 21:29:16.691406 5876 ts_tablet_manager.cc:902] T 72355dd734ea4aeba031171e8c39696d P 1fab5794cf1740288abd82f2cebb9b1a: Initiating tablet copy from peer 7d1e5c1b047443cbb28d7b0a9744babf (127.15.255.129:37897)
I20240502 21:29:16.701687 5876 tablet_copy_client.cc:308] T 72355dd734ea4aeba031171e8c39696d P 1fab5794cf1740288abd82f2cebb9b1a: tablet copy: Beginning tablet copy session from remote peer at address 127.15.255.129:37897
I20240502 21:29:16.702311 1435 tablet_copy_service.cc:140] P 7d1e5c1b047443cbb28d7b0a9744babf: Received BeginTabletCopySession request for tablet 72355dd734ea4aeba031171e8c39696d from peer 1fab5794cf1740288abd82f2cebb9b1a ({username='slave'} at 127.15.255.138:36825)
I20240502 21:29:16.702513 1435 tablet_copy_service.cc:161] P 7d1e5c1b047443cbb28d7b0a9744babf: Beginning new tablet copy session on tablet 72355dd734ea4aeba031171e8c39696d from peer 1fab5794cf1740288abd82f2cebb9b1a at {username='slave'} at 127.15.255.138:36825: session id = 1fab5794cf1740288abd82f2cebb9b1a-72355dd734ea4aeba031171e8c39696d
I20240502 21:29:16.702685 5866 tablet_copy_client.cc:308] T f8bee9bdcbff404280f000bdaeadf465 P 16a97c1394f346f0b987f97cce7371b6: tablet copy: Beginning tablet copy session from remote peer at address 127.15.255.129:37897
I20240502 21:29:16.703334 1434 tablet_copy_service.cc:140] P 7d1e5c1b047443cbb28d7b0a9744babf: Received BeginTabletCopySession request for tablet f8bee9bdcbff404280f000bdaeadf465 from peer 16a97c1394f346f0b987f97cce7371b6 ({username='slave'} at 127.15.255.137:43115)
I20240502 21:29:16.703557 1434 tablet_copy_service.cc:161] P 7d1e5c1b047443cbb28d7b0a9744babf: Beginning new tablet copy session on tablet f8bee9bdcbff404280f000bdaeadf465 from peer 16a97c1394f346f0b987f97cce7371b6 at {username='slave'} at 127.15.255.137:43115: session id = 16a97c1394f346f0b987f97cce7371b6-f8bee9bdcbff404280f000bdaeadf465
I20240502 21:29:16.705657 1435 tablet_copy_source_session.cc:205] T 72355dd734ea4aeba031171e8c39696d P 7d1e5c1b047443cbb28d7b0a9744babf: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:29:16.706475 1434 tablet_copy_source_session.cc:205] T f8bee9bdcbff404280f000bdaeadf465 P 7d1e5c1b047443cbb28d7b0a9744babf: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:29:16.708470 5876 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72355dd734ea4aeba031171e8c39696d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:29:16.709072 5866 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8bee9bdcbff404280f000bdaeadf465. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:29:16.715219 5876 tablet_copy_client.cc:783] T 72355dd734ea4aeba031171e8c39696d P 1fab5794cf1740288abd82f2cebb9b1a: tablet copy: Starting download of 0 data blocks...
I20240502 21:29:16.715453 5876 tablet_copy_client.cc:647] T 72355dd734ea4aeba031171e8c39696d P 1fab5794cf1740288abd82f2cebb9b1a: tablet copy: Starting download of 1 WAL segments...
I20240502 21:29:16.716866 5866 tablet_copy_client.cc:783] T f8bee9bdcbff404280f000bdaeadf465 P 16a97c1394f346f0b987f97cce7371b6: tablet copy: Starting download of 0 data blocks...
I20240502 21:29:16.717111 5866 tablet_copy_client.cc:647] T f8bee9bdcbff404280f000bdaeadf465 P 16a97c1394f346f0b987f97cce7371b6: tablet copy: Starting download of 1 WAL segments...
I20240502 21:29:16.717751 1414 consensus_queue.cc:237] T aae7b5a01a0543afb1b5a7f44a9a864c P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: true } }
I20240502 21:29:16.721006 5866 tablet_copy_client.cc:523] T f8bee9bdcbff404280f000bdaeadf465 P 16a97c1394f346f0b987f97cce7371b6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:29:16.721077 5876 tablet_copy_client.cc:523] T 72355dd734ea4aeba031171e8c39696d P 1fab5794cf1740288abd82f2cebb9b1a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
W20240502 21:29:16.721311 1349 consensus_peers.cc:485] T aae7b5a01a0543afb1b5a7f44a9a864c P 7d1e5c1b047443cbb28d7b0a9744babf -> Peer d8d7297186694f77bb98de9174e4279c (127.15.255.136:41459): Couldn't send request to peer d8d7297186694f77bb98de9174e4279c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: aae7b5a01a0543afb1b5a7f44a9a864c. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:29:16.722179 4119 raft_consensus.cc:1261] T aae7b5a01a0543afb1b5a7f44a9a864c P 1fab5794cf1740288abd82f2cebb9b1a [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:29:16.722530 2225 raft_consensus.cc:1261] T aae7b5a01a0543afb1b5a7f44a9a864c P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:29:16.723059 5763 consensus_queue.cc:1035] T aae7b5a01a0543afb1b5a7f44a9a864c P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:29:16.723302 1954 raft_consensus.cc:1261] T aae7b5a01a0543afb1b5a7f44a9a864c P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:29:16.723459 1685 raft_consensus.cc:1261] T aae7b5a01a0543afb1b5a7f44a9a864c P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:29:16.723867 5761 consensus_queue.cc:1035] T aae7b5a01a0543afb1b5a7f44a9a864c P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:29:16.723991 2087 raft_consensus.cc:1261] T aae7b5a01a0543afb1b5a7f44a9a864c P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:29:16.724588 5876 tablet_bootstrap.cc:492] T 72355dd734ea4aeba031171e8c39696d P 1fab5794cf1740288abd82f2cebb9b1a: Bootstrap starting.
I20240502 21:29:16.725226 5762 consensus_queue.cc:1035] T aae7b5a01a0543afb1b5a7f44a9a864c P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:29:16.725558 5661 consensus_queue.cc:1035] T aae7b5a01a0543afb1b5a7f44a9a864c P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:29:16.726148 5661 consensus_queue.cc:1035] T aae7b5a01a0543afb1b5a7f44a9a864c P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:29:16.727159 1549 raft_consensus.cc:1261] T aae7b5a01a0543afb1b5a7f44a9a864c P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:29:16.730710 5762 consensus_queue.cc:1035] T aae7b5a01a0543afb1b5a7f44a9a864c P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:29:16.742645 5866 tablet_bootstrap.cc:492] T f8bee9bdcbff404280f000bdaeadf465 P 16a97c1394f346f0b987f97cce7371b6: Bootstrap starting.
I20240502 21:29:16.749974 5762 raft_consensus.cc:2932] T aae7b5a01a0543afb1b5a7f44a9a864c P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER d8d7297186694f77bb98de9174e4279c (127.15.255.136) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: true } } }
I20240502 21:29:16.754330 2087 raft_consensus.cc:2932] T aae7b5a01a0543afb1b5a7f44a9a864c P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER d8d7297186694f77bb98de9174e4279c (127.15.255.136) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: true } } }
I20240502 21:29:16.754915 1685 raft_consensus.cc:2932] T aae7b5a01a0543afb1b5a7f44a9a864c P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER d8d7297186694f77bb98de9174e4279c (127.15.255.136) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: true } } }
I20240502 21:29:16.755031 4119 raft_consensus.cc:2932] T aae7b5a01a0543afb1b5a7f44a9a864c P 1fab5794cf1740288abd82f2cebb9b1a [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER d8d7297186694f77bb98de9174e4279c (127.15.255.136) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: true } } }
I20240502 21:29:16.761498 1266 catalog_manager.cc:5484] T aae7b5a01a0543afb1b5a7f44a9a864c P 521bb93035324cecbebed2f6f2a630f8 reported cstate change: config changed from index 7 to 8, NON_VOTER d8d7297186694f77bb98de9174e4279c (127.15.255.136) added. New cstate: current_term: 1 leader_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: true } } }
I20240502 21:29:16.764822 1549 raft_consensus.cc:2932] T aae7b5a01a0543afb1b5a7f44a9a864c P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER d8d7297186694f77bb98de9174e4279c (127.15.255.136) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: true } } }
I20240502 21:29:16.779613 2225 raft_consensus.cc:2932] T aae7b5a01a0543afb1b5a7f44a9a864c P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER d8d7297186694f77bb98de9174e4279c (127.15.255.136) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: true } } }
I20240502 21:29:16.788559 1953 raft_consensus.cc:2932] T aae7b5a01a0543afb1b5a7f44a9a864c P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER d8d7297186694f77bb98de9174e4279c (127.15.255.136) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: true } } }
I20240502 21:29:16.799355 5876 tablet_bootstrap.cc:492] T 72355dd734ea4aeba031171e8c39696d P 1fab5794cf1740288abd82f2cebb9b1a: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:29:16.799669 5876 tablet_bootstrap.cc:492] T 72355dd734ea4aeba031171e8c39696d P 1fab5794cf1740288abd82f2cebb9b1a: Bootstrap complete.
I20240502 21:29:16.799877 5876 ts_tablet_manager.cc:1367] T 72355dd734ea4aeba031171e8c39696d P 1fab5794cf1740288abd82f2cebb9b1a: Time spent bootstrapping tablet: real 0.075s user 0.049s sys 0.016s
I20240502 21:29:16.800474 5876 raft_consensus.cc:348] T 72355dd734ea4aeba031171e8c39696d P 1fab5794cf1740288abd82f2cebb9b1a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { replace: true } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: NON_VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: true } }
I20240502 21:29:16.800808 5876 raft_consensus.cc:729] T 72355dd734ea4aeba031171e8c39696d P 1fab5794cf1740288abd82f2cebb9b1a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1fab5794cf1740288abd82f2cebb9b1a, State: Initialized, Role: LEARNER
I20240502 21:29:16.801048 5876 consensus_queue.cc:260] T 72355dd734ea4aeba031171e8c39696d P 1fab5794cf1740288abd82f2cebb9b1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { replace: true } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: NON_VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: true } }
I20240502 21:29:16.801657 5876 ts_tablet_manager.cc:1398] T 72355dd734ea4aeba031171e8c39696d P 1fab5794cf1740288abd82f2cebb9b1a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:29:16.802472 1434 tablet_copy_service.cc:342] P 7d1e5c1b047443cbb28d7b0a9744babf: Request end of tablet copy session 1fab5794cf1740288abd82f2cebb9b1a-72355dd734ea4aeba031171e8c39696d received from {username='slave'} at 127.15.255.138:36825
I20240502 21:29:16.802616 1434 tablet_copy_service.cc:433] P 7d1e5c1b047443cbb28d7b0a9744babf: ending tablet copy session 1fab5794cf1740288abd82f2cebb9b1a-72355dd734ea4aeba031171e8c39696d on tablet 72355dd734ea4aeba031171e8c39696d with peer 1fab5794cf1740288abd82f2cebb9b1a
I20240502 21:29:16.817684 5866 tablet_bootstrap.cc:492] T f8bee9bdcbff404280f000bdaeadf465 P 16a97c1394f346f0b987f97cce7371b6: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:29:16.818109 5866 tablet_bootstrap.cc:492] T f8bee9bdcbff404280f000bdaeadf465 P 16a97c1394f346f0b987f97cce7371b6: Bootstrap complete.
I20240502 21:29:16.818392 5866 ts_tablet_manager.cc:1367] T f8bee9bdcbff404280f000bdaeadf465 P 16a97c1394f346f0b987f97cce7371b6: Time spent bootstrapping tablet: real 0.076s user 0.040s sys 0.009s
I20240502 21:29:16.819198 5866 raft_consensus.cc:348] T f8bee9bdcbff404280f000bdaeadf465 P 16a97c1394f346f0b987f97cce7371b6 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: NON_VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: true } }
I20240502 21:29:16.819473 5866 raft_consensus.cc:729] T f8bee9bdcbff404280f000bdaeadf465 P 16a97c1394f346f0b987f97cce7371b6 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 16a97c1394f346f0b987f97cce7371b6, State: Initialized, Role: LEARNER
I20240502 21:29:16.819689 5866 consensus_queue.cc:260] T f8bee9bdcbff404280f000bdaeadf465 P 16a97c1394f346f0b987f97cce7371b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: NON_VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: true } }
I20240502 21:29:16.820466 5866 ts_tablet_manager.cc:1398] T f8bee9bdcbff404280f000bdaeadf465 P 16a97c1394f346f0b987f97cce7371b6: Time spent starting tablet: real 0.002s user 0.003s sys 0.001s
I20240502 21:29:16.821062 1434 tablet_copy_service.cc:342] P 7d1e5c1b047443cbb28d7b0a9744babf: Request end of tablet copy session 16a97c1394f346f0b987f97cce7371b6-f8bee9bdcbff404280f000bdaeadf465 received from {username='slave'} at 127.15.255.137:43115
I20240502 21:29:16.821214 1434 tablet_copy_service.cc:433] P 7d1e5c1b047443cbb28d7b0a9744babf: ending tablet copy session 16a97c1394f346f0b987f97cce7371b6-f8bee9bdcbff404280f000bdaeadf465 on tablet f8bee9bdcbff404280f000bdaeadf465 with peer 16a97c1394f346f0b987f97cce7371b6
I20240502 21:29:16.826072 1817 consensus_queue.cc:237] T 13af1b96121c492c9373f9acbb55725a P 153026dd7f8943cc93fedf6f48450fb4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { replace: true } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: NON_VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: true } }
I20240502 21:29:16.829370 3856 raft_consensus.cc:1261] T 13af1b96121c492c9373f9acbb55725a P d8d7297186694f77bb98de9174e4279c [term 1 FOLLOWER]: Refusing update from remote peer 153026dd7f8943cc93fedf6f48450fb4: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:29:16.829957 4789 consensus_queue.cc:1035] T 13af1b96121c492c9373f9acbb55725a P 153026dd7f8943cc93fedf6f48450fb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:29:16.837314 2087 raft_consensus.cc:1261] T 13af1b96121c492c9373f9acbb55725a P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Refusing update from remote peer 153026dd7f8943cc93fedf6f48450fb4: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:29:16.837921 5907 consensus_queue.cc:1035] T 13af1b96121c492c9373f9acbb55725a P 153026dd7f8943cc93fedf6f48450fb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:29:16.838560 1549 raft_consensus.cc:1261] T 13af1b96121c492c9373f9acbb55725a P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Refusing update from remote peer 153026dd7f8943cc93fedf6f48450fb4: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:29:16.839365 1415 raft_consensus.cc:1261] T 13af1b96121c492c9373f9acbb55725a P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Refusing update from remote peer 153026dd7f8943cc93fedf6f48450fb4: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:29:16.839435 4789 consensus_queue.cc:1035] T 13af1b96121c492c9373f9acbb55725a P 153026dd7f8943cc93fedf6f48450fb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:29:16.840135 1685 raft_consensus.cc:1261] T 13af1b96121c492c9373f9acbb55725a P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Refusing update from remote peer 153026dd7f8943cc93fedf6f48450fb4: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:29:16.840632 5907 consensus_queue.cc:1035] T 13af1b96121c492c9373f9acbb55725a P 153026dd7f8943cc93fedf6f48450fb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:29:16.841269 5907 consensus_queue.cc:1035] T 13af1b96121c492c9373f9acbb55725a P 153026dd7f8943cc93fedf6f48450fb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:29:16.844570 1954 raft_consensus.cc:1261] T 13af1b96121c492c9373f9acbb55725a P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Refusing update from remote peer 153026dd7f8943cc93fedf6f48450fb4: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:29:16.845297 4789 consensus_queue.cc:1035] T 13af1b96121c492c9373f9acbb55725a P 153026dd7f8943cc93fedf6f48450fb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240502 21:29:16.855201 1751 consensus_peers.cc:485] T 13af1b96121c492c9373f9acbb55725a P 153026dd7f8943cc93fedf6f48450fb4 -> Peer 16a97c1394f346f0b987f97cce7371b6 (127.15.255.137:43717): Couldn't send request to peer 16a97c1394f346f0b987f97cce7371b6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 13af1b96121c492c9373f9acbb55725a. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:29:16.856745 1550 raft_consensus.cc:2932] T 13af1b96121c492c9373f9acbb55725a P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 16a97c1394f346f0b987f97cce7371b6 (127.15.255.137) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { replace: true } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: NON_VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: true } } }
I20240502 21:29:16.861133 2087 raft_consensus.cc:2932] T 13af1b96121c492c9373f9acbb55725a P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 16a97c1394f346f0b987f97cce7371b6 (127.15.255.137) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { replace: true } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: NON_VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: true } } }
I20240502 21:29:16.866144 1262 catalog_manager.cc:5484] T 13af1b96121c492c9373f9acbb55725a P a62a935ff43d494ca35d0c417943133d reported cstate change: config changed from index 7 to 8, NON_VOTER 16a97c1394f346f0b987f97cce7371b6 (127.15.255.137) added. New cstate: current_term: 1 leader_uuid: "153026dd7f8943cc93fedf6f48450fb4" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { replace: true } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: NON_VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: true } } }
I20240502 21:29:16.856120 5911 raft_consensus.cc:2932] T 13af1b96121c492c9373f9acbb55725a P 153026dd7f8943cc93fedf6f48450fb4 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 16a97c1394f346f0b987f97cce7371b6 (127.15.255.137) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { replace: true } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: NON_VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: true } } }
I20240502 21:29:16.871495 5917 ts_tablet_manager.cc:902] T 8f86d1517df74ead9e397ca9bed44c3f P d8d7297186694f77bb98de9174e4279c: Initiating tablet copy from peer 521bb93035324cecbebed2f6f2a630f8 (127.15.255.134:40821)
I20240502 21:29:16.873451 3856 raft_consensus.cc:2932] T 13af1b96121c492c9373f9acbb55725a P d8d7297186694f77bb98de9174e4279c [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 16a97c1394f346f0b987f97cce7371b6 (127.15.255.137) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { replace: true } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: NON_VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: true } } }
I20240502 21:29:16.876233 1414 raft_consensus.cc:2932] T 13af1b96121c492c9373f9acbb55725a P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 16a97c1394f346f0b987f97cce7371b6 (127.15.255.137) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { replace: true } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: NON_VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: true } } }
I20240502 21:29:16.886464 1954 raft_consensus.cc:2932] T 13af1b96121c492c9373f9acbb55725a P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 16a97c1394f346f0b987f97cce7371b6 (127.15.255.137) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { replace: true } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: NON_VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: true } } }
I20240502 21:29:16.896754 1685 raft_consensus.cc:2932] T 13af1b96121c492c9373f9acbb55725a P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 16a97c1394f346f0b987f97cce7371b6 (127.15.255.137) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { replace: true } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: NON_VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: true } } }
I20240502 21:29:16.939507 5917 tablet_copy_client.cc:308] T 8f86d1517df74ead9e397ca9bed44c3f P d8d7297186694f77bb98de9174e4279c: tablet copy: Beginning tablet copy session from remote peer at address 127.15.255.134:40821
I20240502 21:29:16.940165 2113 tablet_copy_service.cc:140] P 521bb93035324cecbebed2f6f2a630f8: Received BeginTabletCopySession request for tablet 8f86d1517df74ead9e397ca9bed44c3f from peer d8d7297186694f77bb98de9174e4279c ({username='slave'} at 127.15.255.136:49789)
I20240502 21:29:16.940372 2113 tablet_copy_service.cc:161] P 521bb93035324cecbebed2f6f2a630f8: Beginning new tablet copy session on tablet 8f86d1517df74ead9e397ca9bed44c3f from peer d8d7297186694f77bb98de9174e4279c at {username='slave'} at 127.15.255.136:49789: session id = d8d7297186694f77bb98de9174e4279c-8f86d1517df74ead9e397ca9bed44c3f
I20240502 21:29:16.943647 2113 tablet_copy_source_session.cc:205] T 8f86d1517df74ead9e397ca9bed44c3f P 521bb93035324cecbebed2f6f2a630f8: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:29:16.947118 5917 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f86d1517df74ead9e397ca9bed44c3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:29:16.953357 5917 tablet_copy_client.cc:783] T 8f86d1517df74ead9e397ca9bed44c3f P d8d7297186694f77bb98de9174e4279c: tablet copy: Starting download of 0 data blocks...
I20240502 21:29:16.953629 5917 tablet_copy_client.cc:647] T 8f86d1517df74ead9e397ca9bed44c3f P d8d7297186694f77bb98de9174e4279c: tablet copy: Starting download of 1 WAL segments...
I20240502 21:29:16.955847 5917 tablet_copy_client.cc:523] T 8f86d1517df74ead9e397ca9bed44c3f P d8d7297186694f77bb98de9174e4279c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:29:16.959069 5917 tablet_bootstrap.cc:492] T 8f86d1517df74ead9e397ca9bed44c3f P d8d7297186694f77bb98de9174e4279c: Bootstrap starting.
I20240502 21:29:16.964887 5937 ts_tablet_manager.cc:902] T e44bf56d64b24e328803ebd0a3a54ceb P d8d7297186694f77bb98de9174e4279c: Initiating tablet copy from peer 521bb93035324cecbebed2f6f2a630f8 (127.15.255.134:40821)
I20240502 21:29:16.970340 5866 ts_tablet_manager.cc:902] T f8de1fd0869348e6b0aa131b958ce822 P 16a97c1394f346f0b987f97cce7371b6: Initiating tablet copy from peer 521bb93035324cecbebed2f6f2a630f8 (127.15.255.134:40821)
I20240502 21:29:16.979473 5937 tablet_copy_client.cc:308] T e44bf56d64b24e328803ebd0a3a54ceb P d8d7297186694f77bb98de9174e4279c: tablet copy: Beginning tablet copy session from remote peer at address 127.15.255.134:40821
I20240502 21:29:16.980376 2113 tablet_copy_service.cc:140] P 521bb93035324cecbebed2f6f2a630f8: Received BeginTabletCopySession request for tablet e44bf56d64b24e328803ebd0a3a54ceb from peer d8d7297186694f77bb98de9174e4279c ({username='slave'} at 127.15.255.136:49789)
I20240502 21:29:16.980556 2113 tablet_copy_service.cc:161] P 521bb93035324cecbebed2f6f2a630f8: Beginning new tablet copy session on tablet e44bf56d64b24e328803ebd0a3a54ceb from peer d8d7297186694f77bb98de9174e4279c at {username='slave'} at 127.15.255.136:49789: session id = d8d7297186694f77bb98de9174e4279c-e44bf56d64b24e328803ebd0a3a54ceb
I20240502 21:29:16.984320 2113 tablet_copy_source_session.cc:205] T e44bf56d64b24e328803ebd0a3a54ceb P 521bb93035324cecbebed2f6f2a630f8: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:29:16.985754 5937 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e44bf56d64b24e328803ebd0a3a54ceb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:29:16.993252 5937 tablet_copy_client.cc:783] T e44bf56d64b24e328803ebd0a3a54ceb P d8d7297186694f77bb98de9174e4279c: tablet copy: Starting download of 0 data blocks...
I20240502 21:29:16.993573 5937 tablet_copy_client.cc:647] T e44bf56d64b24e328803ebd0a3a54ceb P d8d7297186694f77bb98de9174e4279c: tablet copy: Starting download of 1 WAL segments...
I20240502 21:29:16.996078 5937 tablet_copy_client.cc:523] T e44bf56d64b24e328803ebd0a3a54ceb P d8d7297186694f77bb98de9174e4279c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:29:16.999693 5937 tablet_bootstrap.cc:492] T e44bf56d64b24e328803ebd0a3a54ceb P d8d7297186694f77bb98de9174e4279c: Bootstrap starting.
I20240502 21:29:17.023434 5917 tablet_bootstrap.cc:492] T 8f86d1517df74ead9e397ca9bed44c3f P d8d7297186694f77bb98de9174e4279c: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:29:17.023926 5917 tablet_bootstrap.cc:492] T 8f86d1517df74ead9e397ca9bed44c3f P d8d7297186694f77bb98de9174e4279c: Bootstrap complete.
I20240502 21:29:17.024221 5917 ts_tablet_manager.cc:1367] T 8f86d1517df74ead9e397ca9bed44c3f P d8d7297186694f77bb98de9174e4279c: Time spent bootstrapping tablet: real 0.065s user 0.055s sys 0.008s
I20240502 21:29:17.025033 5917 raft_consensus.cc:348] T 8f86d1517df74ead9e397ca9bed44c3f P d8d7297186694f77bb98de9174e4279c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: true } }
I20240502 21:29:17.025400 5917 raft_consensus.cc:729] T 8f86d1517df74ead9e397ca9bed44c3f P d8d7297186694f77bb98de9174e4279c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d8d7297186694f77bb98de9174e4279c, State: Initialized, Role: LEARNER
I20240502 21:29:17.025640 5917 consensus_queue.cc:260] T 8f86d1517df74ead9e397ca9bed44c3f P d8d7297186694f77bb98de9174e4279c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: true } }
I20240502 21:29:17.026386 5917 ts_tablet_manager.cc:1398] T 8f86d1517df74ead9e397ca9bed44c3f P d8d7297186694f77bb98de9174e4279c: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240502 21:29:17.027890 2113 tablet_copy_service.cc:342] P 521bb93035324cecbebed2f6f2a630f8: Request end of tablet copy session d8d7297186694f77bb98de9174e4279c-8f86d1517df74ead9e397ca9bed44c3f received from {username='slave'} at 127.15.255.136:49789
I20240502 21:29:17.028060 2113 tablet_copy_service.cc:433] P 521bb93035324cecbebed2f6f2a630f8: ending tablet copy session d8d7297186694f77bb98de9174e4279c-8f86d1517df74ead9e397ca9bed44c3f on tablet 8f86d1517df74ead9e397ca9bed44c3f with peer d8d7297186694f77bb98de9174e4279c
I20240502 21:29:17.057353 5866 tablet_copy_client.cc:308] T f8de1fd0869348e6b0aa131b958ce822 P 16a97c1394f346f0b987f97cce7371b6: tablet copy: Beginning tablet copy session from remote peer at address 127.15.255.134:40821
I20240502 21:29:17.058058 2113 tablet_copy_service.cc:140] P 521bb93035324cecbebed2f6f2a630f8: Received BeginTabletCopySession request for tablet f8de1fd0869348e6b0aa131b958ce822 from peer 16a97c1394f346f0b987f97cce7371b6 ({username='slave'} at 127.15.255.137:45019)
I20240502 21:29:17.058249 2113 tablet_copy_service.cc:161] P 521bb93035324cecbebed2f6f2a630f8: Beginning new tablet copy session on tablet f8de1fd0869348e6b0aa131b958ce822 from peer 16a97c1394f346f0b987f97cce7371b6 at {username='slave'} at 127.15.255.137:45019: session id = 16a97c1394f346f0b987f97cce7371b6-f8de1fd0869348e6b0aa131b958ce822
I20240502 21:29:17.061234 2113 tablet_copy_source_session.cc:205] T f8de1fd0869348e6b0aa131b958ce822 P 521bb93035324cecbebed2f6f2a630f8: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:29:17.062098 5876 ts_tablet_manager.cc:902] T 05e885ae741c43808c6dfbcdc5769942 P 1fab5794cf1740288abd82f2cebb9b1a: Initiating tablet copy from peer 6ab8df3c796f4b6cb3e29d5045ad3ea0 (127.15.255.135:34701)
I20240502 21:29:17.073123 5866 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8de1fd0869348e6b0aa131b958ce822. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:29:17.075721 5955 ts_tablet_manager.cc:902] T 1aa57620996e42b6adf941e9d606f7e2 P 16a97c1394f346f0b987f97cce7371b6: Initiating tablet copy from peer 521bb93035324cecbebed2f6f2a630f8 (127.15.255.134:40821)
I20240502 21:29:17.092522 5866 tablet_copy_client.cc:783] T f8de1fd0869348e6b0aa131b958ce822 P 16a97c1394f346f0b987f97cce7371b6: tablet copy: Starting download of 0 data blocks...
I20240502 21:29:17.092787 5866 tablet_copy_client.cc:647] T f8de1fd0869348e6b0aa131b958ce822 P 16a97c1394f346f0b987f97cce7371b6: tablet copy: Starting download of 1 WAL segments...
I20240502 21:29:17.095026 5866 tablet_copy_client.cc:523] T f8de1fd0869348e6b0aa131b958ce822 P 16a97c1394f346f0b987f97cce7371b6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:29:17.111824 5955 tablet_copy_client.cc:308] T 1aa57620996e42b6adf941e9d606f7e2 P 16a97c1394f346f0b987f97cce7371b6: tablet copy: Beginning tablet copy session from remote peer at address 127.15.255.134:40821
I20240502 21:29:17.113678 5866 tablet_bootstrap.cc:492] T f8de1fd0869348e6b0aa131b958ce822 P 16a97c1394f346f0b987f97cce7371b6: Bootstrap starting.
I20240502 21:29:17.115729 2113 tablet_copy_service.cc:140] P 521bb93035324cecbebed2f6f2a630f8: Received BeginTabletCopySession request for tablet 1aa57620996e42b6adf941e9d606f7e2 from peer 16a97c1394f346f0b987f97cce7371b6 ({username='slave'} at 127.15.255.137:45019)
I20240502 21:29:17.115890 2113 tablet_copy_service.cc:161] P 521bb93035324cecbebed2f6f2a630f8: Beginning new tablet copy session on tablet 1aa57620996e42b6adf941e9d606f7e2 from peer 16a97c1394f346f0b987f97cce7371b6 at {username='slave'} at 127.15.255.137:45019: session id = 16a97c1394f346f0b987f97cce7371b6-1aa57620996e42b6adf941e9d606f7e2
I20240502 21:29:17.119238 2113 tablet_copy_source_session.cc:205] T 1aa57620996e42b6adf941e9d606f7e2 P 521bb93035324cecbebed2f6f2a630f8: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:29:17.121346 5955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1aa57620996e42b6adf941e9d606f7e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:29:17.125972 5876 tablet_copy_client.cc:308] T 05e885ae741c43808c6dfbcdc5769942 P 1fab5794cf1740288abd82f2cebb9b1a: tablet copy: Beginning tablet copy session from remote peer at address 127.15.255.135:34701
I20240502 21:29:17.127696 2249 tablet_copy_service.cc:140] P 6ab8df3c796f4b6cb3e29d5045ad3ea0: Received BeginTabletCopySession request for tablet 05e885ae741c43808c6dfbcdc5769942 from peer 1fab5794cf1740288abd82f2cebb9b1a ({username='slave'} at 127.15.255.138:49711)
I20240502 21:29:17.127892 2249 tablet_copy_service.cc:161] P 6ab8df3c796f4b6cb3e29d5045ad3ea0: Beginning new tablet copy session on tablet 05e885ae741c43808c6dfbcdc5769942 from peer 1fab5794cf1740288abd82f2cebb9b1a at {username='slave'} at 127.15.255.138:49711: session id = 1fab5794cf1740288abd82f2cebb9b1a-05e885ae741c43808c6dfbcdc5769942
I20240502 21:29:17.131839 5955 tablet_copy_client.cc:783] T 1aa57620996e42b6adf941e9d606f7e2 P 16a97c1394f346f0b987f97cce7371b6: tablet copy: Starting download of 0 data blocks...
I20240502 21:29:17.132112 5955 tablet_copy_client.cc:647] T 1aa57620996e42b6adf941e9d606f7e2 P 16a97c1394f346f0b987f97cce7371b6: tablet copy: Starting download of 1 WAL segments...
I20240502 21:29:17.132371 2249 tablet_copy_source_session.cc:205] T 05e885ae741c43808c6dfbcdc5769942 P 6ab8df3c796f4b6cb3e29d5045ad3ea0: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:29:17.134186 5876 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05e885ae741c43808c6dfbcdc5769942. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:29:17.134191 5955 tablet_copy_client.cc:523] T 1aa57620996e42b6adf941e9d606f7e2 P 16a97c1394f346f0b987f97cce7371b6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:29:17.138991 5955 tablet_bootstrap.cc:492] T 1aa57620996e42b6adf941e9d606f7e2 P 16a97c1394f346f0b987f97cce7371b6: Bootstrap starting.
I20240502 21:29:17.141170 5876 tablet_copy_client.cc:783] T 05e885ae741c43808c6dfbcdc5769942 P 1fab5794cf1740288abd82f2cebb9b1a: tablet copy: Starting download of 0 data blocks...
I20240502 21:29:17.141388 5876 tablet_copy_client.cc:647] T 05e885ae741c43808c6dfbcdc5769942 P 1fab5794cf1740288abd82f2cebb9b1a: tablet copy: Starting download of 1 WAL segments...
I20240502 21:29:17.143092 5876 tablet_copy_client.cc:523] T 05e885ae741c43808c6dfbcdc5769942 P 1fab5794cf1740288abd82f2cebb9b1a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:29:17.147112 5876 tablet_bootstrap.cc:492] T 05e885ae741c43808c6dfbcdc5769942 P 1fab5794cf1740288abd82f2cebb9b1a: Bootstrap starting.
I20240502 21:29:17.157413 5917 ts_tablet_manager.cc:902] T aae7b5a01a0543afb1b5a7f44a9a864c P d8d7297186694f77bb98de9174e4279c: Initiating tablet copy from peer 7d1e5c1b047443cbb28d7b0a9744babf (127.15.255.129:37897)
I20240502 21:29:17.173199 5937 tablet_bootstrap.cc:492] T e44bf56d64b24e328803ebd0a3a54ceb P d8d7297186694f77bb98de9174e4279c: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:29:17.175779 5937 tablet_bootstrap.cc:492] T e44bf56d64b24e328803ebd0a3a54ceb P d8d7297186694f77bb98de9174e4279c: Bootstrap complete.
I20240502 21:29:17.176136 5937 ts_tablet_manager.cc:1367] T e44bf56d64b24e328803ebd0a3a54ceb P d8d7297186694f77bb98de9174e4279c: Time spent bootstrapping tablet: real 0.177s user 0.048s sys 0.012s
I20240502 21:29:17.177321 5937 raft_consensus.cc:348] T e44bf56d64b24e328803ebd0a3a54ceb P d8d7297186694f77bb98de9174e4279c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { replace: true } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: true } }
I20240502 21:29:17.178789 5937 raft_consensus.cc:729] T e44bf56d64b24e328803ebd0a3a54ceb P d8d7297186694f77bb98de9174e4279c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d8d7297186694f77bb98de9174e4279c, State: Initialized, Role: LEARNER
I20240502 21:29:17.179100 5937 consensus_queue.cc:260] T e44bf56d64b24e328803ebd0a3a54ceb P d8d7297186694f77bb98de9174e4279c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { replace: true } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: true } }
I20240502 21:29:17.182965 5937 ts_tablet_manager.cc:1398] T e44bf56d64b24e328803ebd0a3a54ceb P d8d7297186694f77bb98de9174e4279c: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240502 21:29:17.186079 2113 tablet_copy_service.cc:342] P 521bb93035324cecbebed2f6f2a630f8: Request end of tablet copy session d8d7297186694f77bb98de9174e4279c-e44bf56d64b24e328803ebd0a3a54ceb received from {username='slave'} at 127.15.255.136:49789
I20240502 21:29:17.186232 2113 tablet_copy_service.cc:433] P 521bb93035324cecbebed2f6f2a630f8: ending tablet copy session d8d7297186694f77bb98de9174e4279c-e44bf56d64b24e328803ebd0a3a54ceb on tablet e44bf56d64b24e328803ebd0a3a54ceb with peer d8d7297186694f77bb98de9174e4279c
I20240502 21:29:17.223074 5955 tablet_bootstrap.cc:492] T 1aa57620996e42b6adf941e9d606f7e2 P 16a97c1394f346f0b987f97cce7371b6: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:29:17.223527 5955 tablet_bootstrap.cc:492] T 1aa57620996e42b6adf941e9d606f7e2 P 16a97c1394f346f0b987f97cce7371b6: Bootstrap complete.
I20240502 21:29:17.223838 5955 ts_tablet_manager.cc:1367] T 1aa57620996e42b6adf941e9d606f7e2 P 16a97c1394f346f0b987f97cce7371b6: Time spent bootstrapping tablet: real 0.085s user 0.053s sys 0.004s
I20240502 21:29:17.224725 5955 raft_consensus.cc:348] T 1aa57620996e42b6adf941e9d606f7e2 P 16a97c1394f346f0b987f97cce7371b6 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: NON_VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: true } }
I20240502 21:29:17.225059 5955 raft_consensus.cc:729] T 1aa57620996e42b6adf941e9d606f7e2 P 16a97c1394f346f0b987f97cce7371b6 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 16a97c1394f346f0b987f97cce7371b6, State: Initialized, Role: LEARNER
I20240502 21:29:17.225279 5955 consensus_queue.cc:260] T 1aa57620996e42b6adf941e9d606f7e2 P 16a97c1394f346f0b987f97cce7371b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: NON_VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: true } }
I20240502 21:29:17.226761 5955 ts_tablet_manager.cc:1398] T 1aa57620996e42b6adf941e9d606f7e2 P 16a97c1394f346f0b987f97cce7371b6: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20240502 21:29:17.227700 2113 tablet_copy_service.cc:342] P 521bb93035324cecbebed2f6f2a630f8: Request end of tablet copy session 16a97c1394f346f0b987f97cce7371b6-1aa57620996e42b6adf941e9d606f7e2 received from {username='slave'} at 127.15.255.137:45019
I20240502 21:29:17.227844 2113 tablet_copy_service.cc:433] P 521bb93035324cecbebed2f6f2a630f8: ending tablet copy session 16a97c1394f346f0b987f97cce7371b6-1aa57620996e42b6adf941e9d606f7e2 on tablet 1aa57620996e42b6adf941e9d606f7e2 with peer 16a97c1394f346f0b987f97cce7371b6
I20240502 21:29:17.231118 5917 tablet_copy_client.cc:308] T aae7b5a01a0543afb1b5a7f44a9a864c P d8d7297186694f77bb98de9174e4279c: tablet copy: Beginning tablet copy session from remote peer at address 127.15.255.129:37897
I20240502 21:29:17.231987 1434 tablet_copy_service.cc:140] P 7d1e5c1b047443cbb28d7b0a9744babf: Received BeginTabletCopySession request for tablet aae7b5a01a0543afb1b5a7f44a9a864c from peer d8d7297186694f77bb98de9174e4279c ({username='slave'} at 127.15.255.136:40527)
I20240502 21:29:17.232198 1434 tablet_copy_service.cc:161] P 7d1e5c1b047443cbb28d7b0a9744babf: Beginning new tablet copy session on tablet aae7b5a01a0543afb1b5a7f44a9a864c from peer d8d7297186694f77bb98de9174e4279c at {username='slave'} at 127.15.255.136:40527: session id = d8d7297186694f77bb98de9174e4279c-aae7b5a01a0543afb1b5a7f44a9a864c
I20240502 21:29:17.234855 3987 raft_consensus.cc:1203] T f8bee9bdcbff404280f000bdaeadf465 P 16a97c1394f346f0b987f97cce7371b6 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240502 21:29:17.235248 1434 tablet_copy_source_session.cc:205] T aae7b5a01a0543afb1b5a7f44a9a864c P 7d1e5c1b047443cbb28d7b0a9744babf: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:29:17.237951 5917 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aae7b5a01a0543afb1b5a7f44a9a864c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:29:17.245168 5917 tablet_copy_client.cc:783] T aae7b5a01a0543afb1b5a7f44a9a864c P d8d7297186694f77bb98de9174e4279c: tablet copy: Starting download of 0 data blocks...
I20240502 21:29:17.245479 5917 tablet_copy_client.cc:647] T aae7b5a01a0543afb1b5a7f44a9a864c P d8d7297186694f77bb98de9174e4279c: tablet copy: Starting download of 1 WAL segments...
I20240502 21:29:17.247490 5917 tablet_copy_client.cc:523] T aae7b5a01a0543afb1b5a7f44a9a864c P d8d7297186694f77bb98de9174e4279c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:29:17.251916 5917 tablet_bootstrap.cc:492] T aae7b5a01a0543afb1b5a7f44a9a864c P d8d7297186694f77bb98de9174e4279c: Bootstrap starting.
I20240502 21:29:17.252527 5866 tablet_bootstrap.cc:492] T f8de1fd0869348e6b0aa131b958ce822 P 16a97c1394f346f0b987f97cce7371b6: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:29:17.252974 5866 tablet_bootstrap.cc:492] T f8de1fd0869348e6b0aa131b958ce822 P 16a97c1394f346f0b987f97cce7371b6: Bootstrap complete.
I20240502 21:29:17.253221 5866 ts_tablet_manager.cc:1367] T f8de1fd0869348e6b0aa131b958ce822 P 16a97c1394f346f0b987f97cce7371b6: Time spent bootstrapping tablet: real 0.141s user 0.061s sys 0.001s
I20240502 21:29:17.253928 5866 raft_consensus.cc:348] T f8de1fd0869348e6b0aa131b958ce822 P 16a97c1394f346f0b987f97cce7371b6 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { replace: true } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: NON_VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: true } }
I20240502 21:29:17.254237 5866 raft_consensus.cc:729] T f8de1fd0869348e6b0aa131b958ce822 P 16a97c1394f346f0b987f97cce7371b6 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 16a97c1394f346f0b987f97cce7371b6, State: Initialized, Role: LEARNER
I20240502 21:29:17.254500 5866 consensus_queue.cc:260] T f8de1fd0869348e6b0aa131b958ce822 P 16a97c1394f346f0b987f97cce7371b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { replace: true } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: NON_VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: true } }
I20240502 21:29:17.255101 5866 ts_tablet_manager.cc:1398] T f8de1fd0869348e6b0aa131b958ce822 P 16a97c1394f346f0b987f97cce7371b6: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 21:29:17.257769 5975 ts_tablet_manager.cc:902] T 444c1cba2a6f4729aa01b75d06b6b517 P 1fab5794cf1740288abd82f2cebb9b1a: Initiating tablet copy from peer e632c553ff4341328acf1cbaca678392 (127.15.255.133:37439)
I20240502 21:29:17.258417 2113 tablet_copy_service.cc:342] P 521bb93035324cecbebed2f6f2a630f8: Request end of tablet copy session 16a97c1394f346f0b987f97cce7371b6-f8de1fd0869348e6b0aa131b958ce822 received from {username='slave'} at 127.15.255.137:45019
I20240502 21:29:17.258564 2113 tablet_copy_service.cc:433] P 521bb93035324cecbebed2f6f2a630f8: ending tablet copy session 16a97c1394f346f0b987f97cce7371b6-f8de1fd0869348e6b0aa131b958ce822 on tablet f8de1fd0869348e6b0aa131b958ce822 with peer 16a97c1394f346f0b987f97cce7371b6
I20240502 21:29:17.295769 4119 raft_consensus.cc:1203] T 72355dd734ea4aeba031171e8c39696d P 1fab5794cf1740288abd82f2cebb9b1a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240502 21:29:17.306854 5876 tablet_bootstrap.cc:492] T 05e885ae741c43808c6dfbcdc5769942 P 1fab5794cf1740288abd82f2cebb9b1a: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:29:17.307340 5876 tablet_bootstrap.cc:492] T 05e885ae741c43808c6dfbcdc5769942 P 1fab5794cf1740288abd82f2cebb9b1a: Bootstrap complete.
I20240502 21:29:17.307610 5876 ts_tablet_manager.cc:1367] T 05e885ae741c43808c6dfbcdc5769942 P 1fab5794cf1740288abd82f2cebb9b1a: Time spent bootstrapping tablet: real 0.161s user 0.046s sys 0.011s
I20240502 21:29:17.308401 5876 raft_consensus.cc:348] T 05e885ae741c43808c6dfbcdc5769942 P 1fab5794cf1740288abd82f2cebb9b1a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: NON_VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: true } }
I20240502 21:29:17.311470 5876 raft_consensus.cc:729] T 05e885ae741c43808c6dfbcdc5769942 P 1fab5794cf1740288abd82f2cebb9b1a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1fab5794cf1740288abd82f2cebb9b1a, State: Initialized, Role: LEARNER
I20240502 21:29:17.311726 5876 consensus_queue.cc:260] T 05e885ae741c43808c6dfbcdc5769942 P 1fab5794cf1740288abd82f2cebb9b1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: NON_VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: true } }
I20240502 21:29:17.313408 5917 tablet_bootstrap.cc:492] T aae7b5a01a0543afb1b5a7f44a9a864c P d8d7297186694f77bb98de9174e4279c: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:29:17.313863 5876 ts_tablet_manager.cc:1398] T 05e885ae741c43808c6dfbcdc5769942 P 1fab5794cf1740288abd82f2cebb9b1a: Time spent starting tablet: real 0.006s user 0.002s sys 0.001s
I20240502 21:29:17.313889 5917 tablet_bootstrap.cc:492] T aae7b5a01a0543afb1b5a7f44a9a864c P d8d7297186694f77bb98de9174e4279c: Bootstrap complete.
I20240502 21:29:17.315137 2249 tablet_copy_service.cc:342] P 6ab8df3c796f4b6cb3e29d5045ad3ea0: Request end of tablet copy session 1fab5794cf1740288abd82f2cebb9b1a-05e885ae741c43808c6dfbcdc5769942 received from {username='slave'} at 127.15.255.138:49711
I20240502 21:29:17.315269 2249 tablet_copy_service.cc:433] P 6ab8df3c796f4b6cb3e29d5045ad3ea0: ending tablet copy session 1fab5794cf1740288abd82f2cebb9b1a-05e885ae741c43808c6dfbcdc5769942 on tablet 05e885ae741c43808c6dfbcdc5769942 with peer 1fab5794cf1740288abd82f2cebb9b1a
I20240502 21:29:17.315691 5917 ts_tablet_manager.cc:1367] T aae7b5a01a0543afb1b5a7f44a9a864c P d8d7297186694f77bb98de9174e4279c: Time spent bootstrapping tablet: real 0.064s user 0.049s sys 0.003s
I20240502 21:29:17.317160 5917 raft_consensus.cc:348] T aae7b5a01a0543afb1b5a7f44a9a864c P d8d7297186694f77bb98de9174e4279c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: true } }
I20240502 21:29:17.317660 5917 raft_consensus.cc:729] T aae7b5a01a0543afb1b5a7f44a9a864c P d8d7297186694f77bb98de9174e4279c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d8d7297186694f77bb98de9174e4279c, State: Initialized, Role: LEARNER
I20240502 21:29:17.317915 5917 consensus_queue.cc:260] T aae7b5a01a0543afb1b5a7f44a9a864c P d8d7297186694f77bb98de9174e4279c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: true } }
I20240502 21:29:17.318535 5917 ts_tablet_manager.cc:1398] T aae7b5a01a0543afb1b5a7f44a9a864c P d8d7297186694f77bb98de9174e4279c: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 21:29:17.319528 1434 tablet_copy_service.cc:342] P 7d1e5c1b047443cbb28d7b0a9744babf: Request end of tablet copy session d8d7297186694f77bb98de9174e4279c-aae7b5a01a0543afb1b5a7f44a9a864c received from {username='slave'} at 127.15.255.136:40527
I20240502 21:29:17.319638 1434 tablet_copy_service.cc:433] P 7d1e5c1b047443cbb28d7b0a9744babf: ending tablet copy session d8d7297186694f77bb98de9174e4279c-aae7b5a01a0543afb1b5a7f44a9a864c on tablet aae7b5a01a0543afb1b5a7f44a9a864c with peer d8d7297186694f77bb98de9174e4279c
I20240502 21:29:17.323515 5975 tablet_copy_client.cc:308] T 444c1cba2a6f4729aa01b75d06b6b517 P 1fab5794cf1740288abd82f2cebb9b1a: tablet copy: Beginning tablet copy session from remote peer at address 127.15.255.133:37439
I20240502 21:29:17.324219 1975 tablet_copy_service.cc:140] P e632c553ff4341328acf1cbaca678392: Received BeginTabletCopySession request for tablet 444c1cba2a6f4729aa01b75d06b6b517 from peer 1fab5794cf1740288abd82f2cebb9b1a ({username='slave'} at 127.15.255.138:55671)
I20240502 21:29:17.324389 1975 tablet_copy_service.cc:161] P e632c553ff4341328acf1cbaca678392: Beginning new tablet copy session on tablet 444c1cba2a6f4729aa01b75d06b6b517 from peer 1fab5794cf1740288abd82f2cebb9b1a at {username='slave'} at 127.15.255.138:55671: session id = 1fab5794cf1740288abd82f2cebb9b1a-444c1cba2a6f4729aa01b75d06b6b517
I20240502 21:29:17.326894 1975 tablet_copy_source_session.cc:205] T 444c1cba2a6f4729aa01b75d06b6b517 P e632c553ff4341328acf1cbaca678392: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:29:17.328673 5975 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 444c1cba2a6f4729aa01b75d06b6b517. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:29:17.335518 5975 tablet_copy_client.cc:783] T 444c1cba2a6f4729aa01b75d06b6b517 P 1fab5794cf1740288abd82f2cebb9b1a: tablet copy: Starting download of 0 data blocks...
I20240502 21:29:17.335904 5975 tablet_copy_client.cc:647] T 444c1cba2a6f4729aa01b75d06b6b517 P 1fab5794cf1740288abd82f2cebb9b1a: tablet copy: Starting download of 1 WAL segments...
I20240502 21:29:17.340243 5866 ts_tablet_manager.cc:902] T 13af1b96121c492c9373f9acbb55725a P 16a97c1394f346f0b987f97cce7371b6: Initiating tablet copy from peer 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132:44947)
I20240502 21:29:17.340620 5975 tablet_copy_client.cc:523] T 444c1cba2a6f4729aa01b75d06b6b517 P 1fab5794cf1740288abd82f2cebb9b1a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:29:17.344686 5975 tablet_bootstrap.cc:492] T 444c1cba2a6f4729aa01b75d06b6b517 P 1fab5794cf1740288abd82f2cebb9b1a: Bootstrap starting.
I20240502 21:29:17.384760 5866 tablet_copy_client.cc:308] T 13af1b96121c492c9373f9acbb55725a P 16a97c1394f346f0b987f97cce7371b6: tablet copy: Beginning tablet copy session from remote peer at address 127.15.255.132:44947
I20240502 21:29:17.386955 3856 raft_consensus.cc:1203] T 8f86d1517df74ead9e397ca9bed44c3f P d8d7297186694f77bb98de9174e4279c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240502 21:29:17.393790 1840 tablet_copy_service.cc:140] P 153026dd7f8943cc93fedf6f48450fb4: Received BeginTabletCopySession request for tablet 13af1b96121c492c9373f9acbb55725a from peer 16a97c1394f346f0b987f97cce7371b6 ({username='slave'} at 127.15.255.137:59005)
I20240502 21:29:17.393980 1840 tablet_copy_service.cc:161] P 153026dd7f8943cc93fedf6f48450fb4: Beginning new tablet copy session on tablet 13af1b96121c492c9373f9acbb55725a from peer 16a97c1394f346f0b987f97cce7371b6 at {username='slave'} at 127.15.255.137:59005: session id = 16a97c1394f346f0b987f97cce7371b6-13af1b96121c492c9373f9acbb55725a
I20240502 21:29:17.398135 1840 tablet_copy_source_session.cc:205] T 13af1b96121c492c9373f9acbb55725a P 153026dd7f8943cc93fedf6f48450fb4: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:29:17.400933 5866 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13af1b96121c492c9373f9acbb55725a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:29:17.406500 5866 tablet_copy_client.cc:783] T 13af1b96121c492c9373f9acbb55725a P 16a97c1394f346f0b987f97cce7371b6: tablet copy: Starting download of 0 data blocks...
I20240502 21:29:17.406723 5866 tablet_copy_client.cc:647] T 13af1b96121c492c9373f9acbb55725a P 16a97c1394f346f0b987f97cce7371b6: tablet copy: Starting download of 1 WAL segments...
I20240502 21:29:17.408608 5866 tablet_copy_client.cc:523] T 13af1b96121c492c9373f9acbb55725a P 16a97c1394f346f0b987f97cce7371b6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:29:17.410780 3987 raft_consensus.cc:1203] T f8de1fd0869348e6b0aa131b958ce822 P 16a97c1394f346f0b987f97cce7371b6 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240502 21:29:17.411958 5866 tablet_bootstrap.cc:492] T 13af1b96121c492c9373f9acbb55725a P 16a97c1394f346f0b987f97cce7371b6: Bootstrap starting.
I20240502 21:29:17.415089 5975 tablet_bootstrap.cc:492] T 444c1cba2a6f4729aa01b75d06b6b517 P 1fab5794cf1740288abd82f2cebb9b1a: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:29:17.415493 5975 tablet_bootstrap.cc:492] T 444c1cba2a6f4729aa01b75d06b6b517 P 1fab5794cf1740288abd82f2cebb9b1a: Bootstrap complete.
I20240502 21:29:17.415761 5975 ts_tablet_manager.cc:1367] T 444c1cba2a6f4729aa01b75d06b6b517 P 1fab5794cf1740288abd82f2cebb9b1a: Time spent bootstrapping tablet: real 0.071s user 0.046s sys 0.005s
I20240502 21:29:17.416496 5975 raft_consensus.cc:348] T 444c1cba2a6f4729aa01b75d06b6b517 P 1fab5794cf1740288abd82f2cebb9b1a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { replace: true } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: NON_VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: true } }
I20240502 21:29:17.416854 5975 raft_consensus.cc:729] T 444c1cba2a6f4729aa01b75d06b6b517 P 1fab5794cf1740288abd82f2cebb9b1a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1fab5794cf1740288abd82f2cebb9b1a, State: Initialized, Role: LEARNER
I20240502 21:29:17.417060 5975 consensus_queue.cc:260] T 444c1cba2a6f4729aa01b75d06b6b517 P 1fab5794cf1740288abd82f2cebb9b1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { replace: true } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: NON_VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: true } }
I20240502 21:29:17.417696 5975 ts_tablet_manager.cc:1398] T 444c1cba2a6f4729aa01b75d06b6b517 P 1fab5794cf1740288abd82f2cebb9b1a: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240502 21:29:17.418490 1975 tablet_copy_service.cc:342] P e632c553ff4341328acf1cbaca678392: Request end of tablet copy session 1fab5794cf1740288abd82f2cebb9b1a-444c1cba2a6f4729aa01b75d06b6b517 received from {username='slave'} at 127.15.255.138:55671
I20240502 21:29:17.418620 1975 tablet_copy_service.cc:433] P e632c553ff4341328acf1cbaca678392: ending tablet copy session 1fab5794cf1740288abd82f2cebb9b1a-444c1cba2a6f4729aa01b75d06b6b517 on tablet 444c1cba2a6f4729aa01b75d06b6b517 with peer 1fab5794cf1740288abd82f2cebb9b1a
I20240502 21:29:17.447075 3856 raft_consensus.cc:1203] T e44bf56d64b24e328803ebd0a3a54ceb P d8d7297186694f77bb98de9174e4279c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240502 21:29:17.466440 5866 tablet_bootstrap.cc:492] T 13af1b96121c492c9373f9acbb55725a P 16a97c1394f346f0b987f97cce7371b6: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:29:17.466840 5866 tablet_bootstrap.cc:492] T 13af1b96121c492c9373f9acbb55725a P 16a97c1394f346f0b987f97cce7371b6: Bootstrap complete.
I20240502 21:29:17.467103 5866 ts_tablet_manager.cc:1367] T 13af1b96121c492c9373f9acbb55725a P 16a97c1394f346f0b987f97cce7371b6: Time spent bootstrapping tablet: real 0.055s user 0.029s sys 0.008s
I20240502 21:29:17.467835 4119 raft_consensus.cc:1203] T 05e885ae741c43808c6dfbcdc5769942 P 1fab5794cf1740288abd82f2cebb9b1a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240502 21:29:17.467868 5866 raft_consensus.cc:348] T 13af1b96121c492c9373f9acbb55725a P 16a97c1394f346f0b987f97cce7371b6 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { replace: true } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: NON_VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: true } }
I20240502 21:29:17.468183 5866 raft_consensus.cc:729] T 13af1b96121c492c9373f9acbb55725a P 16a97c1394f346f0b987f97cce7371b6 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 16a97c1394f346f0b987f97cce7371b6, State: Initialized, Role: LEARNER
I20240502 21:29:17.468434 5866 consensus_queue.cc:260] T 13af1b96121c492c9373f9acbb55725a P 16a97c1394f346f0b987f97cce7371b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { replace: true } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: NON_VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: true } }
I20240502 21:29:17.469147 5866 ts_tablet_manager.cc:1398] T 13af1b96121c492c9373f9acbb55725a P 16a97c1394f346f0b987f97cce7371b6: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240502 21:29:17.470486 1840 tablet_copy_service.cc:342] P 153026dd7f8943cc93fedf6f48450fb4: Request end of tablet copy session 16a97c1394f346f0b987f97cce7371b6-13af1b96121c492c9373f9acbb55725a received from {username='slave'} at 127.15.255.137:59005
I20240502 21:29:17.470651 1840 tablet_copy_service.cc:433] P 153026dd7f8943cc93fedf6f48450fb4: ending tablet copy session 16a97c1394f346f0b987f97cce7371b6-13af1b96121c492c9373f9acbb55725a on tablet 13af1b96121c492c9373f9acbb55725a with peer 16a97c1394f346f0b987f97cce7371b6
I20240502 21:29:17.646780 5763 raft_consensus.cc:1050] T f8bee9bdcbff404280f000bdaeadf465 P 7d1e5c1b047443cbb28d7b0a9744babf: attempting to promote NON_VOTER 16a97c1394f346f0b987f97cce7371b6 to VOTER
I20240502 21:29:17.647943 5763 consensus_queue.cc:237] T f8bee9bdcbff404280f000bdaeadf465 P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } }
I20240502 21:29:17.651286 3987 raft_consensus.cc:1261] T f8bee9bdcbff404280f000bdaeadf465 P 16a97c1394f346f0b987f97cce7371b6 [term 1 LEARNER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:29:17.651301 3856 raft_consensus.cc:1261] T f8bee9bdcbff404280f000bdaeadf465 P d8d7297186694f77bb98de9174e4279c [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:29:17.651553 1685 raft_consensus.cc:1261] T f8bee9bdcbff404280f000bdaeadf465 P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:29:17.652745 3987 raft_consensus.cc:1203] T 1aa57620996e42b6adf941e9d606f7e2 P 16a97c1394f346f0b987f97cce7371b6 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240502 21:29:17.654676 5661 consensus_queue.cc:1035] T f8bee9bdcbff404280f000bdaeadf465 P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:29:17.655220 5661 consensus_queue.cc:1035] T f8bee9bdcbff404280f000bdaeadf465 P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:29:17.657178 3856 raft_consensus.cc:1203] T aae7b5a01a0543afb1b5a7f44a9a864c P d8d7297186694f77bb98de9174e4279c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240502 21:29:17.659250 5661 consensus_queue.cc:1035] T f8bee9bdcbff404280f000bdaeadf465 P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:29:17.673488 1954 raft_consensus.cc:1261] T f8bee9bdcbff404280f000bdaeadf465 P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:29:17.673674 2226 raft_consensus.cc:1261] T f8bee9bdcbff404280f000bdaeadf465 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:29:17.674095 5761 consensus_queue.cc:1035] T f8bee9bdcbff404280f000bdaeadf465 P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:29:17.674193 1549 raft_consensus.cc:1261] T f8bee9bdcbff404280f000bdaeadf465 P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:29:17.675312 6046 consensus_queue.cc:1035] T f8bee9bdcbff404280f000bdaeadf465 P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:29:17.676121 5761 consensus_queue.cc:1035] T f8bee9bdcbff404280f000bdaeadf465 P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:29:17.680297 2087 raft_consensus.cc:1261] T f8bee9bdcbff404280f000bdaeadf465 P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:29:17.680817 5761 consensus_queue.cc:1035] T f8bee9bdcbff404280f000bdaeadf465 P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:29:17.682312 6051 raft_consensus.cc:2932] T f8bee9bdcbff404280f000bdaeadf465 P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 16a97c1394f346f0b987f97cce7371b6 (127.15.255.137) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } }
I20240502 21:29:17.683097 1549 raft_consensus.cc:2932] T f8bee9bdcbff404280f000bdaeadf465 P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 16a97c1394f346f0b987f97cce7371b6 (127.15.255.137) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } }
I20240502 21:29:17.687945 3855 raft_consensus.cc:2932] T f8bee9bdcbff404280f000bdaeadf465 P d8d7297186694f77bb98de9174e4279c [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 16a97c1394f346f0b987f97cce7371b6 (127.15.255.137) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } }
I20240502 21:29:17.691190 1264 catalog_manager.cc:5484] T f8bee9bdcbff404280f000bdaeadf465 P a62a935ff43d494ca35d0c417943133d reported cstate change: config changed from index 8 to 9, 16a97c1394f346f0b987f97cce7371b6 (127.15.255.137) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } }
I20240502 21:29:17.692066 3987 raft_consensus.cc:2932] T f8bee9bdcbff404280f000bdaeadf465 P 16a97c1394f346f0b987f97cce7371b6 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 16a97c1394f346f0b987f97cce7371b6 (127.15.255.137) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } }
I20240502 21:29:17.695439 1685 raft_consensus.cc:2932] T f8bee9bdcbff404280f000bdaeadf465 P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 16a97c1394f346f0b987f97cce7371b6 (127.15.255.137) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } }
I20240502 21:29:17.698204 2088 raft_consensus.cc:2932] T f8bee9bdcbff404280f000bdaeadf465 P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 16a97c1394f346f0b987f97cce7371b6 (127.15.255.137) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } }
I20240502 21:29:17.713877 1954 raft_consensus.cc:2932] T f8bee9bdcbff404280f000bdaeadf465 P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 16a97c1394f346f0b987f97cce7371b6 (127.15.255.137) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } }
I20240502 21:29:17.716195 2226 raft_consensus.cc:2932] T f8bee9bdcbff404280f000bdaeadf465 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 16a97c1394f346f0b987f97cce7371b6 (127.15.255.137) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } }
I20240502 21:29:17.727080 5661 raft_consensus.cc:1050] T 72355dd734ea4aeba031171e8c39696d P 7d1e5c1b047443cbb28d7b0a9744babf: attempting to promote NON_VOTER 1fab5794cf1740288abd82f2cebb9b1a to VOTER
I20240502 21:29:17.728231 5661 consensus_queue.cc:237] T 72355dd734ea4aeba031171e8c39696d P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { replace: true } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } }
I20240502 21:29:17.728777 1414 consensus_queue.cc:237] T f8bee9bdcbff404280f000bdaeadf465 P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } }
I20240502 21:29:17.733443 3987 raft_consensus.cc:1261] T f8bee9bdcbff404280f000bdaeadf465 P 16a97c1394f346f0b987f97cce7371b6 [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 21:29:17.734092 6046 consensus_queue.cc:1035] T f8bee9bdcbff404280f000bdaeadf465 P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 21:29:17.734264 3855 raft_consensus.cc:1261] T f8bee9bdcbff404280f000bdaeadf465 P d8d7297186694f77bb98de9174e4279c [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 21:29:17.734349 2226 raft_consensus.cc:1261] T f8bee9bdcbff404280f000bdaeadf465 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 21:29:17.734961 5761 consensus_queue.cc:1035] T f8bee9bdcbff404280f000bdaeadf465 P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 21:29:17.735482 1549 raft_consensus.cc:1261] T f8bee9bdcbff404280f000bdaeadf465 P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 21:29:17.736183 5763 consensus_queue.cc:1035] T f8bee9bdcbff404280f000bdaeadf465 P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 21:29:17.736531 6051 consensus_queue.cc:1035] T f8bee9bdcbff404280f000bdaeadf465 P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 21:29:17.738490 1685 raft_consensus.cc:1261] T 72355dd734ea4aeba031171e8c39696d P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:29:17.738675 3856 raft_consensus.cc:1261] T 72355dd734ea4aeba031171e8c39696d P d8d7297186694f77bb98de9174e4279c [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:29:17.739215 5761 consensus_queue.cc:1035] T 72355dd734ea4aeba031171e8c39696d P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:29:17.740756 5763 consensus_queue.cc:1035] T 72355dd734ea4aeba031171e8c39696d P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:29:17.740840 1954 raft_consensus.cc:1261] T f8bee9bdcbff404280f000bdaeadf465 P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 21:29:17.742750 5761 consensus_queue.cc:1035] T f8bee9bdcbff404280f000bdaeadf465 P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 21:29:17.743319 5761 raft_consensus.cc:2932] T f8bee9bdcbff404280f000bdaeadf465 P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 16b3a337bc5e462f83ba895ad8660deb (127.15.255.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } }
I20240502 21:29:17.746091 2226 raft_consensus.cc:2932] T f8bee9bdcbff404280f000bdaeadf465 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 16b3a337bc5e462f83ba895ad8660deb (127.15.255.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } }
I20240502 21:29:17.748845 1255 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f8bee9bdcbff404280f000bdaeadf465 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:29:17.748589 1549 raft_consensus.cc:2932] T f8bee9bdcbff404280f000bdaeadf465 P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 16b3a337bc5e462f83ba895ad8660deb (127.15.255.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } }
I20240502 21:29:17.754465 1262 catalog_manager.cc:5484] T f8bee9bdcbff404280f000bdaeadf465 P a62a935ff43d494ca35d0c417943133d reported cstate change: config changed from index 9 to 10, VOTER 16b3a337bc5e462f83ba895ad8660deb (127.15.255.131) evicted. New cstate: current_term: 1 leader_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } }
I20240502 21:29:17.756616 1954 raft_consensus.cc:2932] T f8bee9bdcbff404280f000bdaeadf465 P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 16b3a337bc5e462f83ba895ad8660deb (127.15.255.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } }
I20240502 21:29:17.759281 1664 tablet_service.cc:1514] Processing DeleteTablet for tablet f8bee9bdcbff404280f000bdaeadf465 with delete_type TABLET_DATA_TOMBSTONED (TS 16b3a337bc5e462f83ba895ad8660deb not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:42590
I20240502 21:29:17.762565 3987 raft_consensus.cc:2932] T f8bee9bdcbff404280f000bdaeadf465 P 16a97c1394f346f0b987f97cce7371b6 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 16b3a337bc5e462f83ba895ad8660deb (127.15.255.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } }
I20240502 21:29:17.765611 4119 raft_consensus.cc:1203] T 444c1cba2a6f4729aa01b75d06b6b517 P 1fab5794cf1740288abd82f2cebb9b1a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240502 21:29:17.766541 3856 raft_consensus.cc:2932] T f8bee9bdcbff404280f000bdaeadf465 P d8d7297186694f77bb98de9174e4279c [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 16b3a337bc5e462f83ba895ad8660deb (127.15.255.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } }
I20240502 21:29:17.775694 2087 raft_consensus.cc:1261] T f8bee9bdcbff404280f000bdaeadf465 P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 21:29:17.776325 6067 consensus_queue.cc:1035] T f8bee9bdcbff404280f000bdaeadf465 P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 21:29:17.778326 2087 raft_consensus.cc:2932] T f8bee9bdcbff404280f000bdaeadf465 P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 16b3a337bc5e462f83ba895ad8660deb (127.15.255.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } }
I20240502 21:29:17.786576 6072 tablet_replica.cc:330] T f8bee9bdcbff404280f000bdaeadf465 P 16b3a337bc5e462f83ba895ad8660deb: stopping tablet replica
I20240502 21:29:17.787222 6072 raft_consensus.cc:2229] T f8bee9bdcbff404280f000bdaeadf465 P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:29:17.791685 6072 raft_consensus.cc:2258] T f8bee9bdcbff404280f000bdaeadf465 P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:29:17.794121 6072 ts_tablet_manager.cc:1875] T f8bee9bdcbff404280f000bdaeadf465 P 16b3a337bc5e462f83ba895ad8660deb: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:29:17.801913 4119 raft_consensus.cc:1261] T 72355dd734ea4aeba031171e8c39696d P 1fab5794cf1740288abd82f2cebb9b1a [term 1 LEARNER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:29:17.801930 1817 raft_consensus.cc:1261] T 72355dd734ea4aeba031171e8c39696d P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:29:17.802515 1549 raft_consensus.cc:1261] T 72355dd734ea4aeba031171e8c39696d P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:29:17.802608 6066 consensus_queue.cc:1035] T 72355dd734ea4aeba031171e8c39696d P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:29:17.803118 6072 ts_tablet_manager.cc:1888] T f8bee9bdcbff404280f000bdaeadf465 P 16b3a337bc5e462f83ba895ad8660deb: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240502 21:29:17.803093 6046 consensus_queue.cc:1035] T 72355dd734ea4aeba031171e8c39696d P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:29:17.803279 6072 log.cc:1196] T f8bee9bdcbff404280f000bdaeadf465 P 16b3a337bc5e462f83ba895ad8660deb: Deleting WAL directory at /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-2/wal/wals/f8bee9bdcbff404280f000bdaeadf465
I20240502 21:29:17.803359 2226 raft_consensus.cc:1261] T 72355dd734ea4aeba031171e8c39696d P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:29:17.803939 6066 consensus_queue.cc:1035] T 72355dd734ea4aeba031171e8c39696d P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:29:17.804436 1257 catalog_manager.cc:4830] TS 16b3a337bc5e462f83ba895ad8660deb (127.15.255.131:36033): tablet f8bee9bdcbff404280f000bdaeadf465 (table rebalance_test_table_3 [id=a205513324dd4db5a5ee4ed1928aefa6]) successfully deleted
I20240502 21:29:17.804577 6066 consensus_queue.cc:1035] T 72355dd734ea4aeba031171e8c39696d P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:29:17.808511 6046 raft_consensus.cc:2932] T 72355dd734ea4aeba031171e8c39696d P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { replace: true } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } }
I20240502 21:29:17.809561 1954 raft_consensus.cc:1261] T 72355dd734ea4aeba031171e8c39696d P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:29:17.810038 6067 consensus_queue.cc:1035] T 72355dd734ea4aeba031171e8c39696d P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:29:17.812255 1954 raft_consensus.cc:2932] T 72355dd734ea4aeba031171e8c39696d P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { replace: true } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } }
I20240502 21:29:17.813808 2226 raft_consensus.cc:2932] T 72355dd734ea4aeba031171e8c39696d P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { replace: true } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } }
I20240502 21:29:17.814942 3855 raft_consensus.cc:2932] T 72355dd734ea4aeba031171e8c39696d P d8d7297186694f77bb98de9174e4279c [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { replace: true } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } }
I20240502 21:29:17.815089 1685 raft_consensus.cc:2932] T 72355dd734ea4aeba031171e8c39696d P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { replace: true } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } }
I20240502 21:29:17.816179 1549 raft_consensus.cc:2932] T 72355dd734ea4aeba031171e8c39696d P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { replace: true } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } }
I20240502 21:29:17.820765 1262 catalog_manager.cc:5484] T 72355dd734ea4aeba031171e8c39696d P 7d1e5c1b047443cbb28d7b0a9744babf reported cstate change: config changed from index 8 to 9, 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 21:29:17.823875 4119 raft_consensus.cc:2932] T 72355dd734ea4aeba031171e8c39696d P 1fab5794cf1740288abd82f2cebb9b1a [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { replace: true } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } }
I20240502 21:29:17.831948 3987 raft_consensus.cc:1203] T 13af1b96121c492c9373f9acbb55725a P 16a97c1394f346f0b987f97cce7371b6 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240502 21:29:17.841768 1817 raft_consensus.cc:2932] T 72355dd734ea4aeba031171e8c39696d P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { replace: true } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } }
I20240502 21:29:17.857518 5796 raft_consensus.cc:1050] T 8f86d1517df74ead9e397ca9bed44c3f P 521bb93035324cecbebed2f6f2a630f8: attempting to promote NON_VOTER d8d7297186694f77bb98de9174e4279c to VOTER
I20240502 21:29:17.858664 5796 consensus_queue.cc:237] T 8f86d1517df74ead9e397ca9bed44c3f P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } }
I20240502 21:29:17.861997 3855 raft_consensus.cc:1261] T 8f86d1517df74ead9e397ca9bed44c3f P d8d7297186694f77bb98de9174e4279c [term 1 LEARNER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:29:17.862136 4119 raft_consensus.cc:1261] T 8f86d1517df74ead9e397ca9bed44c3f P 1fab5794cf1740288abd82f2cebb9b1a [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:29:17.862548 5797 consensus_queue.cc:1035] T 8f86d1517df74ead9e397ca9bed44c3f P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:29:17.862821 5389 consensus_queue.cc:1035] T 8f86d1517df74ead9e397ca9bed44c3f P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:29:17.877418 2226 raft_consensus.cc:1261] T 8f86d1517df74ead9e397ca9bed44c3f P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:29:17.878301 1685 raft_consensus.cc:1261] T 8f86d1517df74ead9e397ca9bed44c3f P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:29:17.878830 5389 consensus_queue.cc:1035] T 8f86d1517df74ead9e397ca9bed44c3f P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:29:17.878978 1954 raft_consensus.cc:1261] T 8f86d1517df74ead9e397ca9bed44c3f P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:29:17.879449 5797 consensus_queue.cc:1035] T 8f86d1517df74ead9e397ca9bed44c3f P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:29:17.880321 5797 consensus_queue.cc:1035] T 8f86d1517df74ead9e397ca9bed44c3f P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:29:17.880569 1549 raft_consensus.cc:1261] T 8f86d1517df74ead9e397ca9bed44c3f P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:29:17.882699 5797 consensus_queue.cc:1035] T 8f86d1517df74ead9e397ca9bed44c3f P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:29:17.883697 1414 raft_consensus.cc:1261] T 8f86d1517df74ead9e397ca9bed44c3f P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:29:17.884346 5389 consensus_queue.cc:1035] T 8f86d1517df74ead9e397ca9bed44c3f P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:29:17.894007 5389 raft_consensus.cc:2932] T 8f86d1517df74ead9e397ca9bed44c3f P 521bb93035324cecbebed2f6f2a630f8 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, d8d7297186694f77bb98de9174e4279c (127.15.255.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } }
I20240502 21:29:17.897387 1415 raft_consensus.cc:2932] T 8f86d1517df74ead9e397ca9bed44c3f P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, d8d7297186694f77bb98de9174e4279c (127.15.255.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } }
I20240502 21:29:17.898720 1686 raft_consensus.cc:2932] T 8f86d1517df74ead9e397ca9bed44c3f P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, d8d7297186694f77bb98de9174e4279c (127.15.255.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } }
I20240502 21:29:17.903329 1270 catalog_manager.cc:5484] T 8f86d1517df74ead9e397ca9bed44c3f P 7d1e5c1b047443cbb28d7b0a9744babf reported cstate change: config changed from index 8 to 9, d8d7297186694f77bb98de9174e4279c (127.15.255.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "521bb93035324cecbebed2f6f2a630f8" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } }
I20240502 21:29:17.903376 1549 raft_consensus.cc:2932] T 8f86d1517df74ead9e397ca9bed44c3f P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, d8d7297186694f77bb98de9174e4279c (127.15.255.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } }
I20240502 21:29:17.904331 1953 raft_consensus.cc:2932] T 8f86d1517df74ead9e397ca9bed44c3f P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, d8d7297186694f77bb98de9174e4279c (127.15.255.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } }
I20240502 21:29:17.910056 2226 raft_consensus.cc:2932] T 8f86d1517df74ead9e397ca9bed44c3f P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, d8d7297186694f77bb98de9174e4279c (127.15.255.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } }
I20240502 21:29:17.912125 3855 raft_consensus.cc:2932] T 8f86d1517df74ead9e397ca9bed44c3f P d8d7297186694f77bb98de9174e4279c [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, d8d7297186694f77bb98de9174e4279c (127.15.255.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } }
I20240502 21:29:17.913177 4119 raft_consensus.cc:2932] T 8f86d1517df74ead9e397ca9bed44c3f P 1fab5794cf1740288abd82f2cebb9b1a [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, d8d7297186694f77bb98de9174e4279c (127.15.255.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } }
I20240502 21:29:17.926880 2087 consensus_queue.cc:237] T 8f86d1517df74ead9e397ca9bed44c3f P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } }
I20240502 21:29:17.931178 3855 raft_consensus.cc:1261] T 8f86d1517df74ead9e397ca9bed44c3f P d8d7297186694f77bb98de9174e4279c [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 21:29:17.931473 4119 raft_consensus.cc:1261] T 8f86d1517df74ead9e397ca9bed44c3f P 1fab5794cf1740288abd82f2cebb9b1a [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 21:29:17.931959 1953 raft_consensus.cc:1261] T 8f86d1517df74ead9e397ca9bed44c3f P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 21:29:17.932112 5389 consensus_queue.cc:1035] T 8f86d1517df74ead9e397ca9bed44c3f P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 21:29:17.932595 6102 consensus_queue.cc:1035] T 8f86d1517df74ead9e397ca9bed44c3f P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 21:29:17.933676 6102 consensus_queue.cc:1035] T 8f86d1517df74ead9e397ca9bed44c3f P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 21:29:17.935886 2226 raft_consensus.cc:1261] T 8f86d1517df74ead9e397ca9bed44c3f P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 21:29:17.937072 6090 raft_consensus.cc:2932] T 8f86d1517df74ead9e397ca9bed44c3f P 521bb93035324cecbebed2f6f2a630f8 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 16b3a337bc5e462f83ba895ad8660deb (127.15.255.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } }
I20240502 21:29:17.937819 1549 raft_consensus.cc:1261] T 8f86d1517df74ead9e397ca9bed44c3f P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 21:29:17.938153 5389 consensus_queue.cc:1035] T 8f86d1517df74ead9e397ca9bed44c3f P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 21:29:17.938225 1953 raft_consensus.cc:2932] T 8f86d1517df74ead9e397ca9bed44c3f P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 16b3a337bc5e462f83ba895ad8660deb (127.15.255.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } }
I20240502 21:29:17.939627 2226 raft_consensus.cc:2932] T 8f86d1517df74ead9e397ca9bed44c3f P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 16b3a337bc5e462f83ba895ad8660deb (127.15.255.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } }
I20240502 21:29:17.942610 1254 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 8f86d1517df74ead9e397ca9bed44c3f with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:29:17.942668 1414 raft_consensus.cc:1261] T 8f86d1517df74ead9e397ca9bed44c3f P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 21:29:17.942606 4119 raft_consensus.cc:2932] T 8f86d1517df74ead9e397ca9bed44c3f P 1fab5794cf1740288abd82f2cebb9b1a [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 16b3a337bc5e462f83ba895ad8660deb (127.15.255.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } }
I20240502 21:29:17.943627 5797 consensus_queue.cc:1035] T 8f86d1517df74ead9e397ca9bed44c3f P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 21:29:17.944254 5389 consensus_queue.cc:1035] T 8f86d1517df74ead9e397ca9bed44c3f P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 21:29:17.945442 1266 catalog_manager.cc:5484] T 8f86d1517df74ead9e397ca9bed44c3f P 521bb93035324cecbebed2f6f2a630f8 reported cstate change: config changed from index 9 to 10, VOTER 16b3a337bc5e462f83ba895ad8660deb (127.15.255.131) evicted. New cstate: current_term: 1 leader_uuid: "521bb93035324cecbebed2f6f2a630f8" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 21:29:17.945788 1549 raft_consensus.cc:2932] T 8f86d1517df74ead9e397ca9bed44c3f P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 16b3a337bc5e462f83ba895ad8660deb (127.15.255.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } }
I20240502 21:29:17.951350 1414 raft_consensus.cc:2932] T 8f86d1517df74ead9e397ca9bed44c3f P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 16b3a337bc5e462f83ba895ad8660deb (127.15.255.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } }
I20240502 21:29:17.955147 3855 raft_consensus.cc:2932] T 8f86d1517df74ead9e397ca9bed44c3f P d8d7297186694f77bb98de9174e4279c [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 16b3a337bc5e462f83ba895ad8660deb (127.15.255.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } }
I20240502 21:29:17.957222 1664 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f86d1517df74ead9e397ca9bed44c3f with delete_type TABLET_DATA_TOMBSTONED (TS 16b3a337bc5e462f83ba895ad8660deb not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:42590
I20240502 21:29:17.957607 6072 tablet_replica.cc:330] T 8f86d1517df74ead9e397ca9bed44c3f P 16b3a337bc5e462f83ba895ad8660deb: stopping tablet replica
I20240502 21:29:17.957804 6072 raft_consensus.cc:2229] T 8f86d1517df74ead9e397ca9bed44c3f P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:29:17.958261 6072 raft_consensus.cc:2258] T 8f86d1517df74ead9e397ca9bed44c3f P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:29:17.959432 6017 raft_consensus.cc:1050] T 05e885ae741c43808c6dfbcdc5769942 P 6ab8df3c796f4b6cb3e29d5045ad3ea0: attempting to promote NON_VOTER 1fab5794cf1740288abd82f2cebb9b1a to VOTER
I20240502 21:29:17.960682 6017 consensus_queue.cc:237] T 05e885ae741c43808c6dfbcdc5769942 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } }
I20240502 21:29:17.964399 5797 raft_consensus.cc:1050] T e44bf56d64b24e328803ebd0a3a54ceb P 521bb93035324cecbebed2f6f2a630f8: attempting to promote NON_VOTER d8d7297186694f77bb98de9174e4279c to VOTER
I20240502 21:29:17.965509 5797 consensus_queue.cc:237] T e44bf56d64b24e328803ebd0a3a54ceb P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { replace: true } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } }
I20240502 21:29:17.966638 3987 raft_consensus.cc:1261] T 05e885ae741c43808c6dfbcdc5769942 P 16a97c1394f346f0b987f97cce7371b6 [term 1 FOLLOWER]: Refusing update from remote peer 6ab8df3c796f4b6cb3e29d5045ad3ea0: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:29:17.966835 4121 raft_consensus.cc:1261] T 05e885ae741c43808c6dfbcdc5769942 P 1fab5794cf1740288abd82f2cebb9b1a [term 1 LEARNER]: Refusing update from remote peer 6ab8df3c796f4b6cb3e29d5045ad3ea0: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:29:17.967407 5627 consensus_queue.cc:1035] T 05e885ae741c43808c6dfbcdc5769942 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:29:17.968468 6072 ts_tablet_manager.cc:1875] T 8f86d1517df74ead9e397ca9bed44c3f P 16b3a337bc5e462f83ba895ad8660deb: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:29:17.969322 1415 raft_consensus.cc:1261] T e44bf56d64b24e328803ebd0a3a54ceb P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:29:17.969938 2226 raft_consensus.cc:1261] T e44bf56d64b24e328803ebd0a3a54ceb P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:29:17.970043 5833 consensus_queue.cc:1035] T 05e885ae741c43808c6dfbcdc5769942 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:29:17.970578 6090 consensus_queue.cc:1035] T e44bf56d64b24e328803ebd0a3a54ceb P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:29:17.971076 6090 consensus_queue.cc:1035] T e44bf56d64b24e328803ebd0a3a54ceb P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:29:17.972218 1953 raft_consensus.cc:1261] T 05e885ae741c43808c6dfbcdc5769942 P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Refusing update from remote peer 6ab8df3c796f4b6cb3e29d5045ad3ea0: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:29:17.972512 1685 raft_consensus.cc:1261] T 05e885ae741c43808c6dfbcdc5769942 P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Refusing update from remote peer 6ab8df3c796f4b6cb3e29d5045ad3ea0: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:29:17.972978 5627 consensus_queue.cc:1035] T 05e885ae741c43808c6dfbcdc5769942 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:29:17.974686 3855 raft_consensus.cc:1261] T e44bf56d64b24e328803ebd0a3a54ceb P d8d7297186694f77bb98de9174e4279c [term 1 LEARNER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:29:17.975237 5389 consensus_queue.cc:1035] T e44bf56d64b24e328803ebd0a3a54ceb P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:29:17.978421 1549 raft_consensus.cc:1261] T e44bf56d64b24e328803ebd0a3a54ceb P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:29:17.978991 2087 raft_consensus.cc:1261] T 05e885ae741c43808c6dfbcdc5769942 P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Refusing update from remote peer 6ab8df3c796f4b6cb3e29d5045ad3ea0: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:29:17.979420 1685 raft_consensus.cc:1261] T e44bf56d64b24e328803ebd0a3a54ceb P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:29:17.980465 5833 consensus_queue.cc:1035] T 05e885ae741c43808c6dfbcdc5769942 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:29:17.980659 1954 raft_consensus.cc:1261] T e44bf56d64b24e328803ebd0a3a54ceb P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:29:17.981029 5833 consensus_queue.cc:1035] T 05e885ae741c43808c6dfbcdc5769942 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:29:17.982266 6072 ts_tablet_manager.cc:1888] T 8f86d1517df74ead9e397ca9bed44c3f P 16b3a337bc5e462f83ba895ad8660deb: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240502 21:29:17.982862 5389 consensus_queue.cc:1035] T e44bf56d64b24e328803ebd0a3a54ceb P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:29:17.983144 6072 log.cc:1196] T 8f86d1517df74ead9e397ca9bed44c3f P 16b3a337bc5e462f83ba895ad8660deb: Deleting WAL directory at /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-2/wal/wals/8f86d1517df74ead9e397ca9bed44c3f
I20240502 21:29:17.983409 5389 consensus_queue.cc:1035] T e44bf56d64b24e328803ebd0a3a54ceb P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:29:17.983938 5389 consensus_queue.cc:1035] T e44bf56d64b24e328803ebd0a3a54ceb P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:29:17.984561 1257 catalog_manager.cc:4830] TS 16b3a337bc5e462f83ba895ad8660deb (127.15.255.131:36033): tablet 8f86d1517df74ead9e397ca9bed44c3f (table rebalance_test_table_0 [id=82737b0de4b945dc8ca0c7d6b92af37c]) successfully deleted
I20240502 21:29:17.990216 4119 raft_consensus.cc:1261] T e44bf56d64b24e328803ebd0a3a54ceb P 1fab5794cf1740288abd82f2cebb9b1a [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:29:17.990876 5797 consensus_queue.cc:1035] T e44bf56d64b24e328803ebd0a3a54ceb P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:29:18.009318 1413 raft_consensus.cc:1261] T 05e885ae741c43808c6dfbcdc5769942 P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Refusing update from remote peer 6ab8df3c796f4b6cb3e29d5045ad3ea0: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:29:18.011037 1550 raft_consensus.cc:1261] T 05e885ae741c43808c6dfbcdc5769942 P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Refusing update from remote peer 6ab8df3c796f4b6cb3e29d5045ad3ea0: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:29:18.012244 5627 consensus_queue.cc:1035] T 05e885ae741c43808c6dfbcdc5769942 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:29:18.012859 5627 consensus_queue.cc:1035] T 05e885ae741c43808c6dfbcdc5769942 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:29:18.022902 6116 raft_consensus.cc:2932] T e44bf56d64b24e328803ebd0a3a54ceb P 521bb93035324cecbebed2f6f2a630f8 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, d8d7297186694f77bb98de9174e4279c (127.15.255.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { replace: true } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } }
I20240502 21:29:18.023836 5389 raft_consensus.cc:1050] T f8de1fd0869348e6b0aa131b958ce822 P 521bb93035324cecbebed2f6f2a630f8: attempting to promote NON_VOTER 16a97c1394f346f0b987f97cce7371b6 to VOTER
I20240502 21:29:18.025113 5389 consensus_queue.cc:237] T f8de1fd0869348e6b0aa131b958ce822 P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { replace: true } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } }
I20240502 21:29:18.028661 1685 raft_consensus.cc:2932] T e44bf56d64b24e328803ebd0a3a54ceb P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, d8d7297186694f77bb98de9174e4279c (127.15.255.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { replace: true } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } }
I20240502 21:29:18.028892 4120 raft_consensus.cc:2932] T e44bf56d64b24e328803ebd0a3a54ceb P 1fab5794cf1740288abd82f2cebb9b1a [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, d8d7297186694f77bb98de9174e4279c (127.15.255.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { replace: true } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } }
I20240502 21:29:18.027487 1415 raft_consensus.cc:2932] T e44bf56d64b24e328803ebd0a3a54ceb P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, d8d7297186694f77bb98de9174e4279c (127.15.255.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { replace: true } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } }
I20240502 21:29:18.032922 2226 raft_consensus.cc:2932] T e44bf56d64b24e328803ebd0a3a54ceb P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, d8d7297186694f77bb98de9174e4279c (127.15.255.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { replace: true } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } }
I20240502 21:29:18.035900 1270 catalog_manager.cc:5484] T e44bf56d64b24e328803ebd0a3a54ceb P 521bb93035324cecbebed2f6f2a630f8 reported cstate change: config changed from index 8 to 9, d8d7297186694f77bb98de9174e4279c (127.15.255.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "521bb93035324cecbebed2f6f2a630f8" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 21:29:18.052242 3987 raft_consensus.cc:1261] T f8de1fd0869348e6b0aa131b958ce822 P 16a97c1394f346f0b987f97cce7371b6 [term 1 LEARNER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:29:18.052515 1685 raft_consensus.cc:1261] T f8de1fd0869348e6b0aa131b958ce822 P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:29:18.052675 4120 raft_consensus.cc:1261] T f8de1fd0869348e6b0aa131b958ce822 P 1fab5794cf1740288abd82f2cebb9b1a [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:29:18.053084 6116 consensus_queue.cc:1035] T f8de1fd0869348e6b0aa131b958ce822 P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:29:18.053740 6116 consensus_queue.cc:1035] T f8de1fd0869348e6b0aa131b958ce822 P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:29:18.053824 1414 raft_consensus.cc:1261] T f8de1fd0869348e6b0aa131b958ce822 P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:29:18.054311 6116 consensus_queue.cc:1035] T f8de1fd0869348e6b0aa131b958ce822 P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:29:18.054472 1548 raft_consensus.cc:1261] T f8de1fd0869348e6b0aa131b958ce822 P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:29:18.054842 6090 consensus_queue.cc:1035] T f8de1fd0869348e6b0aa131b958ce822 P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:29:18.055569 6090 consensus_queue.cc:1035] T f8de1fd0869348e6b0aa131b958ce822 P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:29:18.055754 1817 raft_consensus.cc:1261] T f8de1fd0869348e6b0aa131b958ce822 P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:29:18.056262 6090 consensus_queue.cc:1035] T f8de1fd0869348e6b0aa131b958ce822 P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:29:18.064011 2226 raft_consensus.cc:1261] T f8de1fd0869348e6b0aa131b958ce822 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:29:18.064560 6116 consensus_queue.cc:1035] T f8de1fd0869348e6b0aa131b958ce822 P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:29:18.073050 6109 raft_consensus.cc:2932] T 05e885ae741c43808c6dfbcdc5769942 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } }
I20240502 21:29:18.077075 4119 raft_consensus.cc:2932] T 05e885ae741c43808c6dfbcdc5769942 P 1fab5794cf1740288abd82f2cebb9b1a [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } }
I20240502 21:29:18.078641 3988 raft_consensus.cc:2932] T 05e885ae741c43808c6dfbcdc5769942 P 16a97c1394f346f0b987f97cce7371b6 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } }
I20240502 21:29:18.082989 1264 catalog_manager.cc:5484] T 05e885ae741c43808c6dfbcdc5769942 P 1fab5794cf1740288abd82f2cebb9b1a reported cstate change: config changed from index 8 to 9, 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } }
I20240502 21:29:18.076324 1954 raft_consensus.cc:2932] T e44bf56d64b24e328803ebd0a3a54ceb P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, d8d7297186694f77bb98de9174e4279c (127.15.255.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { replace: true } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } }
I20240502 21:29:18.085072 1686 raft_consensus.cc:2932] T 05e885ae741c43808c6dfbcdc5769942 P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } }
I20240502 21:29:18.100569 3855 raft_consensus.cc:2932] T e44bf56d64b24e328803ebd0a3a54ceb P d8d7297186694f77bb98de9174e4279c [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, d8d7297186694f77bb98de9174e4279c (127.15.255.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { replace: true } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } }
I20240502 21:29:18.100706 1549 raft_consensus.cc:2932] T e44bf56d64b24e328803ebd0a3a54ceb P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, d8d7297186694f77bb98de9174e4279c (127.15.255.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { replace: true } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } }
I20240502 21:29:18.099093 1550 raft_consensus.cc:2932] T 05e885ae741c43808c6dfbcdc5769942 P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } }
I20240502 21:29:18.103962 2088 consensus_queue.cc:237] T e44bf56d64b24e328803ebd0a3a54ceb P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } }
I20240502 21:29:18.108600 1686 raft_consensus.cc:1261] T e44bf56d64b24e328803ebd0a3a54ceb P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 21:29:18.109704 2225 raft_consensus.cc:1261] T e44bf56d64b24e328803ebd0a3a54ceb P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 21:29:18.110428 3855 raft_consensus.cc:1261] T e44bf56d64b24e328803ebd0a3a54ceb P d8d7297186694f77bb98de9174e4279c [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 21:29:18.110459 4121 raft_consensus.cc:1261] T e44bf56d64b24e328803ebd0a3a54ceb P 1fab5794cf1740288abd82f2cebb9b1a [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 21:29:18.110654 1954 raft_consensus.cc:1261] T e44bf56d64b24e328803ebd0a3a54ceb P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 21:29:18.110913 6116 consensus_queue.cc:1035] T e44bf56d64b24e328803ebd0a3a54ceb P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 21:29:18.111207 6090 consensus_queue.cc:1035] T e44bf56d64b24e328803ebd0a3a54ceb P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 21:29:18.111490 6102 consensus_queue.cc:1035] T e44bf56d64b24e328803ebd0a3a54ceb P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 21:29:18.114156 5796 consensus_queue.cc:1035] T e44bf56d64b24e328803ebd0a3a54ceb P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 21:29:18.115293 6100 consensus_queue.cc:1035] T e44bf56d64b24e328803ebd0a3a54ceb P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 21:29:18.111991 1953 raft_consensus.cc:2932] T 05e885ae741c43808c6dfbcdc5769942 P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } }
I20240502 21:29:18.120472 1412 raft_consensus.cc:2932] T 05e885ae741c43808c6dfbcdc5769942 P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } }
I20240502 21:29:18.121631 1413 raft_consensus.cc:1261] T e44bf56d64b24e328803ebd0a3a54ceb P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 21:29:18.123220 5661 raft_consensus.cc:1050] T aae7b5a01a0543afb1b5a7f44a9a864c P 7d1e5c1b047443cbb28d7b0a9744babf: attempting to promote NON_VOTER d8d7297186694f77bb98de9174e4279c to VOTER
I20240502 21:29:18.124498 5661 consensus_queue.cc:237] T aae7b5a01a0543afb1b5a7f44a9a864c P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } }
I20240502 21:29:18.128692 3855 raft_consensus.cc:1261] T aae7b5a01a0543afb1b5a7f44a9a864c P d8d7297186694f77bb98de9174e4279c [term 1 LEARNER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:29:18.129308 6046 consensus_queue.cc:1035] T aae7b5a01a0543afb1b5a7f44a9a864c P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:29:18.130522 1954 raft_consensus.cc:1261] T aae7b5a01a0543afb1b5a7f44a9a864c P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:29:18.130738 2226 raft_consensus.cc:1261] T aae7b5a01a0543afb1b5a7f44a9a864c P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:29:18.131323 6066 consensus_queue.cc:1035] T aae7b5a01a0543afb1b5a7f44a9a864c P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:29:18.132440 1685 raft_consensus.cc:1261] T aae7b5a01a0543afb1b5a7f44a9a864c P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:29:18.132761 1549 raft_consensus.cc:1261] T aae7b5a01a0543afb1b5a7f44a9a864c P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:29:18.132740 6066 consensus_queue.cc:1035] T aae7b5a01a0543afb1b5a7f44a9a864c P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:29:18.133360 5661 consensus_queue.cc:1035] T aae7b5a01a0543afb1b5a7f44a9a864c P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:29:18.133652 6066 consensus_queue.cc:1035] T aae7b5a01a0543afb1b5a7f44a9a864c P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:29:18.135043 4120 raft_consensus.cc:1261] T aae7b5a01a0543afb1b5a7f44a9a864c P 1fab5794cf1740288abd82f2cebb9b1a [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:29:18.135578 5661 consensus_queue.cc:1035] T aae7b5a01a0543afb1b5a7f44a9a864c P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:29:18.148164 2089 raft_consensus.cc:1261] T aae7b5a01a0543afb1b5a7f44a9a864c P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:29:18.150322 6090 raft_consensus.cc:2932] T f8de1fd0869348e6b0aa131b958ce822 P 521bb93035324cecbebed2f6f2a630f8 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 16a97c1394f346f0b987f97cce7371b6 (127.15.255.137) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { replace: true } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } }
I20240502 21:29:18.152093 6051 raft_consensus.cc:2932] T aae7b5a01a0543afb1b5a7f44a9a864c P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, d8d7297186694f77bb98de9174e4279c (127.15.255.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } }
I20240502 21:29:18.152810 4121 raft_consensus.cc:2932] T aae7b5a01a0543afb1b5a7f44a9a864c P 1fab5794cf1740288abd82f2cebb9b1a [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, d8d7297186694f77bb98de9174e4279c (127.15.255.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } }
I20240502 21:29:18.153470 1685 raft_consensus.cc:2932] T aae7b5a01a0543afb1b5a7f44a9a864c P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, d8d7297186694f77bb98de9174e4279c (127.15.255.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } }
I20240502 21:29:18.157871 2226 raft_consensus.cc:2932] T aae7b5a01a0543afb1b5a7f44a9a864c P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, d8d7297186694f77bb98de9174e4279c (127.15.255.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } }
I20240502 21:29:18.158782 1270 catalog_manager.cc:5484] T aae7b5a01a0543afb1b5a7f44a9a864c P 1fab5794cf1740288abd82f2cebb9b1a reported cstate change: config changed from index 8 to 9, d8d7297186694f77bb98de9174e4279c (127.15.255.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } }
I20240502 21:29:18.160367 5796 raft_consensus.cc:2932] T e44bf56d64b24e328803ebd0a3a54ceb P 521bb93035324cecbebed2f6f2a630f8 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER a62a935ff43d494ca35d0c417943133d (127.15.255.130) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } }
I20240502 21:29:18.160127 1549 raft_consensus.cc:2932] T aae7b5a01a0543afb1b5a7f44a9a864c P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, d8d7297186694f77bb98de9174e4279c (127.15.255.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } }
I20240502 21:29:18.163671 5875 raft_consensus.cc:1050] T 444c1cba2a6f4729aa01b75d06b6b517 P e632c553ff4341328acf1cbaca678392: attempting to promote NON_VOTER 1fab5794cf1740288abd82f2cebb9b1a to VOTER
I20240502 21:29:18.165102 5875 consensus_queue.cc:237] T 444c1cba2a6f4729aa01b75d06b6b517 P e632c553ff4341328acf1cbaca678392 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { replace: true } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } }
I20240502 21:29:18.170025 1954 raft_consensus.cc:2932] T aae7b5a01a0543afb1b5a7f44a9a864c P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, d8d7297186694f77bb98de9174e4279c (127.15.255.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } }
I20240502 21:29:18.182107 1412 raft_consensus.cc:1261] T 444c1cba2a6f4729aa01b75d06b6b517 P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Refusing update from remote peer e632c553ff4341328acf1cbaca678392: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
W20240502 21:29:18.182415 6117 log.cc:924] Time spent T 05e885ae741c43808c6dfbcdc5769942 P 521bb93035324cecbebed2f6f2a630f8: Append to log took a long time: real 0.067s user 0.000s sys 0.000s
I20240502 21:29:18.183022 5735 consensus_queue.cc:1035] T 444c1cba2a6f4729aa01b75d06b6b517 P e632c553ff4341328acf1cbaca678392 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:29:18.183305 4121 raft_consensus.cc:1261] T 444c1cba2a6f4729aa01b75d06b6b517 P 1fab5794cf1740288abd82f2cebb9b1a [term 1 LEARNER]: Refusing update from remote peer e632c553ff4341328acf1cbaca678392: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:29:18.183598 2089 raft_consensus.cc:1261] T 444c1cba2a6f4729aa01b75d06b6b517 P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Refusing update from remote peer e632c553ff4341328acf1cbaca678392: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:29:18.181355 3856 raft_consensus.cc:1261] T 444c1cba2a6f4729aa01b75d06b6b517 P d8d7297186694f77bb98de9174e4279c [term 1 FOLLOWER]: Refusing update from remote peer e632c553ff4341328acf1cbaca678392: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:29:18.185382 5875 consensus_queue.cc:1035] T 444c1cba2a6f4729aa01b75d06b6b517 P e632c553ff4341328acf1cbaca678392 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:29:18.186044 5875 consensus_queue.cc:1035] T 444c1cba2a6f4729aa01b75d06b6b517 P e632c553ff4341328acf1cbaca678392 [LEADER]: Connected to new peer: Peer: permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:29:18.186606 1685 raft_consensus.cc:1261] T 444c1cba2a6f4729aa01b75d06b6b517 P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Refusing update from remote peer e632c553ff4341328acf1cbaca678392: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:29:18.187501 1549 raft_consensus.cc:1261] T 444c1cba2a6f4729aa01b75d06b6b517 P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Refusing update from remote peer e632c553ff4341328acf1cbaca678392: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:29:18.189440 1254 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet e44bf56d64b24e328803ebd0a3a54ceb with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:29:18.191099 3987 raft_consensus.cc:2932] T f8de1fd0869348e6b0aa131b958ce822 P 16a97c1394f346f0b987f97cce7371b6 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 16a97c1394f346f0b987f97cce7371b6 (127.15.255.137) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { replace: true } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } }
I20240502 21:29:18.197887 3855 raft_consensus.cc:2932] T aae7b5a01a0543afb1b5a7f44a9a864c P d8d7297186694f77bb98de9174e4279c [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, d8d7297186694f77bb98de9174e4279c (127.15.255.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } }
I20240502 21:29:18.197885 1817 raft_consensus.cc:2932] T f8de1fd0869348e6b0aa131b958ce822 P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 16a97c1394f346f0b987f97cce7371b6 (127.15.255.137) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { replace: true } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } }
I20240502 21:29:18.202922 1266 catalog_manager.cc:5484] T f8de1fd0869348e6b0aa131b958ce822 P 16a97c1394f346f0b987f97cce7371b6 reported cstate change: config changed from index 8 to 9, 16a97c1394f346f0b987f97cce7371b6 (127.15.255.137) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "521bb93035324cecbebed2f6f2a630f8" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { replace: true } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } }
W20240502 21:29:18.204592 2025 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.138:33375, user_credentials={real_user=slave}} blocked reactor thread for 63566.6us
I20240502 21:29:18.206089 5661 consensus_queue.cc:1035] T aae7b5a01a0543afb1b5a7f44a9a864c P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:29:18.207842 4120 raft_consensus.cc:2932] T e44bf56d64b24e328803ebd0a3a54ceb P 1fab5794cf1740288abd82f2cebb9b1a [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER a62a935ff43d494ca35d0c417943133d (127.15.255.130) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } }
I20240502 21:29:18.210047 3855 raft_consensus.cc:2932] T e44bf56d64b24e328803ebd0a3a54ceb P d8d7297186694f77bb98de9174e4279c [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER a62a935ff43d494ca35d0c417943133d (127.15.255.130) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } }
W20240502 21:29:18.214272 2023 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.135:34701, user_credentials={real_user=slave}} blocked reactor thread for 96739.1us
I20240502 21:29:18.216254 2226 raft_consensus.cc:2932] T e44bf56d64b24e328803ebd0a3a54ceb P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER a62a935ff43d494ca35d0c417943133d (127.15.255.130) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } }
I20240502 21:29:18.221935 2225 consensus_queue.cc:237] T 05e885ae741c43808c6dfbcdc5769942 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } }
I20240502 21:29:18.222303 2226 raft_consensus.cc:2932] T f8de1fd0869348e6b0aa131b958ce822 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 16a97c1394f346f0b987f97cce7371b6 (127.15.255.137) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { replace: true } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } }
I20240502 21:29:18.226332 4120 raft_consensus.cc:1261] T 05e885ae741c43808c6dfbcdc5769942 P 1fab5794cf1740288abd82f2cebb9b1a [term 1 FOLLOWER]: Refusing update from remote peer 6ab8df3c796f4b6cb3e29d5045ad3ea0: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 21:29:18.226915 6109 consensus_queue.cc:1035] T 05e885ae741c43808c6dfbcdc5769942 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 21:29:18.230340 1414 raft_consensus.cc:1261] T 05e885ae741c43808c6dfbcdc5769942 P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Refusing update from remote peer 6ab8df3c796f4b6cb3e29d5045ad3ea0: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 21:29:18.229960 1270 catalog_manager.cc:5484] T e44bf56d64b24e328803ebd0a3a54ceb P 6ab8df3c796f4b6cb3e29d5045ad3ea0 reported cstate change: config changed from index 9 to 10, VOTER a62a935ff43d494ca35d0c417943133d (127.15.255.130) evicted. New cstate: current_term: 1 leader_uuid: "521bb93035324cecbebed2f6f2a630f8" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } }
I20240502 21:29:18.230933 5833 consensus_queue.cc:1035] T 05e885ae741c43808c6dfbcdc5769942 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 21:29:18.230393 1954 raft_consensus.cc:2932] T e44bf56d64b24e328803ebd0a3a54ceb P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER a62a935ff43d494ca35d0c417943133d (127.15.255.130) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } }
I20240502 21:29:18.235534 1528 tablet_service.cc:1514] Processing DeleteTablet for tablet e44bf56d64b24e328803ebd0a3a54ceb with delete_type TABLET_DATA_TOMBSTONED (TS a62a935ff43d494ca35d0c417943133d not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:43860
I20240502 21:29:18.236653 6155 tablet_replica.cc:330] T e44bf56d64b24e328803ebd0a3a54ceb P a62a935ff43d494ca35d0c417943133d: stopping tablet replica
I20240502 21:29:18.237269 6155 raft_consensus.cc:2229] T e44bf56d64b24e328803ebd0a3a54ceb P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:29:18.238334 6155 raft_consensus.cc:2258] T e44bf56d64b24e328803ebd0a3a54ceb P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:29:18.239614 2086 raft_consensus.cc:1261] T 05e885ae741c43808c6dfbcdc5769942 P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Refusing update from remote peer 6ab8df3c796f4b6cb3e29d5045ad3ea0: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 21:29:18.240175 2088 raft_consensus.cc:2932] T 05e885ae741c43808c6dfbcdc5769942 P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } }
I20240502 21:29:18.243168 6155 ts_tablet_manager.cc:1875] T e44bf56d64b24e328803ebd0a3a54ceb P a62a935ff43d494ca35d0c417943133d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:29:18.244925 5833 consensus_queue.cc:1035] T 05e885ae741c43808c6dfbcdc5769942 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:29:18.245887 1549 raft_consensus.cc:1261] T 05e885ae741c43808c6dfbcdc5769942 P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Refusing update from remote peer 6ab8df3c796f4b6cb3e29d5045ad3ea0: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 21:29:18.246527 5833 consensus_queue.cc:1035] T 05e885ae741c43808c6dfbcdc5769942 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 21:29:18.250044 6116 consensus_queue.cc:1035] T e44bf56d64b24e328803ebd0a3a54ceb P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 21:29:18.251636 6109 raft_consensus.cc:2932] T 05e885ae741c43808c6dfbcdc5769942 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 16b3a337bc5e462f83ba895ad8660deb (127.15.255.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } }
I20240502 21:29:18.249714 6157 consensus_queue.cc:1035] T 444c1cba2a6f4729aa01b75d06b6b517 P e632c553ff4341328acf1cbaca678392 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:29:18.254303 6152 consensus_queue.cc:1035] T 444c1cba2a6f4729aa01b75d06b6b517 P e632c553ff4341328acf1cbaca678392 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:29:18.254282 1414 raft_consensus.cc:2932] T e44bf56d64b24e328803ebd0a3a54ceb P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER a62a935ff43d494ca35d0c417943133d (127.15.255.130) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } }
I20240502 21:29:18.256691 1255 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 05e885ae741c43808c6dfbcdc5769942 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:29:18.257059 1412 raft_consensus.cc:2932] T f8de1fd0869348e6b0aa131b958ce822 P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 16a97c1394f346f0b987f97cce7371b6 (127.15.255.137) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { replace: true } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } }
I20240502 21:29:18.259510 6146 raft_consensus.cc:1050] T 1aa57620996e42b6adf941e9d606f7e2 P 521bb93035324cecbebed2f6f2a630f8: attempting to promote NON_VOTER 16a97c1394f346f0b987f97cce7371b6 to VOTER
I20240502 21:29:18.259601 1955 raft_consensus.cc:1261] T 05e885ae741c43808c6dfbcdc5769942 P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Refusing update from remote peer 6ab8df3c796f4b6cb3e29d5045ad3ea0: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 21:29:18.260280 5627 consensus_queue.cc:1035] T 05e885ae741c43808c6dfbcdc5769942 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 21:29:18.262068 1955 raft_consensus.cc:2932] T 05e885ae741c43808c6dfbcdc5769942 P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 16b3a337bc5e462f83ba895ad8660deb (127.15.255.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } }
I20240502 21:29:18.264770 6146 consensus_queue.cc:237] T 1aa57620996e42b6adf941e9d606f7e2 P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } }
I20240502 21:29:18.266701 1550 raft_consensus.cc:2932] T f8de1fd0869348e6b0aa131b958ce822 P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 16a97c1394f346f0b987f97cce7371b6 (127.15.255.137) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { replace: true } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } }
I20240502 21:29:18.268827 1412 raft_consensus.cc:2932] T 05e885ae741c43808c6dfbcdc5769942 P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 16b3a337bc5e462f83ba895ad8660deb (127.15.255.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } }
I20240502 21:29:18.274796 4121 raft_consensus.cc:2932] T 05e885ae741c43808c6dfbcdc5769942 P 1fab5794cf1740288abd82f2cebb9b1a [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 16b3a337bc5e462f83ba895ad8660deb (127.15.255.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } }
I20240502 21:29:18.277411 1550 raft_consensus.cc:2932] T 05e885ae741c43808c6dfbcdc5769942 P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 16b3a337bc5e462f83ba895ad8660deb (127.15.255.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } }
W20240502 21:29:18.280313 1884 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.134:40821, user_credentials={real_user=slave}} blocked reactor thread for 96131.7us
I20240502 21:29:18.280740 3987 raft_consensus.cc:1261] T 05e885ae741c43808c6dfbcdc5769942 P 16a97c1394f346f0b987f97cce7371b6 [term 1 FOLLOWER]: Refusing update from remote peer 6ab8df3c796f4b6cb3e29d5045ad3ea0: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 21:29:18.272233 1268 catalog_manager.cc:5484] T 05e885ae741c43808c6dfbcdc5769942 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 reported cstate change: config changed from index 9 to 10, VOTER 16b3a337bc5e462f83ba895ad8660deb (127.15.255.131) evicted. New cstate: current_term: 1 leader_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:29:18.282433 5627 consensus_queue.cc:1035] T 05e885ae741c43808c6dfbcdc5769942 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 21:29:18.286142 1664 tablet_service.cc:1514] Processing DeleteTablet for tablet 05e885ae741c43808c6dfbcdc5769942 with delete_type TABLET_DATA_TOMBSTONED (TS 16b3a337bc5e462f83ba895ad8660deb not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:42590
I20240502 21:29:18.286823 6072 tablet_replica.cc:330] T 05e885ae741c43808c6dfbcdc5769942 P 16b3a337bc5e462f83ba895ad8660deb: stopping tablet replica
I20240502 21:29:18.287043 6072 raft_consensus.cc:2229] T 05e885ae741c43808c6dfbcdc5769942 P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:29:18.287364 6072 raft_consensus.cc:2258] T 05e885ae741c43808c6dfbcdc5769942 P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:29:18.288316 6072 ts_tablet_manager.cc:1875] T 05e885ae741c43808c6dfbcdc5769942 P 16b3a337bc5e462f83ba895ad8660deb: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:29:18.293944 3987 raft_consensus.cc:2932] T 05e885ae741c43808c6dfbcdc5769942 P 16a97c1394f346f0b987f97cce7371b6 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 16b3a337bc5e462f83ba895ad8660deb (127.15.255.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } }
I20240502 21:29:18.294834 6072 ts_tablet_manager.cc:1888] T 05e885ae741c43808c6dfbcdc5769942 P 16b3a337bc5e462f83ba895ad8660deb: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240502 21:29:18.294991 6072 log.cc:1196] T 05e885ae741c43808c6dfbcdc5769942 P 16b3a337bc5e462f83ba895ad8660deb: Deleting WAL directory at /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-2/wal/wals/05e885ae741c43808c6dfbcdc5769942
I20240502 21:29:18.295773 1257 catalog_manager.cc:4830] TS 16b3a337bc5e462f83ba895ad8660deb (127.15.255.131:36033): tablet 05e885ae741c43808c6dfbcdc5769942 (table rebalance_test_table_4 [id=d523de924ba14cb493e559296c083a34]) successfully deleted
I20240502 21:29:18.296955 6155 ts_tablet_manager.cc:1888] T e44bf56d64b24e328803ebd0a3a54ceb P a62a935ff43d494ca35d0c417943133d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240502 21:29:18.297178 6155 log.cc:1196] T e44bf56d64b24e328803ebd0a3a54ceb P a62a935ff43d494ca35d0c417943133d: Deleting WAL directory at /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-1/wal/wals/e44bf56d64b24e328803ebd0a3a54ceb
I20240502 21:29:18.297839 3856 raft_consensus.cc:1261] T 1aa57620996e42b6adf941e9d606f7e2 P d8d7297186694f77bb98de9174e4279c [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:29:18.298200 3988 raft_consensus.cc:1261] T 1aa57620996e42b6adf941e9d606f7e2 P 16a97c1394f346f0b987f97cce7371b6 [term 1 LEARNER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:29:18.298460 6145 consensus_queue.cc:1035] T 1aa57620996e42b6adf941e9d606f7e2 P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:29:18.298839 6102 consensus_queue.cc:1035] T 1aa57620996e42b6adf941e9d606f7e2 P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:29:18.304247 2089 consensus_queue.cc:237] T f8de1fd0869348e6b0aa131b958ce822 P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } }
I20240502 21:29:18.305310 5875 raft_consensus.cc:2932] T 444c1cba2a6f4729aa01b75d06b6b517 P e632c553ff4341328acf1cbaca678392 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { replace: true } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } }
I20240502 21:29:18.307040 3988 raft_consensus.cc:1261] T f8de1fd0869348e6b0aa131b958ce822 P 16a97c1394f346f0b987f97cce7371b6 [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 21:29:18.307623 6147 consensus_queue.cc:1035] T f8de1fd0869348e6b0aa131b958ce822 P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 21:29:18.309218 2226 raft_consensus.cc:1261] T f8de1fd0869348e6b0aa131b958ce822 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 21:29:18.311579 1412 raft_consensus.cc:1261] T f8de1fd0869348e6b0aa131b958ce822 P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 21:29:18.311393 4121 raft_consensus.cc:2932] T 444c1cba2a6f4729aa01b75d06b6b517 P 1fab5794cf1740288abd82f2cebb9b1a [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { replace: true } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } }
I20240502 21:29:18.312366 1685 raft_consensus.cc:2932] T 444c1cba2a6f4729aa01b75d06b6b517 P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { replace: true } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } }
I20240502 21:29:18.313071 3855 raft_consensus.cc:2932] T 444c1cba2a6f4729aa01b75d06b6b517 P d8d7297186694f77bb98de9174e4279c [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { replace: true } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } }
I20240502 21:29:18.315140 1817 raft_consensus.cc:1261] T f8de1fd0869348e6b0aa131b958ce822 P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 21:29:18.315663 6145 consensus_queue.cc:1035] T f8de1fd0869348e6b0aa131b958ce822 P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 21:29:18.316946 6102 consensus_queue.cc:1035] T f8de1fd0869348e6b0aa131b958ce822 P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 21:29:18.317431 6102 consensus_queue.cc:1035] T f8de1fd0869348e6b0aa131b958ce822 P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 21:29:18.319765 2089 raft_consensus.cc:2932] T 444c1cba2a6f4729aa01b75d06b6b517 P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { replace: true } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } }
I20240502 21:29:18.322283 1270 catalog_manager.cc:5484] T 444c1cba2a6f4729aa01b75d06b6b517 P 16b3a337bc5e462f83ba895ad8660deb reported cstate change: config changed from index 8 to 9, 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e632c553ff4341328acf1cbaca678392" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { replace: true } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } }
I20240502 21:29:18.322350 2226 raft_consensus.cc:2932] T f8de1fd0869348e6b0aa131b958ce822 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER a62a935ff43d494ca35d0c417943133d (127.15.255.130) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } }
I20240502 21:29:18.323155 1818 raft_consensus.cc:1261] T 1aa57620996e42b6adf941e9d606f7e2 P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:29:18.324507 2225 raft_consensus.cc:1261] T 1aa57620996e42b6adf941e9d606f7e2 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:29:18.321051 6116 raft_consensus.cc:2932] T f8de1fd0869348e6b0aa131b958ce822 P 521bb93035324cecbebed2f6f2a630f8 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER a62a935ff43d494ca35d0c417943133d (127.15.255.130) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } }
I20240502 21:29:18.328438 1270 catalog_manager.cc:5484] T f8de1fd0869348e6b0aa131b958ce822 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 reported cstate change: config changed from index 9 to 10, VOTER a62a935ff43d494ca35d0c417943133d (127.15.255.130) evicted. New cstate: current_term: 1 leader_uuid: "521bb93035324cecbebed2f6f2a630f8" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } }
I20240502 21:29:18.333835 1254 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f8de1fd0869348e6b0aa131b958ce822 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:29:18.323825 6090 consensus_queue.cc:1035] T 1aa57620996e42b6adf941e9d606f7e2 P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:29:18.339974 6102 consensus_queue.cc:1035] T 1aa57620996e42b6adf941e9d606f7e2 P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:29:18.343732 1412 consensus_queue.cc:237] T 72355dd734ea4aeba031171e8c39696d P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } }
I20240502 21:29:18.343806 3988 raft_consensus.cc:2932] T f8de1fd0869348e6b0aa131b958ce822 P 16a97c1394f346f0b987f97cce7371b6 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER a62a935ff43d494ca35d0c417943133d (127.15.255.130) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } }
I20240502 21:29:18.346946 3856 raft_consensus.cc:1261] T 72355dd734ea4aeba031171e8c39696d P d8d7297186694f77bb98de9174e4279c [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 21:29:18.347625 5763 consensus_queue.cc:1035] T 72355dd734ea4aeba031171e8c39696d P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 21:29:18.348016 4121 raft_consensus.cc:1261] T 72355dd734ea4aeba031171e8c39696d P 1fab5794cf1740288abd82f2cebb9b1a [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 21:29:18.348608 6051 consensus_queue.cc:1035] T 72355dd734ea4aeba031171e8c39696d P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 21:29:18.348894 1414 raft_consensus.cc:2932] T f8de1fd0869348e6b0aa131b958ce822 P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER a62a935ff43d494ca35d0c417943133d (127.15.255.130) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } }
I20240502 21:29:18.352226 1820 raft_consensus.cc:1261] T 72355dd734ea4aeba031171e8c39696d P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 21:29:18.352926 5763 consensus_queue.cc:1035] T 72355dd734ea4aeba031171e8c39696d P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 21:29:18.356997 1415 raft_consensus.cc:1261] T 1aa57620996e42b6adf941e9d606f7e2 P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:29:18.358738 1686 raft_consensus.cc:2932] T e44bf56d64b24e328803ebd0a3a54ceb P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER a62a935ff43d494ca35d0c417943133d (127.15.255.130) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } }
I20240502 21:29:18.360121 5763 raft_consensus.cc:2932] T 72355dd734ea4aeba031171e8c39696d P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER a62a935ff43d494ca35d0c417943133d (127.15.255.130) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } }
I20240502 21:29:18.361027 1820 raft_consensus.cc:2932] T 72355dd734ea4aeba031171e8c39696d P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER a62a935ff43d494ca35d0c417943133d (127.15.255.130) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } }
I20240502 21:29:18.363488 2226 raft_consensus.cc:1261] T 72355dd734ea4aeba031171e8c39696d P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 21:29:18.364045 1955 raft_consensus.cc:1261] T 72355dd734ea4aeba031171e8c39696d P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 21:29:18.365054 6074 raft_consensus.cc:1050] T 13af1b96121c492c9373f9acbb55725a P 153026dd7f8943cc93fedf6f48450fb4: attempting to promote NON_VOTER 16a97c1394f346f0b987f97cce7371b6 to VOTER
I20240502 21:29:18.366214 6074 consensus_queue.cc:237] T 13af1b96121c492c9373f9acbb55725a P 153026dd7f8943cc93fedf6f48450fb4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { replace: true } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } }
I20240502 21:29:18.368721 5661 consensus_queue.cc:1035] T 72355dd734ea4aeba031171e8c39696d P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 21:29:18.369637 6066 consensus_queue.cc:1035] T 72355dd734ea4aeba031171e8c39696d P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 21:29:18.369745 1255 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 72355dd734ea4aeba031171e8c39696d with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:29:18.370126 3856 raft_consensus.cc:2932] T 72355dd734ea4aeba031171e8c39696d P d8d7297186694f77bb98de9174e4279c [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER a62a935ff43d494ca35d0c417943133d (127.15.255.130) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } }
I20240502 21:29:18.371292 4120 raft_consensus.cc:2932] T 72355dd734ea4aeba031171e8c39696d P 1fab5794cf1740288abd82f2cebb9b1a [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER a62a935ff43d494ca35d0c417943133d (127.15.255.130) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } }
I20240502 21:29:18.371340 2225 raft_consensus.cc:2932] T 72355dd734ea4aeba031171e8c39696d P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER a62a935ff43d494ca35d0c417943133d (127.15.255.130) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } }
I20240502 21:29:18.378332 1955 raft_consensus.cc:2932] T 72355dd734ea4aeba031171e8c39696d P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER a62a935ff43d494ca35d0c417943133d (127.15.255.130) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } }
I20240502 21:29:18.379261 1270 catalog_manager.cc:5484] T 72355dd734ea4aeba031171e8c39696d P 1fab5794cf1740288abd82f2cebb9b1a reported cstate change: config changed from index 9 to 10, VOTER a62a935ff43d494ca35d0c417943133d (127.15.255.130) evicted. New cstate: current_term: 1 leader_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } }
I20240502 21:29:18.381397 6145 consensus_queue.cc:1035] T 1aa57620996e42b6adf941e9d606f7e2 P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:29:18.393209 1817 raft_consensus.cc:2932] T f8de1fd0869348e6b0aa131b958ce822 P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER a62a935ff43d494ca35d0c417943133d (127.15.255.130) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } }
I20240502 21:29:18.397854 1550 raft_consensus.cc:1261] T 1aa57620996e42b6adf941e9d606f7e2 P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:29:18.398721 6144 consensus_queue.cc:1035] T 1aa57620996e42b6adf941e9d606f7e2 P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:29:18.398852 1686 raft_consensus.cc:2932] T f8de1fd0869348e6b0aa131b958ce822 P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 16a97c1394f346f0b987f97cce7371b6 (127.15.255.137) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { replace: true } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } }
W20240502 21:29:18.406003 1885 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.129:37897, user_credentials={real_user=slave}} blocked reactor thread for 223194us
I20240502 21:29:18.406903 5735 consensus_queue.cc:1035] T 444c1cba2a6f4729aa01b75d06b6b517 P e632c553ff4341328acf1cbaca678392 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:29:18.409376 2225 raft_consensus.cc:1261] T 444c1cba2a6f4729aa01b75d06b6b517 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Refusing update from remote peer e632c553ff4341328acf1cbaca678392: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:29:18.410255 5735 consensus_queue.cc:1035] T 444c1cba2a6f4729aa01b75d06b6b517 P e632c553ff4341328acf1cbaca678392 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:29:18.415768 6145 raft_consensus.cc:2932] T 1aa57620996e42b6adf941e9d606f7e2 P 521bb93035324cecbebed2f6f2a630f8 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 16a97c1394f346f0b987f97cce7371b6 (127.15.255.137) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } }
I20240502 21:29:18.416326 1819 raft_consensus.cc:2932] T 1aa57620996e42b6adf941e9d606f7e2 P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 16a97c1394f346f0b987f97cce7371b6 (127.15.255.137) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } }
I20240502 21:29:18.422675 2226 raft_consensus.cc:2932] T 1aa57620996e42b6adf941e9d606f7e2 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 16a97c1394f346f0b987f97cce7371b6 (127.15.255.137) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } }
I20240502 21:29:18.423141 1262 catalog_manager.cc:5484] T 1aa57620996e42b6adf941e9d606f7e2 P 521bb93035324cecbebed2f6f2a630f8 reported cstate change: config changed from index 8 to 9, 16a97c1394f346f0b987f97cce7371b6 (127.15.255.137) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "521bb93035324cecbebed2f6f2a630f8" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 21:29:18.423836 1549 raft_consensus.cc:2932] T 1aa57620996e42b6adf941e9d606f7e2 P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 16a97c1394f346f0b987f97cce7371b6 (127.15.255.137) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } }
I20240502 21:29:18.427692 2225 raft_consensus.cc:2932] T 444c1cba2a6f4729aa01b75d06b6b517 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { replace: true } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } }
I20240502 21:29:18.433288 1412 raft_consensus.cc:2932] T 1aa57620996e42b6adf941e9d606f7e2 P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 16a97c1394f346f0b987f97cce7371b6 (127.15.255.137) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } }
I20240502 21:29:18.437129 4121 raft_consensus.cc:2932] T f8de1fd0869348e6b0aa131b958ce822 P 1fab5794cf1740288abd82f2cebb9b1a [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 16a97c1394f346f0b987f97cce7371b6 (127.15.255.137) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { replace: true } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } }
I20240502 21:29:18.438627 1686 raft_consensus.cc:1261] T f8de1fd0869348e6b0aa131b958ce822 P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 21:29:18.441097 1685 raft_consensus.cc:1261] T 72355dd734ea4aeba031171e8c39696d P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 21:29:18.437623 3855 raft_consensus.cc:2932] T 1aa57620996e42b6adf941e9d606f7e2 P d8d7297186694f77bb98de9174e4279c [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 16a97c1394f346f0b987f97cce7371b6 (127.15.255.137) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } }
I20240502 21:29:18.443234 4120 raft_consensus.cc:1261] T f8de1fd0869348e6b0aa131b958ce822 P 1fab5794cf1740288abd82f2cebb9b1a [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 21:29:18.455282 1685 raft_consensus.cc:1261] T 1aa57620996e42b6adf941e9d606f7e2 P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:29:18.459028 5763 consensus_queue.cc:1035] T 72355dd734ea4aeba031171e8c39696d P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 21:29:18.460042 3986 raft_consensus.cc:2932] T 1aa57620996e42b6adf941e9d606f7e2 P 16a97c1394f346f0b987f97cce7371b6 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 16a97c1394f346f0b987f97cce7371b6 (127.15.255.137) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } }
I20240502 21:29:18.462013 3988 raft_consensus.cc:1261] T 13af1b96121c492c9373f9acbb55725a P 16a97c1394f346f0b987f97cce7371b6 [term 1 LEARNER]: Refusing update from remote peer 153026dd7f8943cc93fedf6f48450fb4: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:29:18.462051 3856 raft_consensus.cc:1261] T 13af1b96121c492c9373f9acbb55725a P d8d7297186694f77bb98de9174e4279c [term 1 FOLLOWER]: Refusing update from remote peer 153026dd7f8943cc93fedf6f48450fb4: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:29:18.462666 6083 consensus_queue.cc:1035] T 13af1b96121c492c9373f9acbb55725a P 153026dd7f8943cc93fedf6f48450fb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:29:18.463271 6083 consensus_queue.cc:1035] T 13af1b96121c492c9373f9acbb55725a P 153026dd7f8943cc93fedf6f48450fb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:29:18.462913 1413 raft_consensus.cc:2932] T 444c1cba2a6f4729aa01b75d06b6b517 P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { replace: true } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } }
W20240502 21:29:18.465833 1480 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.134:40821, user_credentials={real_user=slave}} blocked reactor thread for 210019us
I20240502 21:29:18.466509 1255 catalog_manager.cc:4830] TS a62a935ff43d494ca35d0c417943133d (127.15.255.130:37991): tablet e44bf56d64b24e328803ebd0a3a54ceb (table rebalance_test_table_1 [id=f2592b36bf814d51bd833a6ad315e513]) successfully deleted
I20240502 21:29:18.470465 2087 raft_consensus.cc:2932] T aae7b5a01a0543afb1b5a7f44a9a864c P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, d8d7297186694f77bb98de9174e4279c (127.15.255.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } }
I20240502 21:29:18.471148 2089 consensus_queue.cc:237] T 1aa57620996e42b6adf941e9d606f7e2 P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } }
I20240502 21:29:18.471585 1954 raft_consensus.cc:1261] T 13af1b96121c492c9373f9acbb55725a P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Refusing update from remote peer 153026dd7f8943cc93fedf6f48450fb4: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:29:18.472122 1684 raft_consensus.cc:1261] T 13af1b96121c492c9373f9acbb55725a P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Refusing update from remote peer 153026dd7f8943cc93fedf6f48450fb4: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:29:18.473035 6083 consensus_queue.cc:1035] T 13af1b96121c492c9373f9acbb55725a P 153026dd7f8943cc93fedf6f48450fb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:29:18.473522 6083 consensus_queue.cc:1035] T 13af1b96121c492c9373f9acbb55725a P 153026dd7f8943cc93fedf6f48450fb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:29:18.474012 3988 raft_consensus.cc:1261] T 1aa57620996e42b6adf941e9d606f7e2 P 16a97c1394f346f0b987f97cce7371b6 [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 21:29:18.474232 3856 raft_consensus.cc:1261] T 1aa57620996e42b6adf941e9d606f7e2 P d8d7297186694f77bb98de9174e4279c [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 21:29:18.474684 6145 consensus_queue.cc:1035] T 1aa57620996e42b6adf941e9d606f7e2 P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 21:29:18.475000 6146 consensus_queue.cc:1035] T 1aa57620996e42b6adf941e9d606f7e2 P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 21:29:18.475160 1685 raft_consensus.cc:2932] T 72355dd734ea4aeba031171e8c39696d P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER a62a935ff43d494ca35d0c417943133d (127.15.255.130) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } }
I20240502 21:29:18.478838 2225 raft_consensus.cc:1261] T 1aa57620996e42b6adf941e9d606f7e2 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 21:29:18.480134 6145 consensus_queue.cc:1035] T 1aa57620996e42b6adf941e9d606f7e2 P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 21:29:18.480926 6144 consensus_queue.cc:1035] T f8de1fd0869348e6b0aa131b958ce822 P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 21:29:18.482702 1550 raft_consensus.cc:1261] T 1aa57620996e42b6adf941e9d606f7e2 P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 21:29:18.483248 6146 consensus_queue.cc:1035] T 1aa57620996e42b6adf941e9d606f7e2 P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 21:29:18.485842 1820 raft_consensus.cc:1261] T 1aa57620996e42b6adf941e9d606f7e2 P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 21:29:18.485743 6145 raft_consensus.cc:2932] T 1aa57620996e42b6adf941e9d606f7e2 P 521bb93035324cecbebed2f6f2a630f8 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 16b3a337bc5e462f83ba895ad8660deb (127.15.255.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } }
I20240502 21:29:18.486465 6102 consensus_queue.cc:1035] T 1aa57620996e42b6adf941e9d606f7e2 P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 21:29:18.488534 1413 raft_consensus.cc:1261] T 1aa57620996e42b6adf941e9d606f7e2 P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 21:29:18.490005 1413 raft_consensus.cc:1261] T 13af1b96121c492c9373f9acbb55725a P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Refusing update from remote peer 153026dd7f8943cc93fedf6f48450fb4: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:29:18.490866 1550 raft_consensus.cc:1261] T 13af1b96121c492c9373f9acbb55725a P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Refusing update from remote peer 153026dd7f8943cc93fedf6f48450fb4: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:29:18.491240 1254 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 1aa57620996e42b6adf941e9d606f7e2 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:29:18.491194 6170 consensus_queue.cc:1035] T 13af1b96121c492c9373f9acbb55725a P 153026dd7f8943cc93fedf6f48450fb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20240502 21:29:18.492391 6083 consensus_queue.cc:1035] T 13af1b96121c492c9373f9acbb55725a P 153026dd7f8943cc93fedf6f48450fb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:29:18.492835 3986 raft_consensus.cc:2932] T 1aa57620996e42b6adf941e9d606f7e2 P 16a97c1394f346f0b987f97cce7371b6 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 16b3a337bc5e462f83ba895ad8660deb (127.15.255.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } }
I20240502 21:29:18.494310 2226 raft_consensus.cc:2932] T 1aa57620996e42b6adf941e9d606f7e2 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 16b3a337bc5e462f83ba895ad8660deb (127.15.255.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } }
I20240502 21:29:18.496176 1528 tablet_service.cc:1514] Processing DeleteTablet for tablet f8de1fd0869348e6b0aa131b958ce822 with delete_type TABLET_DATA_TOMBSTONED (TS a62a935ff43d494ca35d0c417943133d not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:43860
I20240502 21:29:18.496567 6155 tablet_replica.cc:330] T f8de1fd0869348e6b0aa131b958ce822 P a62a935ff43d494ca35d0c417943133d: stopping tablet replica
I20240502 21:29:18.496753 6155 raft_consensus.cc:2229] T f8de1fd0869348e6b0aa131b958ce822 P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:29:18.496995 6155 raft_consensus.cc:2258] T f8de1fd0869348e6b0aa131b958ce822 P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:29:18.498071 6155 ts_tablet_manager.cc:1875] T f8de1fd0869348e6b0aa131b958ce822 P a62a935ff43d494ca35d0c417943133d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:29:18.500149 1527 tablet_service.cc:1514] Processing DeleteTablet for tablet 72355dd734ea4aeba031171e8c39696d with delete_type TABLET_DATA_TOMBSTONED (TS a62a935ff43d494ca35d0c417943133d not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:43860
I20240502 21:29:18.500749 1270 catalog_manager.cc:5484] T 1aa57620996e42b6adf941e9d606f7e2 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 reported cstate change: config changed from index 9 to 10, VOTER 16b3a337bc5e462f83ba895ad8660deb (127.15.255.131) evicted. New cstate: current_term: 1 leader_uuid: "521bb93035324cecbebed2f6f2a630f8" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } }
I20240502 21:29:18.501924 6090 consensus_queue.cc:1035] T f8de1fd0869348e6b0aa131b958ce822 P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 21:29:18.502477 6145 consensus_queue.cc:1035] T 1aa57620996e42b6adf941e9d606f7e2 P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 21:29:18.503283 3855 raft_consensus.cc:2932] T 1aa57620996e42b6adf941e9d606f7e2 P d8d7297186694f77bb98de9174e4279c [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 16b3a337bc5e462f83ba895ad8660deb (127.15.255.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } }
I20240502 21:29:18.504464 1414 raft_consensus.cc:2932] T 1aa57620996e42b6adf941e9d606f7e2 P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 16b3a337bc5e462f83ba895ad8660deb (127.15.255.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } }
I20240502 21:29:18.504885 1548 raft_consensus.cc:2932] T 1aa57620996e42b6adf941e9d606f7e2 P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 16b3a337bc5e462f83ba895ad8660deb (127.15.255.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } }
I20240502 21:29:18.506955 6155 ts_tablet_manager.cc:1888] T f8de1fd0869348e6b0aa131b958ce822 P a62a935ff43d494ca35d0c417943133d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240502 21:29:18.507073 6155 log.cc:1196] T f8de1fd0869348e6b0aa131b958ce822 P a62a935ff43d494ca35d0c417943133d: Deleting WAL directory at /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-1/wal/wals/f8de1fd0869348e6b0aa131b958ce822
I20240502 21:29:18.507812 6155 tablet_replica.cc:330] T 72355dd734ea4aeba031171e8c39696d P a62a935ff43d494ca35d0c417943133d: stopping tablet replica
I20240502 21:29:18.508026 6155 raft_consensus.cc:2229] T 72355dd734ea4aeba031171e8c39696d P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:29:18.508296 6155 raft_consensus.cc:2258] T 72355dd734ea4aeba031171e8c39696d P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:29:18.509404 1664 tablet_service.cc:1514] Processing DeleteTablet for tablet 1aa57620996e42b6adf941e9d606f7e2 with delete_type TABLET_DATA_TOMBSTONED (TS 16b3a337bc5e462f83ba895ad8660deb not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:42590
I20240502 21:29:18.509833 6072 tablet_replica.cc:330] T 1aa57620996e42b6adf941e9d606f7e2 P 16b3a337bc5e462f83ba895ad8660deb: stopping tablet replica
I20240502 21:29:18.510048 6072 raft_consensus.cc:2229] T 1aa57620996e42b6adf941e9d606f7e2 P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:29:18.510438 6072 raft_consensus.cc:2258] T 1aa57620996e42b6adf941e9d606f7e2 P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:29:18.511173 6072 ts_tablet_manager.cc:1875] T 1aa57620996e42b6adf941e9d606f7e2 P 16b3a337bc5e462f83ba895ad8660deb: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:29:18.517546 6072 ts_tablet_manager.cc:1888] T 1aa57620996e42b6adf941e9d606f7e2 P 16b3a337bc5e462f83ba895ad8660deb: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240502 21:29:18.519436 6072 log.cc:1196] T 1aa57620996e42b6adf941e9d606f7e2 P 16b3a337bc5e462f83ba895ad8660deb: Deleting WAL directory at /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-2/wal/wals/1aa57620996e42b6adf941e9d606f7e2
I20240502 21:29:18.520351 1257 catalog_manager.cc:4830] TS 16b3a337bc5e462f83ba895ad8660deb (127.15.255.131:36033): tablet 1aa57620996e42b6adf941e9d606f7e2 (table rebalance_test_table_2 [id=900316782c71465f9538d1bbe31708d3]) successfully deleted
I20240502 21:29:18.494818 1820 raft_consensus.cc:2932] T 1aa57620996e42b6adf941e9d606f7e2 P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 16b3a337bc5e462f83ba895ad8660deb (127.15.255.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } }
I20240502 21:29:18.525224 6155 ts_tablet_manager.cc:1875] T 72355dd734ea4aeba031171e8c39696d P a62a935ff43d494ca35d0c417943133d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:29:18.525377 6170 raft_consensus.cc:2932] T 13af1b96121c492c9373f9acbb55725a P 153026dd7f8943cc93fedf6f48450fb4 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 16a97c1394f346f0b987f97cce7371b6 (127.15.255.137) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { replace: true } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } }
I20240502 21:29:18.528754 1413 raft_consensus.cc:2932] T 13af1b96121c492c9373f9acbb55725a P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 16a97c1394f346f0b987f97cce7371b6 (127.15.255.137) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { replace: true } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } }
I20240502 21:29:18.531766 3855 raft_consensus.cc:2932] T 13af1b96121c492c9373f9acbb55725a P d8d7297186694f77bb98de9174e4279c [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 16a97c1394f346f0b987f97cce7371b6 (127.15.255.137) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { replace: true } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } }
I20240502 21:29:18.532547 3988 raft_consensus.cc:2932] T 13af1b96121c492c9373f9acbb55725a P 16a97c1394f346f0b987f97cce7371b6 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 16a97c1394f346f0b987f97cce7371b6 (127.15.255.137) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { replace: true } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } }
I20240502 21:29:18.532698 4121 raft_consensus.cc:2932] T f8de1fd0869348e6b0aa131b958ce822 P 1fab5794cf1740288abd82f2cebb9b1a [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER a62a935ff43d494ca35d0c417943133d (127.15.255.130) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } }
I20240502 21:29:18.534674 1686 raft_consensus.cc:2932] T f8de1fd0869348e6b0aa131b958ce822 P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER a62a935ff43d494ca35d0c417943133d (127.15.255.130) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } }
I20240502 21:29:18.537933 1270 catalog_manager.cc:5484] T 13af1b96121c492c9373f9acbb55725a P d8d7297186694f77bb98de9174e4279c reported cstate change: config changed from index 8 to 9, 16a97c1394f346f0b987f97cce7371b6 (127.15.255.137) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "153026dd7f8943cc93fedf6f48450fb4" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { replace: true } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } }
I20240502 21:29:18.545178 2089 raft_consensus.cc:1261] T 13af1b96121c492c9373f9acbb55725a P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Refusing update from remote peer 153026dd7f8943cc93fedf6f48450fb4: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:29:18.544771 1684 raft_consensus.cc:2932] T 13af1b96121c492c9373f9acbb55725a P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 16a97c1394f346f0b987f97cce7371b6 (127.15.255.137) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { replace: true } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } }
I20240502 21:29:18.545899 6170 consensus_queue.cc:1035] T 13af1b96121c492c9373f9acbb55725a P 153026dd7f8943cc93fedf6f48450fb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:29:18.548926 1954 raft_consensus.cc:2932] T 13af1b96121c492c9373f9acbb55725a P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 16a97c1394f346f0b987f97cce7371b6 (127.15.255.137) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { replace: true } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } }
I20240502 21:29:18.550359 2089 raft_consensus.cc:2932] T 05e885ae741c43808c6dfbcdc5769942 P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 16b3a337bc5e462f83ba895ad8660deb (127.15.255.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } }
I20240502 21:29:18.555330 1255 catalog_manager.cc:4830] TS a62a935ff43d494ca35d0c417943133d (127.15.255.130:37991): tablet f8de1fd0869348e6b0aa131b958ce822 (table rebalance_test_table_6 [id=552d53f5242c4a2eb8e89b63e8eb6a7a]) successfully deleted
I20240502 21:29:18.566817 1548 raft_consensus.cc:2932] T 13af1b96121c492c9373f9acbb55725a P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 16a97c1394f346f0b987f97cce7371b6 (127.15.255.137) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { replace: true } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } }
I20240502 21:29:18.572638 1549 raft_consensus.cc:2932] T 444c1cba2a6f4729aa01b75d06b6b517 P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { replace: true } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } }
I20240502 21:29:18.585352 6155 ts_tablet_manager.cc:1888] T 72355dd734ea4aeba031171e8c39696d P a62a935ff43d494ca35d0c417943133d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240502 21:29:18.585496 6155 log.cc:1196] T 72355dd734ea4aeba031171e8c39696d P a62a935ff43d494ca35d0c417943133d: Deleting WAL directory at /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-1/wal/wals/72355dd734ea4aeba031171e8c39696d
I20240502 21:29:18.586278 1255 catalog_manager.cc:4830] TS a62a935ff43d494ca35d0c417943133d (127.15.255.130:37991): tablet 72355dd734ea4aeba031171e8c39696d (table rebalance_test_table_5 [id=9cb803e5bb2545c8b992c90a894085af]) successfully deleted
I20240502 21:29:18.590234 2087 raft_consensus.cc:2932] T 13af1b96121c492c9373f9acbb55725a P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 16a97c1394f346f0b987f97cce7371b6 (127.15.255.137) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { replace: true } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } }
I20240502 21:29:18.596349 2089 consensus_queue.cc:237] T b300ad3948074c728a25f053512bc37c P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { replace: true } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: true } }
I20240502 21:29:18.603510 3986 raft_consensus.cc:1261] T b300ad3948074c728a25f053512bc37c P 16a97c1394f346f0b987f97cce7371b6 [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
W20240502 21:29:18.603585 2024 consensus_peers.cc:485] T b300ad3948074c728a25f053512bc37c P 521bb93035324cecbebed2f6f2a630f8 -> Peer d8d7297186694f77bb98de9174e4279c (127.15.255.136:41459): Couldn't send request to peer d8d7297186694f77bb98de9174e4279c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b300ad3948074c728a25f053512bc37c. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:29:18.603971 2226 raft_consensus.cc:1261] T b300ad3948074c728a25f053512bc37c P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:29:18.604377 1685 raft_consensus.cc:1261] T b300ad3948074c728a25f053512bc37c P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:29:18.604497 6145 consensus_queue.cc:1035] T b300ad3948074c728a25f053512bc37c P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:29:18.604828 6146 consensus_queue.cc:1035] T b300ad3948074c728a25f053512bc37c P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:29:18.605335 1954 raft_consensus.cc:1261] T b300ad3948074c728a25f053512bc37c P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:29:18.605769 6116 consensus_queue.cc:1035] T b300ad3948074c728a25f053512bc37c P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:29:18.606069 1413 raft_consensus.cc:1261] T b300ad3948074c728a25f053512bc37c P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:29:18.607585 1549 raft_consensus.cc:1261] T b300ad3948074c728a25f053512bc37c P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:29:18.608295 6116 consensus_queue.cc:1035] T b300ad3948074c728a25f053512bc37c P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:29:18.608799 6116 consensus_queue.cc:1035] T b300ad3948074c728a25f053512bc37c P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:29:18.610846 6145 consensus_queue.cc:1035] T b300ad3948074c728a25f053512bc37c P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:29:18.638672 6144 raft_consensus.cc:2932] T b300ad3948074c728a25f053512bc37c P 521bb93035324cecbebed2f6f2a630f8 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER d8d7297186694f77bb98de9174e4279c (127.15.255.136) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { replace: true } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: true } } }
I20240502 21:29:18.641108 1685 raft_consensus.cc:2932] T b300ad3948074c728a25f053512bc37c P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER d8d7297186694f77bb98de9174e4279c (127.15.255.136) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { replace: true } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: true } } }
I20240502 21:29:18.641403 3986 raft_consensus.cc:2932] T b300ad3948074c728a25f053512bc37c P 16a97c1394f346f0b987f97cce7371b6 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER d8d7297186694f77bb98de9174e4279c (127.15.255.136) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { replace: true } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: true } } }
I20240502 21:29:18.642567 2226 raft_consensus.cc:2932] T b300ad3948074c728a25f053512bc37c P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER d8d7297186694f77bb98de9174e4279c (127.15.255.136) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { replace: true } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: true } } }
I20240502 21:29:18.644748 1954 raft_consensus.cc:2932] T b300ad3948074c728a25f053512bc37c P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER d8d7297186694f77bb98de9174e4279c (127.15.255.136) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { replace: true } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: true } } }
I20240502 21:29:18.648362 1262 catalog_manager.cc:5484] T b300ad3948074c728a25f053512bc37c P 16b3a337bc5e462f83ba895ad8660deb reported cstate change: config changed from index 7 to 8, NON_VOTER d8d7297186694f77bb98de9174e4279c (127.15.255.136) added. New cstate: current_term: 1 leader_uuid: "521bb93035324cecbebed2f6f2a630f8" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { replace: true } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: true } } }
I20240502 21:29:18.654354 1413 raft_consensus.cc:2932] T b300ad3948074c728a25f053512bc37c P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER d8d7297186694f77bb98de9174e4279c (127.15.255.136) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { replace: true } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: true } } }
I20240502 21:29:18.654593 1549 raft_consensus.cc:2932] T b300ad3948074c728a25f053512bc37c P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER d8d7297186694f77bb98de9174e4279c (127.15.255.136) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { replace: true } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: true } } }
I20240502 21:29:18.671036 1413 consensus_queue.cc:237] T 815178fbd07b4631b507927683a475c9 P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: NON_VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: true } }
I20240502 21:29:18.673820 3986 raft_consensus.cc:1261] T 815178fbd07b4631b507927683a475c9 P 16a97c1394f346f0b987f97cce7371b6 [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:29:18.674118 1685 raft_consensus.cc:1261] T 815178fbd07b4631b507927683a475c9 P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
W20240502 21:29:18.674412 1350 consensus_peers.cc:485] T 815178fbd07b4631b507927683a475c9 P 7d1e5c1b047443cbb28d7b0a9744babf -> Peer 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138:33375): Couldn't send request to peer 1fab5794cf1740288abd82f2cebb9b1a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 815178fbd07b4631b507927683a475c9. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:29:18.674484 5661 consensus_queue.cc:1035] T 815178fbd07b4631b507927683a475c9 P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:29:18.674926 6051 consensus_queue.cc:1035] T 815178fbd07b4631b507927683a475c9 P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:29:18.676898 2089 raft_consensus.cc:1261] T 815178fbd07b4631b507927683a475c9 P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:29:18.677121 2226 raft_consensus.cc:1261] T 815178fbd07b4631b507927683a475c9 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:29:18.678099 1549 raft_consensus.cc:1261] T 815178fbd07b4631b507927683a475c9 P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:29:18.678169 5661 consensus_queue.cc:1035] T 815178fbd07b4631b507927683a475c9 P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:29:18.678678 5763 consensus_queue.cc:1035] T 815178fbd07b4631b507927683a475c9 P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:29:18.679226 5763 consensus_queue.cc:1035] T 815178fbd07b4631b507927683a475c9 P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:29:18.683867 1954 raft_consensus.cc:1261] T 815178fbd07b4631b507927683a475c9 P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:29:18.684540 5661 consensus_queue.cc:1035] T 815178fbd07b4631b507927683a475c9 P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:29:18.688408 5763 raft_consensus.cc:2932] T 815178fbd07b4631b507927683a475c9 P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: NON_VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: true } } }
I20240502 21:29:18.692399 2089 raft_consensus.cc:2932] T 815178fbd07b4631b507927683a475c9 P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: NON_VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: true } } }
I20240502 21:29:18.695353 1685 raft_consensus.cc:2932] T 815178fbd07b4631b507927683a475c9 P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: NON_VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: true } } }
I20240502 21:29:18.695372 2226 raft_consensus.cc:2932] T 815178fbd07b4631b507927683a475c9 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: NON_VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: true } } }
I20240502 21:29:18.701761 1264 catalog_manager.cc:5484] T 815178fbd07b4631b507927683a475c9 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 reported cstate change: config changed from index 7 to 8, NON_VOTER 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138) added. New cstate: current_term: 1 leader_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: NON_VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: true } } }
I20240502 21:29:18.702106 1548 raft_consensus.cc:2932] T 815178fbd07b4631b507927683a475c9 P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: NON_VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: true } } }
I20240502 21:29:18.708774 3986 raft_consensus.cc:2932] T 815178fbd07b4631b507927683a475c9 P 16a97c1394f346f0b987f97cce7371b6 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: NON_VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: true } } }
I20240502 21:29:18.716125 1954 raft_consensus.cc:2932] T 815178fbd07b4631b507927683a475c9 P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: NON_VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: true } } }
I20240502 21:29:18.789848 1413 consensus_queue.cc:237] T aae7b5a01a0543afb1b5a7f44a9a864c P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } }
I20240502 21:29:18.793166 4121 raft_consensus.cc:1261] T aae7b5a01a0543afb1b5a7f44a9a864c P 1fab5794cf1740288abd82f2cebb9b1a [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 21:29:18.793485 3855 raft_consensus.cc:1261] T aae7b5a01a0543afb1b5a7f44a9a864c P d8d7297186694f77bb98de9174e4279c [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 21:29:18.793881 6051 consensus_queue.cc:1035] T aae7b5a01a0543afb1b5a7f44a9a864c P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 21:29:18.794672 2089 raft_consensus.cc:1261] T aae7b5a01a0543afb1b5a7f44a9a864c P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 21:29:18.795219 1954 raft_consensus.cc:1261] T aae7b5a01a0543afb1b5a7f44a9a864c P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 21:29:18.796133 2226 raft_consensus.cc:1261] T aae7b5a01a0543afb1b5a7f44a9a864c P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 21:29:18.796149 6051 consensus_queue.cc:1035] T aae7b5a01a0543afb1b5a7f44a9a864c P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 21:29:18.796229 1549 raft_consensus.cc:1261] T aae7b5a01a0543afb1b5a7f44a9a864c P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 21:29:18.796726 6051 consensus_queue.cc:1035] T aae7b5a01a0543afb1b5a7f44a9a864c P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 21:29:18.797061 5763 consensus_queue.cc:1035] T aae7b5a01a0543afb1b5a7f44a9a864c P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 21:29:18.797312 6206 consensus_queue.cc:1035] T aae7b5a01a0543afb1b5a7f44a9a864c P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 21:29:18.797597 5661 consensus_queue.cc:1035] T aae7b5a01a0543afb1b5a7f44a9a864c P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 21:29:18.801463 5763 raft_consensus.cc:2932] T aae7b5a01a0543afb1b5a7f44a9a864c P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 16b3a337bc5e462f83ba895ad8660deb (127.15.255.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } }
I20240502 21:29:18.803077 3855 raft_consensus.cc:2932] T aae7b5a01a0543afb1b5a7f44a9a864c P d8d7297186694f77bb98de9174e4279c [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 16b3a337bc5e462f83ba895ad8660deb (127.15.255.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } }
I20240502 21:29:18.803491 1549 raft_consensus.cc:2932] T aae7b5a01a0543afb1b5a7f44a9a864c P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 16b3a337bc5e462f83ba895ad8660deb (127.15.255.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } }
I20240502 21:29:18.803782 2226 raft_consensus.cc:2932] T aae7b5a01a0543afb1b5a7f44a9a864c P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 16b3a337bc5e462f83ba895ad8660deb (127.15.255.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } }
I20240502 21:29:18.805011 2089 raft_consensus.cc:2932] T aae7b5a01a0543afb1b5a7f44a9a864c P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 16b3a337bc5e462f83ba895ad8660deb (127.15.255.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } }
I20240502 21:29:18.808665 1268 catalog_manager.cc:5484] T aae7b5a01a0543afb1b5a7f44a9a864c P d8d7297186694f77bb98de9174e4279c reported cstate change: config changed from index 9 to 10, VOTER 16b3a337bc5e462f83ba895ad8660deb (127.15.255.131) evicted. New cstate: current_term: 1 leader_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } }
I20240502 21:29:18.810870 1255 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet aae7b5a01a0543afb1b5a7f44a9a864c with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:29:18.814111 1954 raft_consensus.cc:2932] T aae7b5a01a0543afb1b5a7f44a9a864c P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 16b3a337bc5e462f83ba895ad8660deb (127.15.255.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } }
I20240502 21:29:18.815110 1664 tablet_service.cc:1514] Processing DeleteTablet for tablet aae7b5a01a0543afb1b5a7f44a9a864c with delete_type TABLET_DATA_TOMBSTONED (TS 16b3a337bc5e462f83ba895ad8660deb not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:42590
I20240502 21:29:18.815605 6072 tablet_replica.cc:330] T aae7b5a01a0543afb1b5a7f44a9a864c P 16b3a337bc5e462f83ba895ad8660deb: stopping tablet replica
I20240502 21:29:18.815809 6072 raft_consensus.cc:2229] T aae7b5a01a0543afb1b5a7f44a9a864c P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:29:18.816133 6072 raft_consensus.cc:2258] T aae7b5a01a0543afb1b5a7f44a9a864c P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:29:18.817317 6072 ts_tablet_manager.cc:1875] T aae7b5a01a0543afb1b5a7f44a9a864c P 16b3a337bc5e462f83ba895ad8660deb: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:29:18.820107 4121 raft_consensus.cc:2932] T aae7b5a01a0543afb1b5a7f44a9a864c P 1fab5794cf1740288abd82f2cebb9b1a [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 16b3a337bc5e462f83ba895ad8660deb (127.15.255.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } }
I20240502 21:29:18.829891 6072 ts_tablet_manager.cc:1888] T aae7b5a01a0543afb1b5a7f44a9a864c P 16b3a337bc5e462f83ba895ad8660deb: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240502 21:29:18.830019 6072 log.cc:1196] T aae7b5a01a0543afb1b5a7f44a9a864c P 16b3a337bc5e462f83ba895ad8660deb: Deleting WAL directory at /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-2/wal/wals/aae7b5a01a0543afb1b5a7f44a9a864c
I20240502 21:29:18.830826 1257 catalog_manager.cc:4830] TS 16b3a337bc5e462f83ba895ad8660deb (127.15.255.131:36033): tablet aae7b5a01a0543afb1b5a7f44a9a864c (table rebalance_test_table_4 [id=d523de924ba14cb493e559296c083a34]) successfully deleted
I20240502 21:29:18.872800 2226 consensus_queue.cc:237] T 4555e51179344acda81670fc9019d968 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: NON_VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: true } }
I20240502 21:29:18.880733 4121 raft_consensus.cc:1261] T 4555e51179344acda81670fc9019d968 P 1fab5794cf1740288abd82f2cebb9b1a [term 1 FOLLOWER]: Refusing update from remote peer 6ab8df3c796f4b6cb3e29d5045ad3ea0: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:29:18.881124 2089 raft_consensus.cc:1261] T 4555e51179344acda81670fc9019d968 P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Refusing update from remote peer 6ab8df3c796f4b6cb3e29d5045ad3ea0: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
W20240502 21:29:18.881421 2158 consensus_peers.cc:485] T 4555e51179344acda81670fc9019d968 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 -> Peer 16a97c1394f346f0b987f97cce7371b6 (127.15.255.137:43717): Couldn't send request to peer 16a97c1394f346f0b987f97cce7371b6. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4555e51179344acda81670fc9019d968. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:29:18.881444 1549 raft_consensus.cc:1261] T 4555e51179344acda81670fc9019d968 P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Refusing update from remote peer 6ab8df3c796f4b6cb3e29d5045ad3ea0: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:29:18.881529 5627 consensus_queue.cc:1035] T 4555e51179344acda81670fc9019d968 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:29:18.881934 1685 raft_consensus.cc:1261] T 4555e51179344acda81670fc9019d968 P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Refusing update from remote peer 6ab8df3c796f4b6cb3e29d5045ad3ea0: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:29:18.882262 5627 consensus_queue.cc:1035] T 4555e51179344acda81670fc9019d968 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:29:18.882741 5833 consensus_queue.cc:1035] T 4555e51179344acda81670fc9019d968 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:29:18.883652 1954 raft_consensus.cc:1261] T 4555e51179344acda81670fc9019d968 P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Refusing update from remote peer 6ab8df3c796f4b6cb3e29d5045ad3ea0: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:29:18.884178 1413 raft_consensus.cc:1261] T 4555e51179344acda81670fc9019d968 P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Refusing update from remote peer 6ab8df3c796f4b6cb3e29d5045ad3ea0: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:29:18.884212 6017 consensus_queue.cc:1035] T 4555e51179344acda81670fc9019d968 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:29:18.884893 6017 consensus_queue.cc:1035] T 4555e51179344acda81670fc9019d968 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:29:18.886303 5833 consensus_queue.cc:1035] T 4555e51179344acda81670fc9019d968 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:29:18.926532 5627 raft_consensus.cc:2932] T 4555e51179344acda81670fc9019d968 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 16a97c1394f346f0b987f97cce7371b6 (127.15.255.137) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: NON_VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: true } } }
I20240502 21:29:18.927332 1549 raft_consensus.cc:2932] T 4555e51179344acda81670fc9019d968 P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 16a97c1394f346f0b987f97cce7371b6 (127.15.255.137) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: NON_VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: true } } }
I20240502 21:29:18.927721 1413 raft_consensus.cc:2932] T 4555e51179344acda81670fc9019d968 P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 16a97c1394f346f0b987f97cce7371b6 (127.15.255.137) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: NON_VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: true } } }
I20240502 21:29:18.934636 1262 catalog_manager.cc:5484] T 4555e51179344acda81670fc9019d968 P a62a935ff43d494ca35d0c417943133d reported cstate change: config changed from index 7 to 8, NON_VOTER 16a97c1394f346f0b987f97cce7371b6 (127.15.255.137) added. New cstate: current_term: 1 leader_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: NON_VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: true } } }
I20240502 21:29:18.937374 4121 raft_consensus.cc:2932] T 4555e51179344acda81670fc9019d968 P 1fab5794cf1740288abd82f2cebb9b1a [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 16a97c1394f346f0b987f97cce7371b6 (127.15.255.137) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: NON_VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: true } } }
I20240502 21:29:18.937327 1954 raft_consensus.cc:2932] T 4555e51179344acda81670fc9019d968 P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 16a97c1394f346f0b987f97cce7371b6 (127.15.255.137) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: NON_VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: true } } }
I20240502 21:29:18.948252 2089 raft_consensus.cc:2932] T 4555e51179344acda81670fc9019d968 P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 16a97c1394f346f0b987f97cce7371b6 (127.15.255.137) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: NON_VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: true } } }
I20240502 21:29:18.941251 1685 raft_consensus.cc:2932] T 4555e51179344acda81670fc9019d968 P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 16a97c1394f346f0b987f97cce7371b6 (127.15.255.137) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: NON_VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: true } } }
I20240502 21:29:18.967454 1820 consensus_queue.cc:237] T 13af1b96121c492c9373f9acbb55725a P 153026dd7f8943cc93fedf6f48450fb4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } }
I20240502 21:29:18.970461 3855 raft_consensus.cc:1261] T 13af1b96121c492c9373f9acbb55725a P d8d7297186694f77bb98de9174e4279c [term 1 FOLLOWER]: Refusing update from remote peer 153026dd7f8943cc93fedf6f48450fb4: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 21:29:18.970510 3986 raft_consensus.cc:1261] T 13af1b96121c492c9373f9acbb55725a P 16a97c1394f346f0b987f97cce7371b6 [term 1 FOLLOWER]: Refusing update from remote peer 153026dd7f8943cc93fedf6f48450fb4: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 21:29:18.970717 2087 raft_consensus.cc:1261] T 13af1b96121c492c9373f9acbb55725a P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Refusing update from remote peer 153026dd7f8943cc93fedf6f48450fb4: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 21:29:18.971061 6170 consensus_queue.cc:1035] T 13af1b96121c492c9373f9acbb55725a P 153026dd7f8943cc93fedf6f48450fb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 21:29:18.972538 1954 raft_consensus.cc:1261] T 13af1b96121c492c9373f9acbb55725a P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Refusing update from remote peer 153026dd7f8943cc93fedf6f48450fb4: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 21:29:18.972924 6083 consensus_queue.cc:1035] T 13af1b96121c492c9373f9acbb55725a P 153026dd7f8943cc93fedf6f48450fb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 21:29:18.973503 6083 consensus_queue.cc:1035] T 13af1b96121c492c9373f9acbb55725a P 153026dd7f8943cc93fedf6f48450fb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 21:29:18.974821 6170 consensus_queue.cc:1035] T 13af1b96121c492c9373f9acbb55725a P 153026dd7f8943cc93fedf6f48450fb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 21:29:18.977219 6170 raft_consensus.cc:2932] T 13af1b96121c492c9373f9acbb55725a P 153026dd7f8943cc93fedf6f48450fb4 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER a62a935ff43d494ca35d0c417943133d (127.15.255.130) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } }
I20240502 21:29:18.979295 1413 raft_consensus.cc:1261] T 13af1b96121c492c9373f9acbb55725a P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Refusing update from remote peer 153026dd7f8943cc93fedf6f48450fb4: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 21:29:18.979153 1955 consensus_queue.cc:237] T 444c1cba2a6f4729aa01b75d06b6b517 P e632c553ff4341328acf1cbaca678392 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } }
I20240502 21:29:18.980003 6178 consensus_queue.cc:1035] T 13af1b96121c492c9373f9acbb55725a P 153026dd7f8943cc93fedf6f48450fb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 21:29:18.980297 1685 raft_consensus.cc:1261] T 13af1b96121c492c9373f9acbb55725a P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Refusing update from remote peer 153026dd7f8943cc93fedf6f48450fb4: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 21:29:18.981819 6178 consensus_queue.cc:1035] T 13af1b96121c492c9373f9acbb55725a P 153026dd7f8943cc93fedf6f48450fb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 21:29:18.982126 1413 raft_consensus.cc:2932] T 13af1b96121c492c9373f9acbb55725a P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER a62a935ff43d494ca35d0c417943133d (127.15.255.130) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } }
I20240502 21:29:18.982360 1954 raft_consensus.cc:2932] T 13af1b96121c492c9373f9acbb55725a P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER a62a935ff43d494ca35d0c417943133d (127.15.255.130) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } }
I20240502 21:29:18.983528 2087 consensus_queue.cc:237] T e45af71e3fc14c579b32ba4dbe2adf78 P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { replace: true } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: NON_VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: true } }
I20240502 21:29:18.984282 3986 raft_consensus.cc:2932] T 13af1b96121c492c9373f9acbb55725a P 16a97c1394f346f0b987f97cce7371b6 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER a62a935ff43d494ca35d0c417943133d (127.15.255.130) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } }
I20240502 21:29:18.985819 3855 raft_consensus.cc:1261] T 444c1cba2a6f4729aa01b75d06b6b517 P d8d7297186694f77bb98de9174e4279c [term 1 FOLLOWER]: Refusing update from remote peer e632c553ff4341328acf1cbaca678392: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 21:29:18.987061 1684 raft_consensus.cc:1261] T 444c1cba2a6f4729aa01b75d06b6b517 P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Refusing update from remote peer e632c553ff4341328acf1cbaca678392: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 21:29:18.987134 4121 raft_consensus.cc:1261] T 444c1cba2a6f4729aa01b75d06b6b517 P 1fab5794cf1740288abd82f2cebb9b1a [term 1 FOLLOWER]: Refusing update from remote peer e632c553ff4341328acf1cbaca678392: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 21:29:18.987704 6152 consensus_queue.cc:1035] T 444c1cba2a6f4729aa01b75d06b6b517 P e632c553ff4341328acf1cbaca678392 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 21:29:18.988127 1685 raft_consensus.cc:2932] T 13af1b96121c492c9373f9acbb55725a P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER a62a935ff43d494ca35d0c417943133d (127.15.255.130) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } }
I20240502 21:29:18.989450 1414 raft_consensus.cc:1261] T 444c1cba2a6f4729aa01b75d06b6b517 P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Refusing update from remote peer e632c553ff4341328acf1cbaca678392: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 21:29:18.990190 5735 consensus_queue.cc:1035] T 444c1cba2a6f4729aa01b75d06b6b517 P e632c553ff4341328acf1cbaca678392 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 21:29:18.990521 6149 consensus_queue.cc:1035] T 444c1cba2a6f4729aa01b75d06b6b517 P e632c553ff4341328acf1cbaca678392 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 21:29:18.990715 1268 catalog_manager.cc:5484] T 13af1b96121c492c9373f9acbb55725a P 7d1e5c1b047443cbb28d7b0a9744babf reported cstate change: config changed from index 9 to 10, VOTER a62a935ff43d494ca35d0c417943133d (127.15.255.130) evicted. New cstate: current_term: 1 leader_uuid: "153026dd7f8943cc93fedf6f48450fb4" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } }
I20240502 21:29:18.991897 2226 raft_consensus.cc:1261] T 444c1cba2a6f4729aa01b75d06b6b517 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Refusing update from remote peer e632c553ff4341328acf1cbaca678392: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 21:29:18.995057 2089 raft_consensus.cc:1261] T 444c1cba2a6f4729aa01b75d06b6b517 P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Refusing update from remote peer e632c553ff4341328acf1cbaca678392: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 21:29:18.994607 3854 raft_consensus.cc:2932] T 13af1b96121c492c9373f9acbb55725a P d8d7297186694f77bb98de9174e4279c [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER a62a935ff43d494ca35d0c417943133d (127.15.255.130) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } }
I20240502 21:29:18.995931 1414 raft_consensus.cc:1261] T e45af71e3fc14c579b32ba4dbe2adf78 P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:29:18.996541 6102 consensus_queue.cc:1035] T e45af71e3fc14c579b32ba4dbe2adf78 P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:29:18.997043 5735 consensus_queue.cc:1035] T 444c1cba2a6f4729aa01b75d06b6b517 P e632c553ff4341328acf1cbaca678392 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 21:29:18.997819 6150 consensus_queue.cc:1035] T 444c1cba2a6f4729aa01b75d06b6b517 P e632c553ff4341328acf1cbaca678392 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 21:29:18.998533 3988 raft_consensus.cc:1261] T e45af71e3fc14c579b32ba4dbe2adf78 P 16a97c1394f346f0b987f97cce7371b6 [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:29:18.999735 2225 raft_consensus.cc:1261] T e45af71e3fc14c579b32ba4dbe2adf78 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:29:18.999910 1549 raft_consensus.cc:1261] T e45af71e3fc14c579b32ba4dbe2adf78 P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:29:19.000049 1820 raft_consensus.cc:1261] T e45af71e3fc14c579b32ba4dbe2adf78 P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:29:19.000116 1256 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 13af1b96121c492c9373f9acbb55725a with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:29:19.000655 6146 consensus_queue.cc:1035] T e45af71e3fc14c579b32ba4dbe2adf78 P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:29:19.001174 6146 consensus_queue.cc:1035] T e45af71e3fc14c579b32ba4dbe2adf78 P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:29:19.001636 6145 consensus_queue.cc:1035] T e45af71e3fc14c579b32ba4dbe2adf78 P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:29:19.003765 6145 consensus_queue.cc:1035] T e45af71e3fc14c579b32ba4dbe2adf78 P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240502 21:29:19.004614 2025 consensus_peers.cc:485] T e45af71e3fc14c579b32ba4dbe2adf78 P 521bb93035324cecbebed2f6f2a630f8 -> Peer 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138:33375): Couldn't send request to peer 1fab5794cf1740288abd82f2cebb9b1a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e45af71e3fc14c579b32ba4dbe2adf78. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:29:19.003897 2087 raft_consensus.cc:2932] T 13af1b96121c492c9373f9acbb55725a P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER a62a935ff43d494ca35d0c417943133d (127.15.255.130) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } }
I20240502 21:29:19.006945 6149 raft_consensus.cc:2932] T 444c1cba2a6f4729aa01b75d06b6b517 P e632c553ff4341328acf1cbaca678392 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER a62a935ff43d494ca35d0c417943133d (127.15.255.130) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } }
I20240502 21:29:19.007630 2226 raft_consensus.cc:2932] T 444c1cba2a6f4729aa01b75d06b6b517 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER a62a935ff43d494ca35d0c417943133d (127.15.255.130) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } }
I20240502 21:29:19.009688 1413 raft_consensus.cc:2932] T 444c1cba2a6f4729aa01b75d06b6b517 P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER a62a935ff43d494ca35d0c417943133d (127.15.255.130) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } }
I20240502 21:29:19.011592 1254 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 444c1cba2a6f4729aa01b75d06b6b517 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:29:19.012773 1684 raft_consensus.cc:2932] T 444c1cba2a6f4729aa01b75d06b6b517 P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER a62a935ff43d494ca35d0c417943133d (127.15.255.130) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } }
I20240502 21:29:19.017450 6152 consensus_queue.cc:1035] T 444c1cba2a6f4729aa01b75d06b6b517 P e632c553ff4341328acf1cbaca678392 [LEADER]: Connected to new peer: Peer: permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 21:29:19.020870 1527 tablet_service.cc:1514] Processing DeleteTablet for tablet 13af1b96121c492c9373f9acbb55725a with delete_type TABLET_DATA_TOMBSTONED (TS a62a935ff43d494ca35d0c417943133d not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:43860
I20240502 21:29:19.021339 6155 tablet_replica.cc:330] T 13af1b96121c492c9373f9acbb55725a P a62a935ff43d494ca35d0c417943133d: stopping tablet replica
I20240502 21:29:19.021539 6155 raft_consensus.cc:2229] T 13af1b96121c492c9373f9acbb55725a P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:29:19.021858 6155 raft_consensus.cc:2258] T 13af1b96121c492c9373f9acbb55725a P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:29:19.022920 6155 ts_tablet_manager.cc:1875] T 13af1b96121c492c9373f9acbb55725a P a62a935ff43d494ca35d0c417943133d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:29:19.027190 4121 raft_consensus.cc:2932] T 444c1cba2a6f4729aa01b75d06b6b517 P 1fab5794cf1740288abd82f2cebb9b1a [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER a62a935ff43d494ca35d0c417943133d (127.15.255.130) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } }
I20240502 21:29:19.028169 1264 catalog_manager.cc:5484] T 444c1cba2a6f4729aa01b75d06b6b517 P 16b3a337bc5e462f83ba895ad8660deb reported cstate change: config changed from index 9 to 10, VOTER a62a935ff43d494ca35d0c417943133d (127.15.255.130) evicted. New cstate: current_term: 1 leader_uuid: "e632c553ff4341328acf1cbaca678392" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } }
I20240502 21:29:19.032505 1685 raft_consensus.cc:1261] T e45af71e3fc14c579b32ba4dbe2adf78 P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:29:19.033123 6145 consensus_queue.cc:1035] T e45af71e3fc14c579b32ba4dbe2adf78 P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:29:19.030792 2089 raft_consensus.cc:2932] T 444c1cba2a6f4729aa01b75d06b6b517 P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER a62a935ff43d494ca35d0c417943133d (127.15.255.130) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } }
I20240502 21:29:19.032308 6155 ts_tablet_manager.cc:1888] T 13af1b96121c492c9373f9acbb55725a P a62a935ff43d494ca35d0c417943133d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240502 21:29:19.036535 6155 log.cc:1196] T 13af1b96121c492c9373f9acbb55725a P a62a935ff43d494ca35d0c417943133d: Deleting WAL directory at /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-1/wal/wals/13af1b96121c492c9373f9acbb55725a
I20240502 21:29:19.039811 1255 catalog_manager.cc:4830] TS a62a935ff43d494ca35d0c417943133d (127.15.255.130:37991): tablet 13af1b96121c492c9373f9acbb55725a (table rebalance_test_table_0 [id=82737b0de4b945dc8ca0c7d6b92af37c]) successfully deleted
I20240502 21:29:19.039851 1527 tablet_service.cc:1514] Processing DeleteTablet for tablet 444c1cba2a6f4729aa01b75d06b6b517 with delete_type TABLET_DATA_TOMBSTONED (TS a62a935ff43d494ca35d0c417943133d not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:43860
I20240502 21:29:19.038762 3855 raft_consensus.cc:2932] T 444c1cba2a6f4729aa01b75d06b6b517 P d8d7297186694f77bb98de9174e4279c [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER a62a935ff43d494ca35d0c417943133d (127.15.255.130) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } }
I20240502 21:29:19.040199 6155 tablet_replica.cc:330] T 444c1cba2a6f4729aa01b75d06b6b517 P a62a935ff43d494ca35d0c417943133d: stopping tablet replica
I20240502 21:29:19.040503 6155 raft_consensus.cc:2229] T 444c1cba2a6f4729aa01b75d06b6b517 P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:29:19.041039 6155 raft_consensus.cc:2258] T 444c1cba2a6f4729aa01b75d06b6b517 P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:29:19.042318 6155 ts_tablet_manager.cc:1875] T 444c1cba2a6f4729aa01b75d06b6b517 P a62a935ff43d494ca35d0c417943133d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:29:19.060519 6155 ts_tablet_manager.cc:1888] T 444c1cba2a6f4729aa01b75d06b6b517 P a62a935ff43d494ca35d0c417943133d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240502 21:29:19.060729 6155 log.cc:1196] T 444c1cba2a6f4729aa01b75d06b6b517 P a62a935ff43d494ca35d0c417943133d: Deleting WAL directory at /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-1/wal/wals/444c1cba2a6f4729aa01b75d06b6b517
I20240502 21:29:19.061512 1255 catalog_manager.cc:4830] TS a62a935ff43d494ca35d0c417943133d (127.15.255.130:37991): tablet 444c1cba2a6f4729aa01b75d06b6b517 (table rebalance_test_table_1 [id=f2592b36bf814d51bd833a6ad315e513]) successfully deleted
I20240502 21:29:19.067054 6292 ts_tablet_manager.cc:902] T b300ad3948074c728a25f053512bc37c P d8d7297186694f77bb98de9174e4279c: Initiating tablet copy from peer 521bb93035324cecbebed2f6f2a630f8 (127.15.255.134:40821)
I20240502 21:29:19.069830 6293 ts_tablet_manager.cc:902] T 815178fbd07b4631b507927683a475c9 P 1fab5794cf1740288abd82f2cebb9b1a: Initiating tablet copy from peer 7d1e5c1b047443cbb28d7b0a9744babf (127.15.255.129:37897)
I20240502 21:29:19.070798 6146 raft_consensus.cc:2932] T e45af71e3fc14c579b32ba4dbe2adf78 P 521bb93035324cecbebed2f6f2a630f8 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { replace: true } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: NON_VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: true } } }
I20240502 21:29:19.073544 1549 raft_consensus.cc:2932] T e45af71e3fc14c579b32ba4dbe2adf78 P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { replace: true } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: NON_VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: true } } }
I20240502 21:29:19.074689 1685 raft_consensus.cc:2932] T e45af71e3fc14c579b32ba4dbe2adf78 P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { replace: true } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: NON_VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: true } } }
I20240502 21:29:19.079579 1262 catalog_manager.cc:5484] T e45af71e3fc14c579b32ba4dbe2adf78 P a62a935ff43d494ca35d0c417943133d reported cstate change: config changed from index 7 to 8, NON_VOTER 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138) added. New cstate: current_term: 1 leader_uuid: "521bb93035324cecbebed2f6f2a630f8" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { replace: true } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: NON_VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: true } } }
I20240502 21:29:19.083091 1820 raft_consensus.cc:2932] T e45af71e3fc14c579b32ba4dbe2adf78 P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { replace: true } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: NON_VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: true } } }
I20240502 21:29:19.085721 2225 raft_consensus.cc:2932] T e45af71e3fc14c579b32ba4dbe2adf78 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { replace: true } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: NON_VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: true } } }
I20240502 21:29:19.090482 1414 raft_consensus.cc:2932] T e45af71e3fc14c579b32ba4dbe2adf78 P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { replace: true } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: NON_VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: true } } }
I20240502 21:29:19.093220 3988 raft_consensus.cc:2932] T e45af71e3fc14c579b32ba4dbe2adf78 P 16a97c1394f346f0b987f97cce7371b6 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { replace: true } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: NON_VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: true } } }
I20240502 21:29:19.098258 6292 tablet_copy_client.cc:308] T b300ad3948074c728a25f053512bc37c P d8d7297186694f77bb98de9174e4279c: tablet copy: Beginning tablet copy session from remote peer at address 127.15.255.134:40821
I20240502 21:29:19.099423 2113 tablet_copy_service.cc:140] P 521bb93035324cecbebed2f6f2a630f8: Received BeginTabletCopySession request for tablet b300ad3948074c728a25f053512bc37c from peer d8d7297186694f77bb98de9174e4279c ({username='slave'} at 127.15.255.136:49789)
I20240502 21:29:19.099584 2113 tablet_copy_service.cc:161] P 521bb93035324cecbebed2f6f2a630f8: Beginning new tablet copy session on tablet b300ad3948074c728a25f053512bc37c from peer d8d7297186694f77bb98de9174e4279c at {username='slave'} at 127.15.255.136:49789: session id = d8d7297186694f77bb98de9174e4279c-b300ad3948074c728a25f053512bc37c
I20240502 21:29:19.102358 2113 tablet_copy_source_session.cc:205] T b300ad3948074c728a25f053512bc37c P 521bb93035324cecbebed2f6f2a630f8: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:29:19.104408 6292 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b300ad3948074c728a25f053512bc37c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:29:19.117601 6292 tablet_copy_client.cc:783] T b300ad3948074c728a25f053512bc37c P d8d7297186694f77bb98de9174e4279c: tablet copy: Starting download of 0 data blocks...
I20240502 21:29:19.117873 6292 tablet_copy_client.cc:647] T b300ad3948074c728a25f053512bc37c P d8d7297186694f77bb98de9174e4279c: tablet copy: Starting download of 1 WAL segments...
I20240502 21:29:19.119519 6292 tablet_copy_client.cc:523] T b300ad3948074c728a25f053512bc37c P d8d7297186694f77bb98de9174e4279c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:29:19.132939 6292 tablet_bootstrap.cc:492] T b300ad3948074c728a25f053512bc37c P d8d7297186694f77bb98de9174e4279c: Bootstrap starting.
I20240502 21:29:19.146600 6293 tablet_copy_client.cc:308] T 815178fbd07b4631b507927683a475c9 P 1fab5794cf1740288abd82f2cebb9b1a: tablet copy: Beginning tablet copy session from remote peer at address 127.15.255.129:37897
I20240502 21:29:19.147390 1434 tablet_copy_service.cc:140] P 7d1e5c1b047443cbb28d7b0a9744babf: Received BeginTabletCopySession request for tablet 815178fbd07b4631b507927683a475c9 from peer 1fab5794cf1740288abd82f2cebb9b1a ({username='slave'} at 127.15.255.138:36825)
I20240502 21:29:19.147559 1434 tablet_copy_service.cc:161] P 7d1e5c1b047443cbb28d7b0a9744babf: Beginning new tablet copy session on tablet 815178fbd07b4631b507927683a475c9 from peer 1fab5794cf1740288abd82f2cebb9b1a at {username='slave'} at 127.15.255.138:36825: session id = 1fab5794cf1740288abd82f2cebb9b1a-815178fbd07b4631b507927683a475c9
I20240502 21:29:19.150060 1434 tablet_copy_source_session.cc:205] T 815178fbd07b4631b507927683a475c9 P 7d1e5c1b047443cbb28d7b0a9744babf: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:29:19.151381 6293 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 815178fbd07b4631b507927683a475c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:29:19.156281 6293 tablet_copy_client.cc:783] T 815178fbd07b4631b507927683a475c9 P 1fab5794cf1740288abd82f2cebb9b1a: tablet copy: Starting download of 0 data blocks...
I20240502 21:29:19.156529 6293 tablet_copy_client.cc:647] T 815178fbd07b4631b507927683a475c9 P 1fab5794cf1740288abd82f2cebb9b1a: tablet copy: Starting download of 1 WAL segments...
I20240502 21:29:19.158672 6293 tablet_copy_client.cc:523] T 815178fbd07b4631b507927683a475c9 P 1fab5794cf1740288abd82f2cebb9b1a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:29:19.161096 6293 tablet_bootstrap.cc:492] T 815178fbd07b4631b507927683a475c9 P 1fab5794cf1740288abd82f2cebb9b1a: Bootstrap starting.
I20240502 21:29:19.202510 6293 tablet_bootstrap.cc:492] T 815178fbd07b4631b507927683a475c9 P 1fab5794cf1740288abd82f2cebb9b1a: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:29:19.202991 6293 tablet_bootstrap.cc:492] T 815178fbd07b4631b507927683a475c9 P 1fab5794cf1740288abd82f2cebb9b1a: Bootstrap complete.
I20240502 21:29:19.203336 6293 ts_tablet_manager.cc:1367] T 815178fbd07b4631b507927683a475c9 P 1fab5794cf1740288abd82f2cebb9b1a: Time spent bootstrapping tablet: real 0.042s user 0.034s sys 0.007s
I20240502 21:29:19.204176 6293 raft_consensus.cc:348] T 815178fbd07b4631b507927683a475c9 P 1fab5794cf1740288abd82f2cebb9b1a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: NON_VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: true } }
I20240502 21:29:19.204598 6293 raft_consensus.cc:729] T 815178fbd07b4631b507927683a475c9 P 1fab5794cf1740288abd82f2cebb9b1a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1fab5794cf1740288abd82f2cebb9b1a, State: Initialized, Role: LEARNER
I20240502 21:29:19.204866 6293 consensus_queue.cc:260] T 815178fbd07b4631b507927683a475c9 P 1fab5794cf1740288abd82f2cebb9b1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: NON_VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: true } }
I20240502 21:29:19.205685 6293 ts_tablet_manager.cc:1398] T 815178fbd07b4631b507927683a475c9 P 1fab5794cf1740288abd82f2cebb9b1a: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240502 21:29:19.206544 1434 tablet_copy_service.cc:342] P 7d1e5c1b047443cbb28d7b0a9744babf: Request end of tablet copy session 1fab5794cf1740288abd82f2cebb9b1a-815178fbd07b4631b507927683a475c9 received from {username='slave'} at 127.15.255.138:36825
I20240502 21:29:19.206688 1434 tablet_copy_service.cc:433] P 7d1e5c1b047443cbb28d7b0a9744babf: ending tablet copy session 1fab5794cf1740288abd82f2cebb9b1a-815178fbd07b4631b507927683a475c9 on tablet 815178fbd07b4631b507927683a475c9 with peer 1fab5794cf1740288abd82f2cebb9b1a
I20240502 21:29:19.222298 6292 tablet_bootstrap.cc:492] T b300ad3948074c728a25f053512bc37c P d8d7297186694f77bb98de9174e4279c: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:29:19.222690 6292 tablet_bootstrap.cc:492] T b300ad3948074c728a25f053512bc37c P d8d7297186694f77bb98de9174e4279c: Bootstrap complete.
I20240502 21:29:19.222952 6292 ts_tablet_manager.cc:1367] T b300ad3948074c728a25f053512bc37c P d8d7297186694f77bb98de9174e4279c: Time spent bootstrapping tablet: real 0.090s user 0.041s sys 0.006s
I20240502 21:29:19.223559 6292 raft_consensus.cc:348] T b300ad3948074c728a25f053512bc37c P d8d7297186694f77bb98de9174e4279c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { replace: true } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: true } }
I20240502 21:29:19.223803 6292 raft_consensus.cc:729] T b300ad3948074c728a25f053512bc37c P d8d7297186694f77bb98de9174e4279c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d8d7297186694f77bb98de9174e4279c, State: Initialized, Role: LEARNER
I20240502 21:29:19.224045 6292 consensus_queue.cc:260] T b300ad3948074c728a25f053512bc37c P d8d7297186694f77bb98de9174e4279c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { replace: true } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: true } }
I20240502 21:29:19.224795 6292 ts_tablet_manager.cc:1398] T b300ad3948074c728a25f053512bc37c P d8d7297186694f77bb98de9174e4279c: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240502 21:29:19.225567 2113 tablet_copy_service.cc:342] P 521bb93035324cecbebed2f6f2a630f8: Request end of tablet copy session d8d7297186694f77bb98de9174e4279c-b300ad3948074c728a25f053512bc37c received from {username='slave'} at 127.15.255.136:49789
I20240502 21:29:19.225688 2113 tablet_copy_service.cc:433] P 521bb93035324cecbebed2f6f2a630f8: ending tablet copy session d8d7297186694f77bb98de9174e4279c-b300ad3948074c728a25f053512bc37c on tablet b300ad3948074c728a25f053512bc37c with peer d8d7297186694f77bb98de9174e4279c
I20240502 21:29:19.384622 6360 ts_tablet_manager.cc:902] T 4555e51179344acda81670fc9019d968 P 16a97c1394f346f0b987f97cce7371b6: Initiating tablet copy from peer 6ab8df3c796f4b6cb3e29d5045ad3ea0 (127.15.255.135:34701)
I20240502 21:29:19.386329 6360 tablet_copy_client.cc:308] T 4555e51179344acda81670fc9019d968 P 16a97c1394f346f0b987f97cce7371b6: tablet copy: Beginning tablet copy session from remote peer at address 127.15.255.135:34701
I20240502 21:29:19.387073 2249 tablet_copy_service.cc:140] P 6ab8df3c796f4b6cb3e29d5045ad3ea0: Received BeginTabletCopySession request for tablet 4555e51179344acda81670fc9019d968 from peer 16a97c1394f346f0b987f97cce7371b6 ({username='slave'} at 127.15.255.137:35291)
I20240502 21:29:19.387259 2249 tablet_copy_service.cc:161] P 6ab8df3c796f4b6cb3e29d5045ad3ea0: Beginning new tablet copy session on tablet 4555e51179344acda81670fc9019d968 from peer 16a97c1394f346f0b987f97cce7371b6 at {username='slave'} at 127.15.255.137:35291: session id = 16a97c1394f346f0b987f97cce7371b6-4555e51179344acda81670fc9019d968
I20240502 21:29:19.389485 2249 tablet_copy_source_session.cc:205] T 4555e51179344acda81670fc9019d968 P 6ab8df3c796f4b6cb3e29d5045ad3ea0: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:29:19.390632 6360 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4555e51179344acda81670fc9019d968. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:29:19.395223 6360 tablet_copy_client.cc:783] T 4555e51179344acda81670fc9019d968 P 16a97c1394f346f0b987f97cce7371b6: tablet copy: Starting download of 0 data blocks...
I20240502 21:29:19.395475 6360 tablet_copy_client.cc:647] T 4555e51179344acda81670fc9019d968 P 16a97c1394f346f0b987f97cce7371b6: tablet copy: Starting download of 1 WAL segments...
I20240502 21:29:19.397068 6360 tablet_copy_client.cc:523] T 4555e51179344acda81670fc9019d968 P 16a97c1394f346f0b987f97cce7371b6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:29:19.399423 6360 tablet_bootstrap.cc:492] T 4555e51179344acda81670fc9019d968 P 16a97c1394f346f0b987f97cce7371b6: Bootstrap starting.
I20240502 21:29:19.422650 6293 ts_tablet_manager.cc:902] T e45af71e3fc14c579b32ba4dbe2adf78 P 1fab5794cf1740288abd82f2cebb9b1a: Initiating tablet copy from peer 521bb93035324cecbebed2f6f2a630f8 (127.15.255.134:40821)
I20240502 21:29:19.429162 6293 tablet_copy_client.cc:308] T e45af71e3fc14c579b32ba4dbe2adf78 P 1fab5794cf1740288abd82f2cebb9b1a: tablet copy: Beginning tablet copy session from remote peer at address 127.15.255.134:40821
I20240502 21:29:19.429791 2113 tablet_copy_service.cc:140] P 521bb93035324cecbebed2f6f2a630f8: Received BeginTabletCopySession request for tablet e45af71e3fc14c579b32ba4dbe2adf78 from peer 1fab5794cf1740288abd82f2cebb9b1a ({username='slave'} at 127.15.255.138:37811)
I20240502 21:29:19.429973 2113 tablet_copy_service.cc:161] P 521bb93035324cecbebed2f6f2a630f8: Beginning new tablet copy session on tablet e45af71e3fc14c579b32ba4dbe2adf78 from peer 1fab5794cf1740288abd82f2cebb9b1a at {username='slave'} at 127.15.255.138:37811: session id = 1fab5794cf1740288abd82f2cebb9b1a-e45af71e3fc14c579b32ba4dbe2adf78
I20240502 21:29:19.432703 6360 tablet_bootstrap.cc:492] T 4555e51179344acda81670fc9019d968 P 16a97c1394f346f0b987f97cce7371b6: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:29:19.432971 2113 tablet_copy_source_session.cc:205] T e45af71e3fc14c579b32ba4dbe2adf78 P 521bb93035324cecbebed2f6f2a630f8: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:29:19.433141 6360 tablet_bootstrap.cc:492] T 4555e51179344acda81670fc9019d968 P 16a97c1394f346f0b987f97cce7371b6: Bootstrap complete.
I20240502 21:29:19.433439 6360 ts_tablet_manager.cc:1367] T 4555e51179344acda81670fc9019d968 P 16a97c1394f346f0b987f97cce7371b6: Time spent bootstrapping tablet: real 0.034s user 0.031s sys 0.004s
I20240502 21:29:19.434156 6360 raft_consensus.cc:348] T 4555e51179344acda81670fc9019d968 P 16a97c1394f346f0b987f97cce7371b6 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: NON_VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: true } }
I20240502 21:29:19.434399 6293 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e45af71e3fc14c579b32ba4dbe2adf78. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:29:19.434464 6360 raft_consensus.cc:729] T 4555e51179344acda81670fc9019d968 P 16a97c1394f346f0b987f97cce7371b6 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 16a97c1394f346f0b987f97cce7371b6, State: Initialized, Role: LEARNER
I20240502 21:29:19.434676 6360 consensus_queue.cc:260] T 4555e51179344acda81670fc9019d968 P 16a97c1394f346f0b987f97cce7371b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: NON_VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: true } }
I20240502 21:29:19.435250 6360 ts_tablet_manager.cc:1398] T 4555e51179344acda81670fc9019d968 P 16a97c1394f346f0b987f97cce7371b6: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 21:29:19.436272 2249 tablet_copy_service.cc:342] P 6ab8df3c796f4b6cb3e29d5045ad3ea0: Request end of tablet copy session 16a97c1394f346f0b987f97cce7371b6-4555e51179344acda81670fc9019d968 received from {username='slave'} at 127.15.255.137:35291
I20240502 21:29:19.436452 2249 tablet_copy_service.cc:433] P 6ab8df3c796f4b6cb3e29d5045ad3ea0: ending tablet copy session 16a97c1394f346f0b987f97cce7371b6-4555e51179344acda81670fc9019d968 on tablet 4555e51179344acda81670fc9019d968 with peer 16a97c1394f346f0b987f97cce7371b6
I20240502 21:29:19.439844 6293 tablet_copy_client.cc:783] T e45af71e3fc14c579b32ba4dbe2adf78 P 1fab5794cf1740288abd82f2cebb9b1a: tablet copy: Starting download of 0 data blocks...
I20240502 21:29:19.440059 6293 tablet_copy_client.cc:647] T e45af71e3fc14c579b32ba4dbe2adf78 P 1fab5794cf1740288abd82f2cebb9b1a: tablet copy: Starting download of 1 WAL segments...
I20240502 21:29:19.442649 6293 tablet_copy_client.cc:523] T e45af71e3fc14c579b32ba4dbe2adf78 P 1fab5794cf1740288abd82f2cebb9b1a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:29:19.445303 6293 tablet_bootstrap.cc:492] T e45af71e3fc14c579b32ba4dbe2adf78 P 1fab5794cf1740288abd82f2cebb9b1a: Bootstrap starting.
I20240502 21:29:19.500120 6293 tablet_bootstrap.cc:492] T e45af71e3fc14c579b32ba4dbe2adf78 P 1fab5794cf1740288abd82f2cebb9b1a: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:29:19.500514 6293 tablet_bootstrap.cc:492] T e45af71e3fc14c579b32ba4dbe2adf78 P 1fab5794cf1740288abd82f2cebb9b1a: Bootstrap complete.
I20240502 21:29:19.500743 6293 ts_tablet_manager.cc:1367] T e45af71e3fc14c579b32ba4dbe2adf78 P 1fab5794cf1740288abd82f2cebb9b1a: Time spent bootstrapping tablet: real 0.056s user 0.030s sys 0.007s
I20240502 21:29:19.501385 6293 raft_consensus.cc:348] T e45af71e3fc14c579b32ba4dbe2adf78 P 1fab5794cf1740288abd82f2cebb9b1a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { replace: true } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: NON_VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: true } }
I20240502 21:29:19.501715 6293 raft_consensus.cc:729] T e45af71e3fc14c579b32ba4dbe2adf78 P 1fab5794cf1740288abd82f2cebb9b1a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1fab5794cf1740288abd82f2cebb9b1a, State: Initialized, Role: LEARNER
I20240502 21:29:19.501964 6293 consensus_queue.cc:260] T e45af71e3fc14c579b32ba4dbe2adf78 P 1fab5794cf1740288abd82f2cebb9b1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { replace: true } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: NON_VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: true } }
I20240502 21:29:19.502686 6293 ts_tablet_manager.cc:1398] T e45af71e3fc14c579b32ba4dbe2adf78 P 1fab5794cf1740288abd82f2cebb9b1a: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 21:29:19.503391 2113 tablet_copy_service.cc:342] P 521bb93035324cecbebed2f6f2a630f8: Request end of tablet copy session 1fab5794cf1740288abd82f2cebb9b1a-e45af71e3fc14c579b32ba4dbe2adf78 received from {username='slave'} at 127.15.255.138:37811
I20240502 21:29:19.503540 2113 tablet_copy_service.cc:433] P 521bb93035324cecbebed2f6f2a630f8: ending tablet copy session 1fab5794cf1740288abd82f2cebb9b1a-e45af71e3fc14c579b32ba4dbe2adf78 on tablet e45af71e3fc14c579b32ba4dbe2adf78 with peer 1fab5794cf1740288abd82f2cebb9b1a
I20240502 21:29:19.512759 4121 raft_consensus.cc:1203] T 815178fbd07b4631b507927683a475c9 P 1fab5794cf1740288abd82f2cebb9b1a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240502 21:29:19.616046 3855 raft_consensus.cc:1203] T b300ad3948074c728a25f053512bc37c P d8d7297186694f77bb98de9174e4279c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240502 21:29:19.761615 3988 raft_consensus.cc:1203] T 4555e51179344acda81670fc9019d968 P 16a97c1394f346f0b987f97cce7371b6 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240502 21:29:19.923118 5763 raft_consensus.cc:1050] T 815178fbd07b4631b507927683a475c9 P 7d1e5c1b047443cbb28d7b0a9744babf: attempting to promote NON_VOTER 1fab5794cf1740288abd82f2cebb9b1a to VOTER
I20240502 21:29:19.923978 5763 consensus_queue.cc:237] T 815178fbd07b4631b507927683a475c9 P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } }
I20240502 21:29:19.927467 3988 raft_consensus.cc:1261] T 815178fbd07b4631b507927683a475c9 P 16a97c1394f346f0b987f97cce7371b6 [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:29:19.927479 1685 raft_consensus.cc:1261] T 815178fbd07b4631b507927683a475c9 P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:29:19.928257 6065 consensus_queue.cc:1035] T 815178fbd07b4631b507927683a475c9 P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:29:19.928750 6051 consensus_queue.cc:1035] T 815178fbd07b4631b507927683a475c9 P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:29:19.933027 4120 raft_consensus.cc:1261] T 815178fbd07b4631b507927683a475c9 P 1fab5794cf1740288abd82f2cebb9b1a [term 1 LEARNER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:29:19.933665 6051 consensus_queue.cc:1035] T 815178fbd07b4631b507927683a475c9 P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:29:19.938593 2225 raft_consensus.cc:1261] T 815178fbd07b4631b507927683a475c9 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:29:19.938750 1955 raft_consensus.cc:1261] T 815178fbd07b4631b507927683a475c9 P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:29:19.939225 6065 consensus_queue.cc:1035] T 815178fbd07b4631b507927683a475c9 P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:29:19.939793 6065 consensus_queue.cc:1035] T 815178fbd07b4631b507927683a475c9 P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:29:19.939869 1549 raft_consensus.cc:1261] T 815178fbd07b4631b507927683a475c9 P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:29:19.940771 2089 raft_consensus.cc:1261] T 815178fbd07b4631b507927683a475c9 P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:29:19.942891 6051 consensus_queue.cc:1035] T 815178fbd07b4631b507927683a475c9 P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:29:19.944098 6065 consensus_queue.cc:1035] T 815178fbd07b4631b507927683a475c9 P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:29:19.968284 6065 raft_consensus.cc:2932] T 815178fbd07b4631b507927683a475c9 P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } }
I20240502 21:29:19.974198 2089 raft_consensus.cc:2932] T 815178fbd07b4631b507927683a475c9 P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } }
I20240502 21:29:19.996217 1266 catalog_manager.cc:5484] T 815178fbd07b4631b507927683a475c9 P 521bb93035324cecbebed2f6f2a630f8 reported cstate change: config changed from index 8 to 9, 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } }
I20240502 21:29:20.001986 4120 raft_consensus.cc:1203] T e45af71e3fc14c579b32ba4dbe2adf78 P 1fab5794cf1740288abd82f2cebb9b1a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240502 21:29:20.048707 1954 raft_consensus.cc:2932] T 815178fbd07b4631b507927683a475c9 P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } }
I20240502 21:29:20.053838 1548 raft_consensus.cc:2932] T 815178fbd07b4631b507927683a475c9 P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } }
I20240502 21:29:20.060941 1685 raft_consensus.cc:2932] T 815178fbd07b4631b507927683a475c9 P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } }
I20240502 21:29:20.061667 4120 raft_consensus.cc:2932] T 815178fbd07b4631b507927683a475c9 P 1fab5794cf1740288abd82f2cebb9b1a [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } }
I20240502 21:29:20.065448 2225 raft_consensus.cc:2932] T 815178fbd07b4631b507927683a475c9 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } }
I20240502 21:29:20.068161 3988 raft_consensus.cc:2932] T 815178fbd07b4631b507927683a475c9 P 16a97c1394f346f0b987f97cce7371b6 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } }
I20240502 21:29:20.073319 1413 consensus_queue.cc:237] T 815178fbd07b4631b507927683a475c9 P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } }
I20240502 21:29:20.081296 1954 raft_consensus.cc:1261] T 815178fbd07b4631b507927683a475c9 P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 21:29:20.081923 2226 raft_consensus.cc:1261] T 815178fbd07b4631b507927683a475c9 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 21:29:20.082024 4120 raft_consensus.cc:1261] T 815178fbd07b4631b507927683a475c9 P 1fab5794cf1740288abd82f2cebb9b1a [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 21:29:20.082329 5763 consensus_queue.cc:1035] T 815178fbd07b4631b507927683a475c9 P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 21:29:20.082664 6065 consensus_queue.cc:1035] T 815178fbd07b4631b507927683a475c9 P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 21:29:20.082927 2089 raft_consensus.cc:1261] T 815178fbd07b4631b507927683a475c9 P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 21:29:20.083155 6417 consensus_queue.cc:1035] T 815178fbd07b4631b507927683a475c9 P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 21:29:20.083659 6417 consensus_queue.cc:1035] T 815178fbd07b4631b507927683a475c9 P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 21:29:20.085456 1549 raft_consensus.cc:1261] T 815178fbd07b4631b507927683a475c9 P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 21:29:20.086048 6417 consensus_queue.cc:1035] T 815178fbd07b4631b507927683a475c9 P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 21:29:20.086253 3988 raft_consensus.cc:1261] T 815178fbd07b4631b507927683a475c9 P 16a97c1394f346f0b987f97cce7371b6 [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 21:29:20.087957 6417 consensus_queue.cc:1035] T 815178fbd07b4631b507927683a475c9 P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 21:29:20.088694 5763 raft_consensus.cc:2932] T 815178fbd07b4631b507927683a475c9 P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 16b3a337bc5e462f83ba895ad8660deb (127.15.255.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } }
I20240502 21:29:20.089318 2087 raft_consensus.cc:2932] T 815178fbd07b4631b507927683a475c9 P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 16b3a337bc5e462f83ba895ad8660deb (127.15.255.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } }
I20240502 21:29:20.092391 3988 raft_consensus.cc:2932] T 815178fbd07b4631b507927683a475c9 P 16a97c1394f346f0b987f97cce7371b6 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 16b3a337bc5e462f83ba895ad8660deb (127.15.255.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } }
I20240502 21:29:20.093477 1255 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 815178fbd07b4631b507927683a475c9 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:29:20.094084 4120 raft_consensus.cc:2932] T 815178fbd07b4631b507927683a475c9 P 1fab5794cf1740288abd82f2cebb9b1a [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 16b3a337bc5e462f83ba895ad8660deb (127.15.255.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } }
I20240502 21:29:20.095160 1954 raft_consensus.cc:2932] T 815178fbd07b4631b507927683a475c9 P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 16b3a337bc5e462f83ba895ad8660deb (127.15.255.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } }
I20240502 21:29:20.100713 1264 catalog_manager.cc:5484] T 815178fbd07b4631b507927683a475c9 P 1fab5794cf1740288abd82f2cebb9b1a reported cstate change: config changed from index 9 to 10, VOTER 16b3a337bc5e462f83ba895ad8660deb (127.15.255.131) evicted. New cstate: current_term: 1 leader_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } }
I20240502 21:29:20.103857 2226 raft_consensus.cc:2932] T 815178fbd07b4631b507927683a475c9 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 16b3a337bc5e462f83ba895ad8660deb (127.15.255.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } }
I20240502 21:29:20.105441 1664 tablet_service.cc:1514] Processing DeleteTablet for tablet 815178fbd07b4631b507927683a475c9 with delete_type TABLET_DATA_TOMBSTONED (TS 16b3a337bc5e462f83ba895ad8660deb not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:42590
I20240502 21:29:20.106571 1548 raft_consensus.cc:2932] T 815178fbd07b4631b507927683a475c9 P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 16b3a337bc5e462f83ba895ad8660deb (127.15.255.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } }
I20240502 21:29:20.116323 6437 tablet_replica.cc:330] T 815178fbd07b4631b507927683a475c9 P 16b3a337bc5e462f83ba895ad8660deb: stopping tablet replica
I20240502 21:29:20.117019 6437 raft_consensus.cc:2229] T 815178fbd07b4631b507927683a475c9 P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:29:20.117450 6437 raft_consensus.cc:2258] T 815178fbd07b4631b507927683a475c9 P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:29:20.119729 6437 ts_tablet_manager.cc:1875] T 815178fbd07b4631b507927683a475c9 P 16b3a337bc5e462f83ba895ad8660deb: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:29:20.129839 6437 ts_tablet_manager.cc:1888] T 815178fbd07b4631b507927683a475c9 P 16b3a337bc5e462f83ba895ad8660deb: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240502 21:29:20.130019 6437 log.cc:1196] T 815178fbd07b4631b507927683a475c9 P 16b3a337bc5e462f83ba895ad8660deb: Deleting WAL directory at /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-2/wal/wals/815178fbd07b4631b507927683a475c9
I20240502 21:29:20.130868 1257 catalog_manager.cc:4830] TS 16b3a337bc5e462f83ba895ad8660deb (127.15.255.131:36033): tablet 815178fbd07b4631b507927683a475c9 (table rebalance_test_table_3 [id=a205513324dd4db5a5ee4ed1928aefa6]) successfully deleted
I20240502 21:29:20.204324 6146 raft_consensus.cc:1050] T b300ad3948074c728a25f053512bc37c P 521bb93035324cecbebed2f6f2a630f8: attempting to promote NON_VOTER d8d7297186694f77bb98de9174e4279c to VOTER
I20240502 21:29:20.205189 6146 consensus_queue.cc:237] T b300ad3948074c728a25f053512bc37c P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { replace: true } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } }
I20240502 21:29:20.208847 3855 raft_consensus.cc:1261] T b300ad3948074c728a25f053512bc37c P d8d7297186694f77bb98de9174e4279c [term 1 LEARNER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:29:20.208863 2226 raft_consensus.cc:1261] T b300ad3948074c728a25f053512bc37c P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:29:20.209622 1548 raft_consensus.cc:1261] T b300ad3948074c728a25f053512bc37c P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:29:20.209738 1685 raft_consensus.cc:1261] T b300ad3948074c728a25f053512bc37c P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:29:20.209761 6470 consensus_queue.cc:1035] T b300ad3948074c728a25f053512bc37c P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:29:20.210209 1413 raft_consensus.cc:1261] T b300ad3948074c728a25f053512bc37c P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:29:20.210500 6469 consensus_queue.cc:1035] T b300ad3948074c728a25f053512bc37c P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:29:20.210795 6146 consensus_queue.cc:1035] T b300ad3948074c728a25f053512bc37c P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:29:20.211315 6395 consensus_queue.cc:1035] T b300ad3948074c728a25f053512bc37c P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:29:20.211611 1954 raft_consensus.cc:1261] T b300ad3948074c728a25f053512bc37c P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:29:20.212168 3988 raft_consensus.cc:1261] T b300ad3948074c728a25f053512bc37c P 16a97c1394f346f0b987f97cce7371b6 [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:29:20.212133 6395 consensus_queue.cc:1035] T b300ad3948074c728a25f053512bc37c P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:29:20.213470 6146 consensus_queue.cc:1035] T b300ad3948074c728a25f053512bc37c P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:29:20.214072 6470 consensus_queue.cc:1035] T b300ad3948074c728a25f053512bc37c P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:29:20.218374 6469 raft_consensus.cc:2932] T b300ad3948074c728a25f053512bc37c P 521bb93035324cecbebed2f6f2a630f8 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, d8d7297186694f77bb98de9174e4279c (127.15.255.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { replace: true } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } }
I20240502 21:29:20.218806 1413 raft_consensus.cc:2932] T b300ad3948074c728a25f053512bc37c P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, d8d7297186694f77bb98de9174e4279c (127.15.255.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { replace: true } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } }
I20240502 21:29:20.221701 3988 raft_consensus.cc:2932] T b300ad3948074c728a25f053512bc37c P 16a97c1394f346f0b987f97cce7371b6 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, d8d7297186694f77bb98de9174e4279c (127.15.255.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { replace: true } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } }
I20240502 21:29:20.223680 3855 raft_consensus.cc:2932] T b300ad3948074c728a25f053512bc37c P d8d7297186694f77bb98de9174e4279c [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, d8d7297186694f77bb98de9174e4279c (127.15.255.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { replace: true } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } }
I20240502 21:29:20.223948 1954 raft_consensus.cc:2932] T b300ad3948074c728a25f053512bc37c P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, d8d7297186694f77bb98de9174e4279c (127.15.255.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { replace: true } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } }
I20240502 21:29:20.229593 1268 catalog_manager.cc:5484] T b300ad3948074c728a25f053512bc37c P e632c553ff4341328acf1cbaca678392 reported cstate change: config changed from index 8 to 9, d8d7297186694f77bb98de9174e4279c (127.15.255.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "521bb93035324cecbebed2f6f2a630f8" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { replace: true } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } }
I20240502 21:29:20.233669 1548 raft_consensus.cc:2932] T b300ad3948074c728a25f053512bc37c P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, d8d7297186694f77bb98de9174e4279c (127.15.255.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { replace: true } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } }
I20240502 21:29:20.249094 1685 raft_consensus.cc:2932] T b300ad3948074c728a25f053512bc37c P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, d8d7297186694f77bb98de9174e4279c (127.15.255.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { replace: true } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } }
I20240502 21:29:20.258581 2226 raft_consensus.cc:2932] T b300ad3948074c728a25f053512bc37c P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, d8d7297186694f77bb98de9174e4279c (127.15.255.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { replace: true } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } }
I20240502 21:29:20.261097 2087 consensus_queue.cc:237] T b300ad3948074c728a25f053512bc37c P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } }
I20240502 21:29:20.264257 2226 raft_consensus.cc:1261] T b300ad3948074c728a25f053512bc37c P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 21:29:20.264698 1414 raft_consensus.cc:1261] T b300ad3948074c728a25f053512bc37c P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 21:29:20.265318 6470 consensus_queue.cc:1035] T b300ad3948074c728a25f053512bc37c P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 21:29:20.265578 3986 raft_consensus.cc:1261] T b300ad3948074c728a25f053512bc37c P 16a97c1394f346f0b987f97cce7371b6 [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 21:29:20.265630 3855 raft_consensus.cc:1261] T b300ad3948074c728a25f053512bc37c P d8d7297186694f77bb98de9174e4279c [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 21:29:20.266170 1954 raft_consensus.cc:1261] T b300ad3948074c728a25f053512bc37c P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 21:29:20.266108 6470 consensus_queue.cc:1035] T b300ad3948074c728a25f053512bc37c P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 21:29:20.266327 6146 consensus_queue.cc:1035] T b300ad3948074c728a25f053512bc37c P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 21:29:20.266709 6146 consensus_queue.cc:1035] T b300ad3948074c728a25f053512bc37c P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 21:29:20.267563 1685 raft_consensus.cc:1261] T b300ad3948074c728a25f053512bc37c P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 21:29:20.268337 6479 consensus_queue.cc:1035] T b300ad3948074c728a25f053512bc37c P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 21:29:20.268723 6146 consensus_queue.cc:1035] T b300ad3948074c728a25f053512bc37c P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 21:29:20.270720 6116 raft_consensus.cc:2932] T b300ad3948074c728a25f053512bc37c P 521bb93035324cecbebed2f6f2a630f8 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER a62a935ff43d494ca35d0c417943133d (127.15.255.130) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } }
I20240502 21:29:20.271550 2226 raft_consensus.cc:2932] T b300ad3948074c728a25f053512bc37c P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER a62a935ff43d494ca35d0c417943133d (127.15.255.130) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } }
I20240502 21:29:20.272622 1413 raft_consensus.cc:2932] T b300ad3948074c728a25f053512bc37c P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER a62a935ff43d494ca35d0c417943133d (127.15.255.130) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } }
I20240502 21:29:20.274571 3986 raft_consensus.cc:2932] T b300ad3948074c728a25f053512bc37c P 16a97c1394f346f0b987f97cce7371b6 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER a62a935ff43d494ca35d0c417943133d (127.15.255.130) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } }
I20240502 21:29:20.277192 1262 catalog_manager.cc:5484] T b300ad3948074c728a25f053512bc37c P 6ab8df3c796f4b6cb3e29d5045ad3ea0 reported cstate change: config changed from index 9 to 10, VOTER a62a935ff43d494ca35d0c417943133d (127.15.255.130) evicted. New cstate: current_term: 1 leader_uuid: "521bb93035324cecbebed2f6f2a630f8" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } }
I20240502 21:29:20.280186 1254 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet b300ad3948074c728a25f053512bc37c with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:29:20.280586 3855 raft_consensus.cc:2932] T b300ad3948074c728a25f053512bc37c P d8d7297186694f77bb98de9174e4279c [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER a62a935ff43d494ca35d0c417943133d (127.15.255.130) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } }
I20240502 21:29:20.284385 1955 raft_consensus.cc:2932] T b300ad3948074c728a25f053512bc37c P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER a62a935ff43d494ca35d0c417943133d (127.15.255.130) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } }
I20240502 21:29:20.284930 1527 tablet_service.cc:1514] Processing DeleteTablet for tablet b300ad3948074c728a25f053512bc37c with delete_type TABLET_DATA_TOMBSTONED (TS a62a935ff43d494ca35d0c417943133d not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:43860
I20240502 21:29:20.285714 1685 raft_consensus.cc:2932] T b300ad3948074c728a25f053512bc37c P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER a62a935ff43d494ca35d0c417943133d (127.15.255.130) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } }
I20240502 21:29:20.291858 6480 tablet_replica.cc:330] T b300ad3948074c728a25f053512bc37c P a62a935ff43d494ca35d0c417943133d: stopping tablet replica
I20240502 21:29:20.292280 6480 raft_consensus.cc:2229] T b300ad3948074c728a25f053512bc37c P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:29:20.292582 6480 raft_consensus.cc:2258] T b300ad3948074c728a25f053512bc37c P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:29:20.293673 6480 ts_tablet_manager.cc:1875] T b300ad3948074c728a25f053512bc37c P a62a935ff43d494ca35d0c417943133d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:29:20.298300 6480 ts_tablet_manager.cc:1888] T b300ad3948074c728a25f053512bc37c P a62a935ff43d494ca35d0c417943133d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240502 21:29:20.298419 6480 log.cc:1196] T b300ad3948074c728a25f053512bc37c P a62a935ff43d494ca35d0c417943133d: Deleting WAL directory at /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-1/wal/wals/b300ad3948074c728a25f053512bc37c
I20240502 21:29:20.299103 1255 catalog_manager.cc:4830] TS a62a935ff43d494ca35d0c417943133d (127.15.255.130:37991): tablet b300ad3948074c728a25f053512bc37c (table rebalance_test_table_2 [id=900316782c71465f9538d1bbe31708d3]) successfully deleted
I20240502 21:29:20.324867 6017 raft_consensus.cc:1050] T 4555e51179344acda81670fc9019d968 P 6ab8df3c796f4b6cb3e29d5045ad3ea0: attempting to promote NON_VOTER 16a97c1394f346f0b987f97cce7371b6 to VOTER
I20240502 21:29:20.325714 6017 consensus_queue.cc:237] T 4555e51179344acda81670fc9019d968 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } }
I20240502 21:29:20.328612 3986 raft_consensus.cc:1261] T 4555e51179344acda81670fc9019d968 P 16a97c1394f346f0b987f97cce7371b6 [term 1 LEARNER]: Refusing update from remote peer 6ab8df3c796f4b6cb3e29d5045ad3ea0: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:29:20.329336 1685 raft_consensus.cc:1261] T 4555e51179344acda81670fc9019d968 P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Refusing update from remote peer 6ab8df3c796f4b6cb3e29d5045ad3ea0: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:29:20.329438 4120 raft_consensus.cc:1261] T 4555e51179344acda81670fc9019d968 P 1fab5794cf1740288abd82f2cebb9b1a [term 1 FOLLOWER]: Refusing update from remote peer 6ab8df3c796f4b6cb3e29d5045ad3ea0: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:29:20.330185 1548 raft_consensus.cc:1261] T 4555e51179344acda81670fc9019d968 P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Refusing update from remote peer 6ab8df3c796f4b6cb3e29d5045ad3ea0: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:29:20.330242 1413 raft_consensus.cc:1261] T 4555e51179344acda81670fc9019d968 P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Refusing update from remote peer 6ab8df3c796f4b6cb3e29d5045ad3ea0: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:29:20.330291 1955 raft_consensus.cc:1261] T 4555e51179344acda81670fc9019d968 P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Refusing update from remote peer 6ab8df3c796f4b6cb3e29d5045ad3ea0: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:29:20.330322 2087 raft_consensus.cc:1261] T 4555e51179344acda81670fc9019d968 P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Refusing update from remote peer 6ab8df3c796f4b6cb3e29d5045ad3ea0: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:29:20.331061 5627 consensus_queue.cc:1035] T 4555e51179344acda81670fc9019d968 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:29:20.331604 5627 consensus_queue.cc:1035] T 4555e51179344acda81670fc9019d968 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:29:20.332201 6017 consensus_queue.cc:1035] T 4555e51179344acda81670fc9019d968 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:29:20.332820 5627 consensus_queue.cc:1035] T 4555e51179344acda81670fc9019d968 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:29:20.336308 6482 consensus_queue.cc:1035] T 4555e51179344acda81670fc9019d968 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:29:20.349174 5627 consensus_queue.cc:1035] T 4555e51179344acda81670fc9019d968 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:29:20.351131 5627 consensus_queue.cc:1035] T 4555e51179344acda81670fc9019d968 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:29:20.366917 5627 raft_consensus.cc:2932] T 4555e51179344acda81670fc9019d968 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 16a97c1394f346f0b987f97cce7371b6 (127.15.255.137) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } }
I20240502 21:29:20.368455 1414 raft_consensus.cc:2932] T 4555e51179344acda81670fc9019d968 P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 16a97c1394f346f0b987f97cce7371b6 (127.15.255.137) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } }
I20240502 21:29:20.372400 1549 raft_consensus.cc:2932] T 4555e51179344acda81670fc9019d968 P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 16a97c1394f346f0b987f97cce7371b6 (127.15.255.137) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } }
I20240502 21:29:20.372604 4120 raft_consensus.cc:2932] T 4555e51179344acda81670fc9019d968 P 1fab5794cf1740288abd82f2cebb9b1a [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 16a97c1394f346f0b987f97cce7371b6 (127.15.255.137) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } }
I20240502 21:29:20.376233 1684 raft_consensus.cc:2932] T 4555e51179344acda81670fc9019d968 P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 16a97c1394f346f0b987f97cce7371b6 (127.15.255.137) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } }
I20240502 21:29:20.379024 1262 catalog_manager.cc:5484] T 4555e51179344acda81670fc9019d968 P 1fab5794cf1740288abd82f2cebb9b1a reported cstate change: config changed from index 8 to 9, 16a97c1394f346f0b987f97cce7371b6 (127.15.255.137) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } }
I20240502 21:29:20.381326 2087 raft_consensus.cc:2932] T 4555e51179344acda81670fc9019d968 P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 16a97c1394f346f0b987f97cce7371b6 (127.15.255.137) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } }
I20240502 21:29:20.384019 3986 raft_consensus.cc:2932] T 4555e51179344acda81670fc9019d968 P 16a97c1394f346f0b987f97cce7371b6 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 16a97c1394f346f0b987f97cce7371b6 (127.15.255.137) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } }
I20240502 21:29:20.397944 1955 raft_consensus.cc:2932] T 4555e51179344acda81670fc9019d968 P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 16a97c1394f346f0b987f97cce7371b6 (127.15.255.137) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } }
I20240502 21:29:20.400295 2225 consensus_queue.cc:237] T 4555e51179344acda81670fc9019d968 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } }
I20240502 21:29:20.403417 2087 raft_consensus.cc:1261] T 4555e51179344acda81670fc9019d968 P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Refusing update from remote peer 6ab8df3c796f4b6cb3e29d5045ad3ea0: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 21:29:20.403491 1955 raft_consensus.cc:1261] T 4555e51179344acda81670fc9019d968 P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Refusing update from remote peer 6ab8df3c796f4b6cb3e29d5045ad3ea0: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 21:29:20.403764 4120 raft_consensus.cc:1261] T 4555e51179344acda81670fc9019d968 P 1fab5794cf1740288abd82f2cebb9b1a [term 1 FOLLOWER]: Refusing update from remote peer 6ab8df3c796f4b6cb3e29d5045ad3ea0: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 21:29:20.404340 6486 consensus_queue.cc:1035] T 4555e51179344acda81670fc9019d968 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 21:29:20.404865 6486 consensus_queue.cc:1035] T 4555e51179344acda81670fc9019d968 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 21:29:20.405781 6483 consensus_queue.cc:1035] T 4555e51179344acda81670fc9019d968 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 21:29:20.407188 3986 raft_consensus.cc:1261] T 4555e51179344acda81670fc9019d968 P 16a97c1394f346f0b987f97cce7371b6 [term 1 FOLLOWER]: Refusing update from remote peer 6ab8df3c796f4b6cb3e29d5045ad3ea0: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 21:29:20.407480 1549 raft_consensus.cc:1261] T 4555e51179344acda81670fc9019d968 P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Refusing update from remote peer 6ab8df3c796f4b6cb3e29d5045ad3ea0: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 21:29:20.409056 1413 raft_consensus.cc:1261] T 4555e51179344acda81670fc9019d968 P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Refusing update from remote peer 6ab8df3c796f4b6cb3e29d5045ad3ea0: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 21:29:20.409212 6483 consensus_queue.cc:1035] T 4555e51179344acda81670fc9019d968 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 21:29:20.409749 6483 consensus_queue.cc:1035] T 4555e51179344acda81670fc9019d968 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 21:29:20.410364 6486 consensus_queue.cc:1035] T 4555e51179344acda81670fc9019d968 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 21:29:20.412868 6484 raft_consensus.cc:2932] T 4555e51179344acda81670fc9019d968 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 16b3a337bc5e462f83ba895ad8660deb (127.15.255.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } }
I20240502 21:29:20.413520 1414 raft_consensus.cc:2932] T 4555e51179344acda81670fc9019d968 P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 16b3a337bc5e462f83ba895ad8660deb (127.15.255.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } }
I20240502 21:29:20.413481 3986 raft_consensus.cc:2932] T 4555e51179344acda81670fc9019d968 P 16a97c1394f346f0b987f97cce7371b6 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 16b3a337bc5e462f83ba895ad8660deb (127.15.255.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } }
I20240502 21:29:20.418968 1266 catalog_manager.cc:5484] T 4555e51179344acda81670fc9019d968 P 7d1e5c1b047443cbb28d7b0a9744babf reported cstate change: config changed from index 9 to 10, VOTER 16b3a337bc5e462f83ba895ad8660deb (127.15.255.131) evicted. New cstate: current_term: 1 leader_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } }
I20240502 21:29:20.419756 1255 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 4555e51179344acda81670fc9019d968 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:29:20.422041 2089 raft_consensus.cc:2932] T 4555e51179344acda81670fc9019d968 P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 16b3a337bc5e462f83ba895ad8660deb (127.15.255.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } }
I20240502 21:29:20.422374 4120 raft_consensus.cc:2932] T 4555e51179344acda81670fc9019d968 P 1fab5794cf1740288abd82f2cebb9b1a [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 16b3a337bc5e462f83ba895ad8660deb (127.15.255.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } }
I20240502 21:29:20.425103 1664 tablet_service.cc:1514] Processing DeleteTablet for tablet 4555e51179344acda81670fc9019d968 with delete_type TABLET_DATA_TOMBSTONED (TS 16b3a337bc5e462f83ba895ad8660deb not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:42590
I20240502 21:29:20.425526 6437 tablet_replica.cc:330] T 4555e51179344acda81670fc9019d968 P 16b3a337bc5e462f83ba895ad8660deb: stopping tablet replica
I20240502 21:29:20.425731 6437 raft_consensus.cc:2229] T 4555e51179344acda81670fc9019d968 P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:29:20.426043 6437 raft_consensus.cc:2258] T 4555e51179344acda81670fc9019d968 P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:29:20.427120 6437 ts_tablet_manager.cc:1875] T 4555e51179344acda81670fc9019d968 P 16b3a337bc5e462f83ba895ad8660deb: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:29:20.431713 1549 raft_consensus.cc:2932] T 4555e51179344acda81670fc9019d968 P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 16b3a337bc5e462f83ba895ad8660deb (127.15.255.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } }
I20240502 21:29:20.434881 6437 ts_tablet_manager.cc:1888] T 4555e51179344acda81670fc9019d968 P 16b3a337bc5e462f83ba895ad8660deb: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240502 21:29:20.435272 1955 raft_consensus.cc:2932] T 4555e51179344acda81670fc9019d968 P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 16b3a337bc5e462f83ba895ad8660deb (127.15.255.131) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } }
I20240502 21:29:20.437775 6437 log.cc:1196] T 4555e51179344acda81670fc9019d968 P 16b3a337bc5e462f83ba895ad8660deb: Deleting WAL directory at /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-2/wal/wals/4555e51179344acda81670fc9019d968
I20240502 21:29:20.438532 1257 catalog_manager.cc:4830] TS 16b3a337bc5e462f83ba895ad8660deb (127.15.255.131:36033): tablet 4555e51179344acda81670fc9019d968 (table rebalance_test_table_6 [id=552d53f5242c4a2eb8e89b63e8eb6a7a]) successfully deleted
I20240502 21:29:20.583979 6116 raft_consensus.cc:1050] T e45af71e3fc14c579b32ba4dbe2adf78 P 521bb93035324cecbebed2f6f2a630f8: attempting to promote NON_VOTER 1fab5794cf1740288abd82f2cebb9b1a to VOTER
I20240502 21:29:20.585211 6116 consensus_queue.cc:237] T e45af71e3fc14c579b32ba4dbe2adf78 P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { replace: true } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } }
I20240502 21:29:20.588554 3986 raft_consensus.cc:1261] T e45af71e3fc14c579b32ba4dbe2adf78 P 16a97c1394f346f0b987f97cce7371b6 [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:29:20.589093 6469 consensus_queue.cc:1035] T e45af71e3fc14c579b32ba4dbe2adf78 P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:29:20.590662 1684 raft_consensus.cc:1261] T e45af71e3fc14c579b32ba4dbe2adf78 P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:29:20.591210 1413 raft_consensus.cc:1261] T e45af71e3fc14c579b32ba4dbe2adf78 P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:29:20.591250 6146 consensus_queue.cc:1035] T e45af71e3fc14c579b32ba4dbe2adf78 P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:29:20.591900 6146 consensus_queue.cc:1035] T e45af71e3fc14c579b32ba4dbe2adf78 P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:29:20.594544 1549 raft_consensus.cc:1261] T e45af71e3fc14c579b32ba4dbe2adf78 P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:29:20.596400 6146 consensus_queue.cc:1035] T e45af71e3fc14c579b32ba4dbe2adf78 P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:29:20.597554 1820 raft_consensus.cc:1261] T e45af71e3fc14c579b32ba4dbe2adf78 P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:29:20.597604 4120 raft_consensus.cc:1261] T e45af71e3fc14c579b32ba4dbe2adf78 P 1fab5794cf1740288abd82f2cebb9b1a [term 1 LEARNER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:29:20.598047 2225 raft_consensus.cc:1261] T e45af71e3fc14c579b32ba4dbe2adf78 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:29:20.598160 6470 consensus_queue.cc:1035] T e45af71e3fc14c579b32ba4dbe2adf78 P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:29:20.598385 6479 consensus_queue.cc:1035] T e45af71e3fc14c579b32ba4dbe2adf78 P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:29:20.600783 6146 consensus_queue.cc:1035] T e45af71e3fc14c579b32ba4dbe2adf78 P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:29:20.606189 1684 raft_consensus.cc:2932] T e45af71e3fc14c579b32ba4dbe2adf78 P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { replace: true } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } }
I20240502 21:29:20.606169 1549 raft_consensus.cc:2932] T e45af71e3fc14c579b32ba4dbe2adf78 P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { replace: true } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } }
I20240502 21:29:20.608858 6116 raft_consensus.cc:2932] T e45af71e3fc14c579b32ba4dbe2adf78 P 521bb93035324cecbebed2f6f2a630f8 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { replace: true } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } }
I20240502 21:29:20.611969 2225 raft_consensus.cc:2932] T e45af71e3fc14c579b32ba4dbe2adf78 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { replace: true } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } }
I20240502 21:29:20.612371 1262 catalog_manager.cc:5484] T e45af71e3fc14c579b32ba4dbe2adf78 P 16b3a337bc5e462f83ba895ad8660deb reported cstate change: config changed from index 8 to 9, 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "521bb93035324cecbebed2f6f2a630f8" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { replace: true } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } }
I20240502 21:29:20.619045 4120 raft_consensus.cc:2932] T e45af71e3fc14c579b32ba4dbe2adf78 P 1fab5794cf1740288abd82f2cebb9b1a [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { replace: true } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } }
I20240502 21:29:20.621058 3986 raft_consensus.cc:2932] T e45af71e3fc14c579b32ba4dbe2adf78 P 16a97c1394f346f0b987f97cce7371b6 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { replace: true } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } }
I20240502 21:29:20.623219 1413 raft_consensus.cc:2932] T e45af71e3fc14c579b32ba4dbe2adf78 P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { replace: true } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } }
I20240502 21:29:20.623191 1820 raft_consensus.cc:2932] T e45af71e3fc14c579b32ba4dbe2adf78 P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { replace: true } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } }
I20240502 21:29:20.632992 2089 consensus_queue.cc:237] T e45af71e3fc14c579b32ba4dbe2adf78 P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } }
I20240502 21:29:20.635223 1684 raft_consensus.cc:1261] T e45af71e3fc14c579b32ba4dbe2adf78 P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 21:29:20.635619 1413 raft_consensus.cc:1261] T e45af71e3fc14c579b32ba4dbe2adf78 P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 21:29:20.635798 6395 consensus_queue.cc:1035] T e45af71e3fc14c579b32ba4dbe2adf78 P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 21:29:20.636111 1820 raft_consensus.cc:1261] T e45af71e3fc14c579b32ba4dbe2adf78 P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 21:29:20.636716 6469 consensus_queue.cc:1035] T e45af71e3fc14c579b32ba4dbe2adf78 P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 21:29:20.637257 6469 consensus_queue.cc:1035] T e45af71e3fc14c579b32ba4dbe2adf78 P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 21:29:20.638010 3986 raft_consensus.cc:1261] T e45af71e3fc14c579b32ba4dbe2adf78 P 16a97c1394f346f0b987f97cce7371b6 [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 21:29:20.638532 2225 raft_consensus.cc:1261] T e45af71e3fc14c579b32ba4dbe2adf78 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 21:29:20.639086 6469 consensus_queue.cc:1035] T e45af71e3fc14c579b32ba4dbe2adf78 P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 21:29:20.639695 6470 consensus_queue.cc:1035] T e45af71e3fc14c579b32ba4dbe2adf78 P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 21:29:20.640779 6146 raft_consensus.cc:2932] T e45af71e3fc14c579b32ba4dbe2adf78 P 521bb93035324cecbebed2f6f2a630f8 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER a62a935ff43d494ca35d0c417943133d (127.15.255.130) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } }
I20240502 21:29:20.640849 1817 raft_consensus.cc:2932] T e45af71e3fc14c579b32ba4dbe2adf78 P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER a62a935ff43d494ca35d0c417943133d (127.15.255.130) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } }
I20240502 21:29:20.641949 2225 raft_consensus.cc:2932] T e45af71e3fc14c579b32ba4dbe2adf78 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER a62a935ff43d494ca35d0c417943133d (127.15.255.130) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } }
I20240502 21:29:20.644462 1254 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet e45af71e3fc14c579b32ba4dbe2adf78 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:29:20.645669 1684 raft_consensus.cc:2932] T e45af71e3fc14c579b32ba4dbe2adf78 P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER a62a935ff43d494ca35d0c417943133d (127.15.255.130) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } }
I20240502 21:29:20.646948 1413 raft_consensus.cc:2932] T e45af71e3fc14c579b32ba4dbe2adf78 P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER a62a935ff43d494ca35d0c417943133d (127.15.255.130) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } }
I20240502 21:29:20.647495 1262 catalog_manager.cc:5484] T e45af71e3fc14c579b32ba4dbe2adf78 P 521bb93035324cecbebed2f6f2a630f8 reported cstate change: config changed from index 9 to 10, VOTER a62a935ff43d494ca35d0c417943133d (127.15.255.130) evicted. New cstate: current_term: 1 leader_uuid: "521bb93035324cecbebed2f6f2a630f8" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 21:29:20.651113 3986 raft_consensus.cc:2932] T e45af71e3fc14c579b32ba4dbe2adf78 P 16a97c1394f346f0b987f97cce7371b6 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER a62a935ff43d494ca35d0c417943133d (127.15.255.130) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } }
I20240502 21:29:20.652622 1527 tablet_service.cc:1514] Processing DeleteTablet for tablet e45af71e3fc14c579b32ba4dbe2adf78 with delete_type TABLET_DATA_TOMBSTONED (TS a62a935ff43d494ca35d0c417943133d not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:43860
I20240502 21:29:20.653074 6480 tablet_replica.cc:330] T e45af71e3fc14c579b32ba4dbe2adf78 P a62a935ff43d494ca35d0c417943133d: stopping tablet replica
I20240502 21:29:20.653311 6480 raft_consensus.cc:2229] T e45af71e3fc14c579b32ba4dbe2adf78 P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:29:20.653440 4120 raft_consensus.cc:1261] T e45af71e3fc14c579b32ba4dbe2adf78 P 1fab5794cf1740288abd82f2cebb9b1a [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 21:29:20.653574 6480 raft_consensus.cc:2258] T e45af71e3fc14c579b32ba4dbe2adf78 P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:29:20.656008 6116 consensus_queue.cc:1035] T e45af71e3fc14c579b32ba4dbe2adf78 P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 21:29:20.657253 6480 ts_tablet_manager.cc:1875] T e45af71e3fc14c579b32ba4dbe2adf78 P a62a935ff43d494ca35d0c417943133d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:29:20.657564 4121 raft_consensus.cc:2932] T e45af71e3fc14c579b32ba4dbe2adf78 P 1fab5794cf1740288abd82f2cebb9b1a [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER a62a935ff43d494ca35d0c417943133d (127.15.255.130) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } }
I20240502 21:29:20.662885 6480 ts_tablet_manager.cc:1888] T e45af71e3fc14c579b32ba4dbe2adf78 P a62a935ff43d494ca35d0c417943133d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240502 21:29:20.663000 6480 log.cc:1196] T e45af71e3fc14c579b32ba4dbe2adf78 P a62a935ff43d494ca35d0c417943133d: Deleting WAL directory at /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-1/wal/wals/e45af71e3fc14c579b32ba4dbe2adf78
I20240502 21:29:20.663683 1255 catalog_manager.cc:4830] TS a62a935ff43d494ca35d0c417943133d (127.15.255.130:37991): tablet e45af71e3fc14c579b32ba4dbe2adf78 (table rebalance_test_table_5 [id=9cb803e5bb2545c8b992c90a894085af]) successfully deleted
I20240502 21:29:20.812054 1798 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:29:20.817376 2068 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 21:29:20.828557 2205 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:29:20.836577 3968 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:20.837991 1664 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:20.883988 1394 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:29:20.901968 4101 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:20.936695 3836 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:20.948797 1527 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:29:20.997241 1931 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:29:21.233685 1798 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:29:21.274138 4101 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:21.286873 3968 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:21.298840 1664 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:21.309407 2205 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:29:21.312144 2068 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 21:29:21.365633 3836 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:21.370719 1931 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:29:21.374356 1394 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:29:21.397166 1527 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:29:21.521274 2225 consensus_queue.cc:237] T 05e885ae741c43808c6dfbcdc5769942 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: NON_VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { promote: true } }
W20240502 21:29:21.525208 2161 consensus_peers.cc:485] T 05e885ae741c43808c6dfbcdc5769942 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 -> Peer 16b3a337bc5e462f83ba895ad8660deb (127.15.255.131:36033): Couldn't send request to peer 16b3a337bc5e462f83ba895ad8660deb. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:29:21.526014 3986 raft_consensus.cc:1261] T 05e885ae741c43808c6dfbcdc5769942 P 16a97c1394f346f0b987f97cce7371b6 [term 1 FOLLOWER]: Refusing update from remote peer 6ab8df3c796f4b6cb3e29d5045ad3ea0: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 21:29:21.526290 4120 raft_consensus.cc:1261] T 05e885ae741c43808c6dfbcdc5769942 P 1fab5794cf1740288abd82f2cebb9b1a [term 1 FOLLOWER]: Refusing update from remote peer 6ab8df3c796f4b6cb3e29d5045ad3ea0: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 21:29:21.526679 6574 consensus_queue.cc:1035] T 05e885ae741c43808c6dfbcdc5769942 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 21:29:21.527390 6574 consensus_queue.cc:1035] T 05e885ae741c43808c6dfbcdc5769942 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 21:29:21.529577 2089 raft_consensus.cc:1261] T 05e885ae741c43808c6dfbcdc5769942 P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Refusing update from remote peer 6ab8df3c796f4b6cb3e29d5045ad3ea0: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 21:29:21.529590 1954 raft_consensus.cc:1261] T 05e885ae741c43808c6dfbcdc5769942 P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Refusing update from remote peer 6ab8df3c796f4b6cb3e29d5045ad3ea0: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 21:29:21.530210 6574 consensus_queue.cc:1035] T 05e885ae741c43808c6dfbcdc5769942 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 21:29:21.530736 6574 consensus_queue.cc:1035] T 05e885ae741c43808c6dfbcdc5769942 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 21:29:21.566874 2087 consensus_queue.cc:237] T c46e36b0bceb45b4bafbf1ffaf8e039e P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false replace: true } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: true } }
I20240502 21:29:21.589568 1413 raft_consensus.cc:1261] T 05e885ae741c43808c6dfbcdc5769942 P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Refusing update from remote peer 6ab8df3c796f4b6cb3e29d5045ad3ea0: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 21:29:21.590131 1548 raft_consensus.cc:1261] T 05e885ae741c43808c6dfbcdc5769942 P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Refusing update from remote peer 6ab8df3c796f4b6cb3e29d5045ad3ea0: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 21:29:21.591449 1684 raft_consensus.cc:1261] T c46e36b0bceb45b4bafbf1ffaf8e039e P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
W20240502 21:29:21.591496 2024 consensus_peers.cc:485] T c46e36b0bceb45b4bafbf1ffaf8e039e P 521bb93035324cecbebed2f6f2a630f8 -> Peer d8d7297186694f77bb98de9174e4279c (127.15.255.136:41459): Couldn't send request to peer d8d7297186694f77bb98de9174e4279c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c46e36b0bceb45b4bafbf1ffaf8e039e. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:29:21.591271 6574 raft_consensus.cc:2932] T 05e885ae741c43808c6dfbcdc5769942 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 16b3a337bc5e462f83ba895ad8660deb (127.15.255.131) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: NON_VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { promote: true } } }
I20240502 21:29:21.593619 6479 consensus_queue.cc:1035] T c46e36b0bceb45b4bafbf1ffaf8e039e P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:29:21.593674 2088 raft_consensus.cc:2932] T 05e885ae741c43808c6dfbcdc5769942 P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 16b3a337bc5e462f83ba895ad8660deb (127.15.255.131) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: NON_VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { promote: true } } }
I20240502 21:29:21.599552 1413 raft_consensus.cc:1261] T c46e36b0bceb45b4bafbf1ffaf8e039e P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:29:21.600147 6395 consensus_queue.cc:1035] T c46e36b0bceb45b4bafbf1ffaf8e039e P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:29:21.600337 1264 catalog_manager.cc:5484] T 05e885ae741c43808c6dfbcdc5769942 P 521bb93035324cecbebed2f6f2a630f8 reported cstate change: config changed from index 10 to 11, NON_VOTER 16b3a337bc5e462f83ba895ad8660deb (127.15.255.131) added. New cstate: current_term: 1 leader_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: NON_VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { promote: true } } }
I20240502 21:29:21.618218 4121 raft_consensus.cc:1261] T c46e36b0bceb45b4bafbf1ffaf8e039e P 1fab5794cf1740288abd82f2cebb9b1a [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:29:21.620457 6395 consensus_queue.cc:1035] T c46e36b0bceb45b4bafbf1ffaf8e039e P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:29:21.625450 1548 raft_consensus.cc:1261] T c46e36b0bceb45b4bafbf1ffaf8e039e P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:29:21.626106 6395 consensus_queue.cc:1035] T c46e36b0bceb45b4bafbf1ffaf8e039e P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:29:21.629959 6660 consensus_queue.cc:1035] T 05e885ae741c43808c6dfbcdc5769942 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 21:29:21.632267 6660 consensus_queue.cc:1035] T 05e885ae741c43808c6dfbcdc5769942 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 21:29:21.636112 1954 raft_consensus.cc:1261] T c46e36b0bceb45b4bafbf1ffaf8e039e P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:29:21.636714 6479 consensus_queue.cc:1035] T c46e36b0bceb45b4bafbf1ffaf8e039e P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:29:21.637192 2225 raft_consensus.cc:1261] T c46e36b0bceb45b4bafbf1ffaf8e039e P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:29:21.637708 6395 consensus_queue.cc:1035] T c46e36b0bceb45b4bafbf1ffaf8e039e P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:29:21.640993 4120 raft_consensus.cc:2932] T 05e885ae741c43808c6dfbcdc5769942 P 1fab5794cf1740288abd82f2cebb9b1a [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 16b3a337bc5e462f83ba895ad8660deb (127.15.255.131) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: NON_VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { promote: true } } }
W20240502 21:29:21.655613 2159 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.129:37897, user_credentials={real_user=slave}} blocked reactor thread for 63649us
I20240502 21:29:21.675981 1414 raft_consensus.cc:2932] T 05e885ae741c43808c6dfbcdc5769942 P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 16b3a337bc5e462f83ba895ad8660deb (127.15.255.131) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: NON_VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { promote: true } } }
I20240502 21:29:21.714864 6395 raft_consensus.cc:2932] T c46e36b0bceb45b4bafbf1ffaf8e039e P 521bb93035324cecbebed2f6f2a630f8 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER d8d7297186694f77bb98de9174e4279c (127.15.255.136) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false replace: true } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: true } } }
W20240502 21:29:21.720640 2158 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.134:40821, user_credentials={real_user=slave}} blocked reactor thread for 118353us
I20240502 21:29:21.721062 1550 raft_consensus.cc:2932] T c46e36b0bceb45b4bafbf1ffaf8e039e P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER d8d7297186694f77bb98de9174e4279c (127.15.255.136) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false replace: true } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: true } } }
I20240502 21:29:21.722069 1684 raft_consensus.cc:2932] T c46e36b0bceb45b4bafbf1ffaf8e039e P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER d8d7297186694f77bb98de9174e4279c (127.15.255.136) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false replace: true } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: true } } }
I20240502 21:29:21.730425 1549 raft_consensus.cc:2932] T 05e885ae741c43808c6dfbcdc5769942 P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 16b3a337bc5e462f83ba895ad8660deb (127.15.255.131) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: NON_VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { promote: true } } }
I20240502 21:29:21.735954 1262 catalog_manager.cc:5484] T c46e36b0bceb45b4bafbf1ffaf8e039e P a62a935ff43d494ca35d0c417943133d reported cstate change: config changed from index 7 to 8, NON_VOTER d8d7297186694f77bb98de9174e4279c (127.15.255.136) added. New cstate: current_term: 1 leader_uuid: "521bb93035324cecbebed2f6f2a630f8" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false replace: true } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: true } } }
I20240502 21:29:21.738922 4120 raft_consensus.cc:2932] T c46e36b0bceb45b4bafbf1ffaf8e039e P 1fab5794cf1740288abd82f2cebb9b1a [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER d8d7297186694f77bb98de9174e4279c (127.15.255.136) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false replace: true } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: true } } }
I20240502 21:29:21.740275 1955 raft_consensus.cc:2932] T c46e36b0bceb45b4bafbf1ffaf8e039e P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER d8d7297186694f77bb98de9174e4279c (127.15.255.136) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false replace: true } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: true } } }
I20240502 21:29:21.748770 1413 raft_consensus.cc:2932] T c46e36b0bceb45b4bafbf1ffaf8e039e P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER d8d7297186694f77bb98de9174e4279c (127.15.255.136) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false replace: true } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: true } } }
I20240502 21:29:21.753268 3986 raft_consensus.cc:2932] T 05e885ae741c43808c6dfbcdc5769942 P 16a97c1394f346f0b987f97cce7371b6 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 16b3a337bc5e462f83ba895ad8660deb (127.15.255.131) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: NON_VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { promote: true } } }
I20240502 21:29:21.762868 1954 raft_consensus.cc:2932] T 05e885ae741c43808c6dfbcdc5769942 P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 16b3a337bc5e462f83ba895ad8660deb (127.15.255.131) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: NON_VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { promote: true } } }
I20240502 21:29:21.770355 2225 raft_consensus.cc:2932] T c46e36b0bceb45b4bafbf1ffaf8e039e P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER d8d7297186694f77bb98de9174e4279c (127.15.255.136) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false replace: true } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: true } } }
I20240502 21:29:21.788208 1414 consensus_queue.cc:237] T 815178fbd07b4631b507927683a475c9 P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false replace: true } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: true } }
I20240502 21:29:21.805251 3986 raft_consensus.cc:1261] T 815178fbd07b4631b507927683a475c9 P 16a97c1394f346f0b987f97cce7371b6 [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 21:29:21.805516 4121 raft_consensus.cc:1261] T 815178fbd07b4631b507927683a475c9 P 1fab5794cf1740288abd82f2cebb9b1a [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 21:29:21.805519 2225 raft_consensus.cc:1261] T 815178fbd07b4631b507927683a475c9 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
W20240502 21:29:21.805884 1349 consensus_peers.cc:485] T 815178fbd07b4631b507927683a475c9 P 7d1e5c1b047443cbb28d7b0a9744babf -> Peer d8d7297186694f77bb98de9174e4279c (127.15.255.136:41459): Couldn't send request to peer d8d7297186694f77bb98de9174e4279c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 815178fbd07b4631b507927683a475c9. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:29:21.806216 6652 consensus_queue.cc:1035] T 815178fbd07b4631b507927683a475c9 P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 21:29:21.806739 6652 consensus_queue.cc:1035] T 815178fbd07b4631b507927683a475c9 P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 21:29:21.807942 6654 consensus_queue.cc:1035] T 815178fbd07b4631b507927683a475c9 P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 21:29:21.835685 1549 raft_consensus.cc:1261] T 815178fbd07b4631b507927683a475c9 P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 21:29:21.836448 6652 consensus_queue.cc:1035] T 815178fbd07b4631b507927683a475c9 P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 21:29:21.836620 1954 raft_consensus.cc:1261] T 815178fbd07b4631b507927683a475c9 P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 21:29:21.837174 2087 raft_consensus.cc:1261] T 815178fbd07b4631b507927683a475c9 P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 21:29:21.837388 6652 consensus_queue.cc:1035] T 815178fbd07b4631b507927683a475c9 P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 21:29:21.838225 6652 consensus_queue.cc:1035] T 815178fbd07b4631b507927683a475c9 P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 21:29:21.841542 1954 raft_consensus.cc:2932] T 815178fbd07b4631b507927683a475c9 P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER d8d7297186694f77bb98de9174e4279c (127.15.255.136) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false replace: true } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: true } } }
I20240502 21:29:21.841704 2225 raft_consensus.cc:2932] T 815178fbd07b4631b507927683a475c9 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER d8d7297186694f77bb98de9174e4279c (127.15.255.136) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false replace: true } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: true } } }
I20240502 21:29:21.842427 1549 raft_consensus.cc:2932] T 815178fbd07b4631b507927683a475c9 P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER d8d7297186694f77bb98de9174e4279c (127.15.255.136) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false replace: true } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: true } } }
I20240502 21:29:21.844892 6051 raft_consensus.cc:2932] T 815178fbd07b4631b507927683a475c9 P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER d8d7297186694f77bb98de9174e4279c (127.15.255.136) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false replace: true } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: true } } }
I20240502 21:29:21.850934 1263 catalog_manager.cc:5484] T 815178fbd07b4631b507927683a475c9 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 reported cstate change: config changed from index 10 to 11, NON_VOTER d8d7297186694f77bb98de9174e4279c (127.15.255.136) added. New cstate: current_term: 1 leader_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false replace: true } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: true } } }
I20240502 21:29:21.853530 2087 raft_consensus.cc:2932] T 815178fbd07b4631b507927683a475c9 P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER d8d7297186694f77bb98de9174e4279c (127.15.255.136) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false replace: true } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: true } } }
I20240502 21:29:21.855491 4121 raft_consensus.cc:2932] T 815178fbd07b4631b507927683a475c9 P 1fab5794cf1740288abd82f2cebb9b1a [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER d8d7297186694f77bb98de9174e4279c (127.15.255.136) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false replace: true } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: true } } }
I20240502 21:29:21.856977 3986 raft_consensus.cc:2932] T 815178fbd07b4631b507927683a475c9 P 16a97c1394f346f0b987f97cce7371b6 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER d8d7297186694f77bb98de9174e4279c (127.15.255.136) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false replace: true } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: true } } }
I20240502 21:29:21.880169 1549 consensus_queue.cc:237] T 3c88daaac3734d918955a774829532c6 P a62a935ff43d494ca35d0c417943133d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false replace: true } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: NON_VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: true } }
I20240502 21:29:21.885107 3854 raft_consensus.cc:1261] T 3c88daaac3734d918955a774829532c6 P d8d7297186694f77bb98de9174e4279c [term 1 FOLLOWER]: Refusing update from remote peer a62a935ff43d494ca35d0c417943133d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:29:21.885506 1684 raft_consensus.cc:1261] T 3c88daaac3734d918955a774829532c6 P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Refusing update from remote peer a62a935ff43d494ca35d0c417943133d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:29:21.886175 6365 consensus_queue.cc:1035] T 3c88daaac3734d918955a774829532c6 P a62a935ff43d494ca35d0c417943133d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:29:21.886812 6365 consensus_queue.cc:1035] T 3c88daaac3734d918955a774829532c6 P a62a935ff43d494ca35d0c417943133d [LEADER]: Connected to new peer: Peer: permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:29:21.912678 2225 raft_consensus.cc:1261] T 3c88daaac3734d918955a774829532c6 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Refusing update from remote peer a62a935ff43d494ca35d0c417943133d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:29:21.913512 1414 raft_consensus.cc:1261] T 3c88daaac3734d918955a774829532c6 P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Refusing update from remote peer a62a935ff43d494ca35d0c417943133d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:29:21.913465 6714 consensus_queue.cc:1035] T 3c88daaac3734d918955a774829532c6 P a62a935ff43d494ca35d0c417943133d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:29:21.920106 6715 consensus_queue.cc:1035] T 3c88daaac3734d918955a774829532c6 P a62a935ff43d494ca35d0c417943133d [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:29:21.923141 2087 raft_consensus.cc:1261] T 3c88daaac3734d918955a774829532c6 P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Refusing update from remote peer a62a935ff43d494ca35d0c417943133d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:29:21.924065 1817 raft_consensus.cc:1261] T 3c88daaac3734d918955a774829532c6 P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Refusing update from remote peer a62a935ff43d494ca35d0c417943133d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:29:21.924757 6715 consensus_queue.cc:1035] T 3c88daaac3734d918955a774829532c6 P a62a935ff43d494ca35d0c417943133d [LEADER]: Connected to new peer: Peer: permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:29:21.925520 6715 consensus_queue.cc:1035] T 3c88daaac3734d918955a774829532c6 P a62a935ff43d494ca35d0c417943133d [LEADER]: Connected to new peer: Peer: permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:29:21.931625 6365 raft_consensus.cc:2932] T 3c88daaac3734d918955a774829532c6 P a62a935ff43d494ca35d0c417943133d [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false replace: true } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: NON_VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: true } } }
I20240502 21:29:21.938355 1413 raft_consensus.cc:2932] T 3c88daaac3734d918955a774829532c6 P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false replace: true } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: NON_VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: true } } }
I20240502 21:29:21.952682 2087 raft_consensus.cc:2932] T 3c88daaac3734d918955a774829532c6 P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false replace: true } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: NON_VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: true } } }
I20240502 21:29:21.955071 1263 catalog_manager.cc:5484] T 3c88daaac3734d918955a774829532c6 P 7d1e5c1b047443cbb28d7b0a9744babf reported cstate change: config changed from index 7 to 8, NON_VOTER 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138) added. New cstate: current_term: 1 leader_uuid: "a62a935ff43d494ca35d0c417943133d" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false replace: true } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: NON_VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: true } } }
I20240502 21:29:21.959623 2225 raft_consensus.cc:2932] T 3c88daaac3734d918955a774829532c6 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false replace: true } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: NON_VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: true } } }
I20240502 21:29:21.970046 3854 raft_consensus.cc:2932] T 3c88daaac3734d918955a774829532c6 P d8d7297186694f77bb98de9174e4279c [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false replace: true } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: NON_VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: true } } }
I20240502 21:29:21.995878 1684 raft_consensus.cc:2932] T 3c88daaac3734d918955a774829532c6 P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false replace: true } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: NON_VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: true } } }
I20240502 21:29:21.997182 1817 raft_consensus.cc:2932] T 3c88daaac3734d918955a774829532c6 P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false replace: true } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: NON_VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: true } } }
W20240502 21:29:22.008486 1483 consensus_peers.cc:485] T 3c88daaac3734d918955a774829532c6 P a62a935ff43d494ca35d0c417943133d -> Peer 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138:33375): Couldn't send request to peer 1fab5794cf1740288abd82f2cebb9b1a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3c88daaac3734d918955a774829532c6. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:29:22.055920 6736 ts_tablet_manager.cc:902] T 05e885ae741c43808c6dfbcdc5769942 P 16b3a337bc5e462f83ba895ad8660deb: Initiating tablet copy from peer 6ab8df3c796f4b6cb3e29d5045ad3ea0 (127.15.255.135:34701)
I20240502 21:29:22.075544 6736 tablet_copy_client.cc:308] T 05e885ae741c43808c6dfbcdc5769942 P 16b3a337bc5e462f83ba895ad8660deb: tablet copy: Beginning tablet copy session from remote peer at address 127.15.255.135:34701
I20240502 21:29:22.078214 2249 tablet_copy_service.cc:140] P 6ab8df3c796f4b6cb3e29d5045ad3ea0: Received BeginTabletCopySession request for tablet 05e885ae741c43808c6dfbcdc5769942 from peer 16b3a337bc5e462f83ba895ad8660deb ({username='slave'} at 127.15.255.131:34045)
I20240502 21:29:22.078394 2249 tablet_copy_service.cc:161] P 6ab8df3c796f4b6cb3e29d5045ad3ea0: Beginning new tablet copy session on tablet 05e885ae741c43808c6dfbcdc5769942 from peer 16b3a337bc5e462f83ba895ad8660deb at {username='slave'} at 127.15.255.131:34045: session id = 16b3a337bc5e462f83ba895ad8660deb-05e885ae741c43808c6dfbcdc5769942
I20240502 21:29:22.081001 2249 tablet_copy_source_session.cc:205] T 05e885ae741c43808c6dfbcdc5769942 P 6ab8df3c796f4b6cb3e29d5045ad3ea0: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:29:22.082804 6736 ts_tablet_manager.cc:1875] T 05e885ae741c43808c6dfbcdc5769942 P 16b3a337bc5e462f83ba895ad8660deb: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240502 21:29:22.091646 6736 ts_tablet_manager.cc:1888] T 05e885ae741c43808c6dfbcdc5769942 P 16b3a337bc5e462f83ba895ad8660deb: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.9
I20240502 21:29:22.091925 6736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05e885ae741c43808c6dfbcdc5769942. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:29:22.095335 6736 tablet_copy_client.cc:783] T 05e885ae741c43808c6dfbcdc5769942 P 16b3a337bc5e462f83ba895ad8660deb: tablet copy: Starting download of 0 data blocks...
I20240502 21:29:22.095577 6736 tablet_copy_client.cc:647] T 05e885ae741c43808c6dfbcdc5769942 P 16b3a337bc5e462f83ba895ad8660deb: tablet copy: Starting download of 1 WAL segments...
I20240502 21:29:22.097970 6736 tablet_copy_client.cc:523] T 05e885ae741c43808c6dfbcdc5769942 P 16b3a337bc5e462f83ba895ad8660deb: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:29:22.101857 6736 tablet_bootstrap.cc:492] T 05e885ae741c43808c6dfbcdc5769942 P 16b3a337bc5e462f83ba895ad8660deb: Bootstrap starting.
I20240502 21:29:22.160678 6736 tablet_bootstrap.cc:492] T 05e885ae741c43808c6dfbcdc5769942 P 16b3a337bc5e462f83ba895ad8660deb: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:29:22.161139 6736 tablet_bootstrap.cc:492] T 05e885ae741c43808c6dfbcdc5769942 P 16b3a337bc5e462f83ba895ad8660deb: Bootstrap complete.
I20240502 21:29:22.161464 6736 ts_tablet_manager.cc:1367] T 05e885ae741c43808c6dfbcdc5769942 P 16b3a337bc5e462f83ba895ad8660deb: Time spent bootstrapping tablet: real 0.060s user 0.035s sys 0.019s
I20240502 21:29:22.162345 6736 raft_consensus.cc:348] T 05e885ae741c43808c6dfbcdc5769942 P 16b3a337bc5e462f83ba895ad8660deb [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: NON_VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { promote: true } }
I20240502 21:29:22.162684 6736 raft_consensus.cc:729] T 05e885ae741c43808c6dfbcdc5769942 P 16b3a337bc5e462f83ba895ad8660deb [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 16b3a337bc5e462f83ba895ad8660deb, State: Initialized, Role: LEARNER
I20240502 21:29:22.162977 6736 consensus_queue.cc:260] T 05e885ae741c43808c6dfbcdc5769942 P 16b3a337bc5e462f83ba895ad8660deb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 1.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: NON_VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { promote: true } }
I20240502 21:29:22.163619 6736 ts_tablet_manager.cc:1398] T 05e885ae741c43808c6dfbcdc5769942 P 16b3a337bc5e462f83ba895ad8660deb: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 21:29:22.164712 2249 tablet_copy_service.cc:342] P 6ab8df3c796f4b6cb3e29d5045ad3ea0: Request end of tablet copy session 16b3a337bc5e462f83ba895ad8660deb-05e885ae741c43808c6dfbcdc5769942 received from {username='slave'} at 127.15.255.131:34045
I20240502 21:29:22.164850 2249 tablet_copy_service.cc:433] P 6ab8df3c796f4b6cb3e29d5045ad3ea0: ending tablet copy session 16b3a337bc5e462f83ba895ad8660deb-05e885ae741c43808c6dfbcdc5769942 on tablet 05e885ae741c43808c6dfbcdc5769942 with peer 16b3a337bc5e462f83ba895ad8660deb
I20240502 21:29:22.196677 6774 ts_tablet_manager.cc:902] T c46e36b0bceb45b4bafbf1ffaf8e039e P d8d7297186694f77bb98de9174e4279c: Initiating tablet copy from peer 521bb93035324cecbebed2f6f2a630f8 (127.15.255.134:40821)
I20240502 21:29:22.203125 6773 ts_tablet_manager.cc:902] T 815178fbd07b4631b507927683a475c9 P d8d7297186694f77bb98de9174e4279c: Initiating tablet copy from peer 7d1e5c1b047443cbb28d7b0a9744babf (127.15.255.129:37897)
I20240502 21:29:22.205372 3968 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:22.219713 1664 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:22.232831 1798 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:29:22.240859 6773 tablet_copy_client.cc:308] T 815178fbd07b4631b507927683a475c9 P d8d7297186694f77bb98de9174e4279c: tablet copy: Beginning tablet copy session from remote peer at address 127.15.255.129:37897
I20240502 21:29:22.241575 1434 tablet_copy_service.cc:140] P 7d1e5c1b047443cbb28d7b0a9744babf: Received BeginTabletCopySession request for tablet 815178fbd07b4631b507927683a475c9 from peer d8d7297186694f77bb98de9174e4279c ({username='slave'} at 127.15.255.136:40527)
I20240502 21:29:22.241767 1434 tablet_copy_service.cc:161] P 7d1e5c1b047443cbb28d7b0a9744babf: Beginning new tablet copy session on tablet 815178fbd07b4631b507927683a475c9 from peer d8d7297186694f77bb98de9174e4279c at {username='slave'} at 127.15.255.136:40527: session id = d8d7297186694f77bb98de9174e4279c-815178fbd07b4631b507927683a475c9
I20240502 21:29:22.244812 1434 tablet_copy_source_session.cc:205] T 815178fbd07b4631b507927683a475c9 P 7d1e5c1b047443cbb28d7b0a9744babf: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:29:22.246240 6773 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 815178fbd07b4631b507927683a475c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:29:22.259232 6773 tablet_copy_client.cc:783] T 815178fbd07b4631b507927683a475c9 P d8d7297186694f77bb98de9174e4279c: tablet copy: Starting download of 0 data blocks...
I20240502 21:29:22.259491 6773 tablet_copy_client.cc:647] T 815178fbd07b4631b507927683a475c9 P d8d7297186694f77bb98de9174e4279c: tablet copy: Starting download of 1 WAL segments...
I20240502 21:29:22.266266 6773 tablet_copy_client.cc:523] T 815178fbd07b4631b507927683a475c9 P d8d7297186694f77bb98de9174e4279c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:29:22.272369 4101 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:22.273434 6774 tablet_copy_client.cc:308] T c46e36b0bceb45b4bafbf1ffaf8e039e P d8d7297186694f77bb98de9174e4279c: tablet copy: Beginning tablet copy session from remote peer at address 127.15.255.134:40821
I20240502 21:29:22.274049 2113 tablet_copy_service.cc:140] P 521bb93035324cecbebed2f6f2a630f8: Received BeginTabletCopySession request for tablet c46e36b0bceb45b4bafbf1ffaf8e039e from peer d8d7297186694f77bb98de9174e4279c ({username='slave'} at 127.15.255.136:49789)
I20240502 21:29:22.274219 2113 tablet_copy_service.cc:161] P 521bb93035324cecbebed2f6f2a630f8: Beginning new tablet copy session on tablet c46e36b0bceb45b4bafbf1ffaf8e039e from peer d8d7297186694f77bb98de9174e4279c at {username='slave'} at 127.15.255.136:49789: session id = d8d7297186694f77bb98de9174e4279c-c46e36b0bceb45b4bafbf1ffaf8e039e
I20240502 21:29:22.275324 6773 tablet_bootstrap.cc:492] T 815178fbd07b4631b507927683a475c9 P d8d7297186694f77bb98de9174e4279c: Bootstrap starting.
I20240502 21:29:22.276968 2113 tablet_copy_source_session.cc:205] T c46e36b0bceb45b4bafbf1ffaf8e039e P 521bb93035324cecbebed2f6f2a630f8: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:29:22.278635 6774 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c46e36b0bceb45b4bafbf1ffaf8e039e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:29:22.285914 2068 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 21:29:22.288681 6774 tablet_copy_client.cc:783] T c46e36b0bceb45b4bafbf1ffaf8e039e P d8d7297186694f77bb98de9174e4279c: tablet copy: Starting download of 0 data blocks...
I20240502 21:29:22.288938 6774 tablet_copy_client.cc:647] T c46e36b0bceb45b4bafbf1ffaf8e039e P d8d7297186694f77bb98de9174e4279c: tablet copy: Starting download of 1 WAL segments...
I20240502 21:29:22.294951 6774 tablet_copy_client.cc:523] T c46e36b0bceb45b4bafbf1ffaf8e039e P d8d7297186694f77bb98de9174e4279c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:29:22.319531 6774 tablet_bootstrap.cc:492] T c46e36b0bceb45b4bafbf1ffaf8e039e P d8d7297186694f77bb98de9174e4279c: Bootstrap starting.
I20240502 21:29:22.374255 1394 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:29:22.402411 2205 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:29:22.404165 1527 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:29:22.439455 6773 tablet_bootstrap.cc:492] T 815178fbd07b4631b507927683a475c9 P d8d7297186694f77bb98de9174e4279c: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:29:22.439905 6773 tablet_bootstrap.cc:492] T 815178fbd07b4631b507927683a475c9 P d8d7297186694f77bb98de9174e4279c: Bootstrap complete.
I20240502 21:29:22.440217 6773 ts_tablet_manager.cc:1367] T 815178fbd07b4631b507927683a475c9 P d8d7297186694f77bb98de9174e4279c: Time spent bootstrapping tablet: real 0.165s user 0.060s sys 0.016s
I20240502 21:29:22.441275 6774 tablet_bootstrap.cc:492] T c46e36b0bceb45b4bafbf1ffaf8e039e P d8d7297186694f77bb98de9174e4279c: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:29:22.441769 6774 tablet_bootstrap.cc:492] T c46e36b0bceb45b4bafbf1ffaf8e039e P d8d7297186694f77bb98de9174e4279c: Bootstrap complete.
I20240502 21:29:22.442042 6774 ts_tablet_manager.cc:1367] T c46e36b0bceb45b4bafbf1ffaf8e039e P d8d7297186694f77bb98de9174e4279c: Time spent bootstrapping tablet: real 0.123s user 0.051s sys 0.008s
I20240502 21:29:22.442754 6774 raft_consensus.cc:348] T c46e36b0bceb45b4bafbf1ffaf8e039e P d8d7297186694f77bb98de9174e4279c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false replace: true } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: true } }
I20240502 21:29:22.443087 6774 raft_consensus.cc:729] T c46e36b0bceb45b4bafbf1ffaf8e039e P d8d7297186694f77bb98de9174e4279c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d8d7297186694f77bb98de9174e4279c, State: Initialized, Role: LEARNER
I20240502 21:29:22.443286 6774 consensus_queue.cc:260] T c46e36b0bceb45b4bafbf1ffaf8e039e P d8d7297186694f77bb98de9174e4279c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false replace: true } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: true } }
I20240502 21:29:22.443956 6774 ts_tablet_manager.cc:1398] T c46e36b0bceb45b4bafbf1ffaf8e039e P d8d7297186694f77bb98de9174e4279c: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 21:29:22.444408 6773 raft_consensus.cc:348] T 815178fbd07b4631b507927683a475c9 P d8d7297186694f77bb98de9174e4279c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false replace: true } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: true } }
I20240502 21:29:22.444737 6773 raft_consensus.cc:729] T 815178fbd07b4631b507927683a475c9 P d8d7297186694f77bb98de9174e4279c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d8d7297186694f77bb98de9174e4279c, State: Initialized, Role: LEARNER
I20240502 21:29:22.444962 6773 consensus_queue.cc:260] T 815178fbd07b4631b507927683a475c9 P d8d7297186694f77bb98de9174e4279c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 1.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false replace: true } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: NON_VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: true } }
I20240502 21:29:22.445583 6773 ts_tablet_manager.cc:1398] T 815178fbd07b4631b507927683a475c9 P d8d7297186694f77bb98de9174e4279c: Time spent starting tablet: real 0.005s user 0.003s sys 0.002s
I20240502 21:29:22.446395 2113 tablet_copy_service.cc:342] P 521bb93035324cecbebed2f6f2a630f8: Request end of tablet copy session d8d7297186694f77bb98de9174e4279c-c46e36b0bceb45b4bafbf1ffaf8e039e received from {username='slave'} at 127.15.255.136:49789
I20240502 21:29:22.446524 2113 tablet_copy_service.cc:433] P 521bb93035324cecbebed2f6f2a630f8: ending tablet copy session d8d7297186694f77bb98de9174e4279c-c46e36b0bceb45b4bafbf1ffaf8e039e on tablet c46e36b0bceb45b4bafbf1ffaf8e039e with peer d8d7297186694f77bb98de9174e4279c
I20240502 21:29:22.448078 3836 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:22.448938 1434 tablet_copy_service.cc:342] P 7d1e5c1b047443cbb28d7b0a9744babf: Request end of tablet copy session d8d7297186694f77bb98de9174e4279c-815178fbd07b4631b507927683a475c9 received from {username='slave'} at 127.15.255.136:40527
I20240502 21:29:22.449057 1434 tablet_copy_service.cc:433] P 7d1e5c1b047443cbb28d7b0a9744babf: ending tablet copy session d8d7297186694f77bb98de9174e4279c-815178fbd07b4631b507927683a475c9 on tablet 815178fbd07b4631b507927683a475c9 with peer d8d7297186694f77bb98de9174e4279c
I20240502 21:29:22.462322 6802 ts_tablet_manager.cc:902] T 3c88daaac3734d918955a774829532c6 P 1fab5794cf1740288abd82f2cebb9b1a: Initiating tablet copy from peer a62a935ff43d494ca35d0c417943133d (127.15.255.130:37991)
I20240502 21:29:22.503964 6802 tablet_copy_client.cc:308] T 3c88daaac3734d918955a774829532c6 P 1fab5794cf1740288abd82f2cebb9b1a: tablet copy: Beginning tablet copy session from remote peer at address 127.15.255.130:37991
I20240502 21:29:22.516135 1931 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:29:22.588268 1570 tablet_copy_service.cc:140] P a62a935ff43d494ca35d0c417943133d: Received BeginTabletCopySession request for tablet 3c88daaac3734d918955a774829532c6 from peer 1fab5794cf1740288abd82f2cebb9b1a ({username='slave'} at 127.15.255.138:53287)
I20240502 21:29:22.588457 1570 tablet_copy_service.cc:161] P a62a935ff43d494ca35d0c417943133d: Beginning new tablet copy session on tablet 3c88daaac3734d918955a774829532c6 from peer 1fab5794cf1740288abd82f2cebb9b1a at {username='slave'} at 127.15.255.138:53287: session id = 1fab5794cf1740288abd82f2cebb9b1a-3c88daaac3734d918955a774829532c6
I20240502 21:29:22.593189 1570 tablet_copy_source_session.cc:205] T 3c88daaac3734d918955a774829532c6 P a62a935ff43d494ca35d0c417943133d: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:29:22.594589 6802 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c88daaac3734d918955a774829532c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:29:22.601045 6802 tablet_copy_client.cc:783] T 3c88daaac3734d918955a774829532c6 P 1fab5794cf1740288abd82f2cebb9b1a: tablet copy: Starting download of 0 data blocks...
I20240502 21:29:22.601310 6802 tablet_copy_client.cc:647] T 3c88daaac3734d918955a774829532c6 P 1fab5794cf1740288abd82f2cebb9b1a: tablet copy: Starting download of 1 WAL segments...
I20240502 21:29:22.602830 6802 tablet_copy_client.cc:523] T 3c88daaac3734d918955a774829532c6 P 1fab5794cf1740288abd82f2cebb9b1a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:29:22.606145 6802 tablet_bootstrap.cc:492] T 3c88daaac3734d918955a774829532c6 P 1fab5794cf1740288abd82f2cebb9b1a: Bootstrap starting.
I20240502 21:29:22.624289 1684 raft_consensus.cc:1203] T 05e885ae741c43808c6dfbcdc5769942 P 16b3a337bc5e462f83ba895ad8660deb [term 1 LEARNER]: Deduplicated request from leader. Original: 1.10->[1.11-1.11] Dedup: 1.11->[]
I20240502 21:29:22.656775 6802 tablet_bootstrap.cc:492] T 3c88daaac3734d918955a774829532c6 P 1fab5794cf1740288abd82f2cebb9b1a: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:29:22.657334 6802 tablet_bootstrap.cc:492] T 3c88daaac3734d918955a774829532c6 P 1fab5794cf1740288abd82f2cebb9b1a: Bootstrap complete.
I20240502 21:29:22.657677 6802 ts_tablet_manager.cc:1367] T 3c88daaac3734d918955a774829532c6 P 1fab5794cf1740288abd82f2cebb9b1a: Time spent bootstrapping tablet: real 0.052s user 0.047s sys 0.002s
I20240502 21:29:22.658537 6802 raft_consensus.cc:348] T 3c88daaac3734d918955a774829532c6 P 1fab5794cf1740288abd82f2cebb9b1a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false replace: true } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: NON_VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: true } }
I20240502 21:29:22.658967 6802 raft_consensus.cc:729] T 3c88daaac3734d918955a774829532c6 P 1fab5794cf1740288abd82f2cebb9b1a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1fab5794cf1740288abd82f2cebb9b1a, State: Initialized, Role: LEARNER
I20240502 21:29:22.659298 6802 consensus_queue.cc:260] T 3c88daaac3734d918955a774829532c6 P 1fab5794cf1740288abd82f2cebb9b1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false replace: true } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: NON_VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: true } }
I20240502 21:29:22.660658 3854 raft_consensus.cc:1203] T c46e36b0bceb45b4bafbf1ffaf8e039e P d8d7297186694f77bb98de9174e4279c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240502 21:29:22.672952 6802 ts_tablet_manager.cc:1398] T 3c88daaac3734d918955a774829532c6 P 1fab5794cf1740288abd82f2cebb9b1a: Time spent starting tablet: real 0.015s user 0.007s sys 0.008s
I20240502 21:29:22.674546 1570 tablet_copy_service.cc:342] P a62a935ff43d494ca35d0c417943133d: Request end of tablet copy session 1fab5794cf1740288abd82f2cebb9b1a-3c88daaac3734d918955a774829532c6 received from {username='slave'} at 127.15.255.138:53287
I20240502 21:29:22.674697 1570 tablet_copy_service.cc:433] P a62a935ff43d494ca35d0c417943133d: ending tablet copy session 1fab5794cf1740288abd82f2cebb9b1a-3c88daaac3734d918955a774829532c6 on tablet 3c88daaac3734d918955a774829532c6 with peer 1fab5794cf1740288abd82f2cebb9b1a
W20240502 21:29:22.694433 2161 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.131:36033, user_credentials={real_user=slave}} blocked reactor thread for 62281.6us
W20240502 21:29:22.694816 2023 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.135:34701, user_credentials={real_user=slave}} blocked reactor thread for 63623.7us
I20240502 21:29:22.733151 3854 raft_consensus.cc:1203] T 815178fbd07b4631b507927683a475c9 P d8d7297186694f77bb98de9174e4279c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.10->[1.11-1.11] Dedup: 1.11->[]
I20240502 21:29:22.739760 1413 consensus_queue.cc:237] T aae7b5a01a0543afb1b5a7f44a9a864c P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } attrs { replace: true } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: NON_VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { promote: true } }
I20240502 21:29:22.744019 2225 raft_consensus.cc:1261] T aae7b5a01a0543afb1b5a7f44a9a864c P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 21:29:22.744319 2087 raft_consensus.cc:1261] T aae7b5a01a0543afb1b5a7f44a9a864c P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 21:29:22.744597 4121 raft_consensus.cc:1261] T aae7b5a01a0543afb1b5a7f44a9a864c P 1fab5794cf1740288abd82f2cebb9b1a [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 21:29:22.744740 1954 raft_consensus.cc:1261] T aae7b5a01a0543afb1b5a7f44a9a864c P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
W20240502 21:29:22.745033 1349 consensus_peers.cc:485] T aae7b5a01a0543afb1b5a7f44a9a864c P 7d1e5c1b047443cbb28d7b0a9744babf -> Peer 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132:44947): Couldn't send request to peer 153026dd7f8943cc93fedf6f48450fb4. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:29:22.745165 6051 consensus_queue.cc:1035] T aae7b5a01a0543afb1b5a7f44a9a864c P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 21:29:22.745455 3854 raft_consensus.cc:1261] T aae7b5a01a0543afb1b5a7f44a9a864c P d8d7297186694f77bb98de9174e4279c [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 21:29:22.745617 6794 consensus_queue.cc:1035] T aae7b5a01a0543afb1b5a7f44a9a864c P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 21:29:22.745980 6698 consensus_queue.cc:1035] T aae7b5a01a0543afb1b5a7f44a9a864c P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 21:29:22.746546 6794 consensus_queue.cc:1035] T aae7b5a01a0543afb1b5a7f44a9a864c P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 21:29:22.747305 1954 consensus_queue.cc:237] T 444c1cba2a6f4729aa01b75d06b6b517 P e632c553ff4341328acf1cbaca678392 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: NON_VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { promote: true } }
I20240502 21:29:22.749387 6051 consensus_queue.cc:1035] T aae7b5a01a0543afb1b5a7f44a9a864c P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 21:29:22.750123 1549 raft_consensus.cc:1261] T aae7b5a01a0543afb1b5a7f44a9a864c P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 21:29:22.751065 6051 consensus_queue.cc:1035] T aae7b5a01a0543afb1b5a7f44a9a864c P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
W20240502 21:29:22.764871 1885 consensus_peers.cc:485] T 444c1cba2a6f4729aa01b75d06b6b517 P e632c553ff4341328acf1cbaca678392 -> Peer a62a935ff43d494ca35d0c417943133d (127.15.255.130:37991): Couldn't send request to peer a62a935ff43d494ca35d0c417943133d. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:29:22.767014 3854 raft_consensus.cc:1261] T 444c1cba2a6f4729aa01b75d06b6b517 P d8d7297186694f77bb98de9174e4279c [term 1 FOLLOWER]: Refusing update from remote peer e632c553ff4341328acf1cbaca678392: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 21:29:22.767783 1684 raft_consensus.cc:1261] T 444c1cba2a6f4729aa01b75d06b6b517 P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Refusing update from remote peer e632c553ff4341328acf1cbaca678392: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 21:29:22.767763 2087 raft_consensus.cc:2932] T aae7b5a01a0543afb1b5a7f44a9a864c P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } attrs { replace: true } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: NON_VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { promote: true } } }
I20240502 21:29:22.768431 6750 consensus_queue.cc:1035] T 444c1cba2a6f4729aa01b75d06b6b517 P e632c553ff4341328acf1cbaca678392 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 21:29:22.773305 1548 raft_consensus.cc:2932] T aae7b5a01a0543afb1b5a7f44a9a864c P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } attrs { replace: true } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: NON_VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { promote: true } } }
I20240502 21:29:22.773783 6606 consensus_queue.cc:1035] T 444c1cba2a6f4729aa01b75d06b6b517 P e632c553ff4341328acf1cbaca678392 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 21:29:22.775470 4120 raft_consensus.cc:1261] T 444c1cba2a6f4729aa01b75d06b6b517 P 1fab5794cf1740288abd82f2cebb9b1a [term 1 FOLLOWER]: Refusing update from remote peer e632c553ff4341328acf1cbaca678392: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 21:29:22.776072 6606 consensus_queue.cc:1035] T 444c1cba2a6f4729aa01b75d06b6b517 P e632c553ff4341328acf1cbaca678392 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 21:29:22.766127 6840 raft_consensus.cc:2932] T aae7b5a01a0543afb1b5a7f44a9a864c P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } attrs { replace: true } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: NON_VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { promote: true } } }
I20240502 21:29:22.779278 1265 catalog_manager.cc:5484] T aae7b5a01a0543afb1b5a7f44a9a864c P a62a935ff43d494ca35d0c417943133d reported cstate change: config changed from index 10 to 11, NON_VOTER 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132) added. New cstate: current_term: 1 leader_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } attrs { replace: true } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: NON_VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { promote: true } } }
I20240502 21:29:22.781994 1413 raft_consensus.cc:1261] T 444c1cba2a6f4729aa01b75d06b6b517 P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Refusing update from remote peer e632c553ff4341328acf1cbaca678392: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 21:29:22.784863 6606 consensus_queue.cc:1035] T 444c1cba2a6f4729aa01b75d06b6b517 P e632c553ff4341328acf1cbaca678392 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 21:29:22.786479 3855 raft_consensus.cc:2932] T aae7b5a01a0543afb1b5a7f44a9a864c P d8d7297186694f77bb98de9174e4279c [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } attrs { replace: true } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: NON_VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { promote: true } } }
I20240502 21:29:22.787139 2225 raft_consensus.cc:2932] T aae7b5a01a0543afb1b5a7f44a9a864c P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } attrs { replace: true } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: NON_VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { promote: true } } }
I20240502 21:29:22.794497 1955 raft_consensus.cc:2932] T aae7b5a01a0543afb1b5a7f44a9a864c P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } attrs { replace: true } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: NON_VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { promote: true } } }
I20240502 21:29:22.808844 6750 raft_consensus.cc:2932] T 444c1cba2a6f4729aa01b75d06b6b517 P e632c553ff4341328acf1cbaca678392 [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER a62a935ff43d494ca35d0c417943133d (127.15.255.130) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: NON_VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { promote: true } } }
I20240502 21:29:22.812778 2087 raft_consensus.cc:1261] T 444c1cba2a6f4729aa01b75d06b6b517 P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Refusing update from remote peer e632c553ff4341328acf1cbaca678392: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 21:29:22.815455 2225 raft_consensus.cc:1261] T 444c1cba2a6f4729aa01b75d06b6b517 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Refusing update from remote peer e632c553ff4341328acf1cbaca678392: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 21:29:22.816826 6858 consensus_queue.cc:1035] T 444c1cba2a6f4729aa01b75d06b6b517 P e632c553ff4341328acf1cbaca678392 [LEADER]: Connected to new peer: Peer: permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 21:29:22.817330 3854 raft_consensus.cc:2932] T 444c1cba2a6f4729aa01b75d06b6b517 P d8d7297186694f77bb98de9174e4279c [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER a62a935ff43d494ca35d0c417943133d (127.15.255.130) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: NON_VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { promote: true } } }
I20240502 21:29:22.821588 6606 consensus_queue.cc:1035] T 444c1cba2a6f4729aa01b75d06b6b517 P e632c553ff4341328acf1cbaca678392 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 21:29:22.826026 1268 catalog_manager.cc:5484] T 444c1cba2a6f4729aa01b75d06b6b517 P d8d7297186694f77bb98de9174e4279c reported cstate change: config changed from index 10 to 11, NON_VOTER a62a935ff43d494ca35d0c417943133d (127.15.255.130) added. New cstate: current_term: 1 leader_uuid: "e632c553ff4341328acf1cbaca678392" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: NON_VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { promote: true } } }
I20240502 21:29:22.837831 2087 raft_consensus.cc:2932] T 444c1cba2a6f4729aa01b75d06b6b517 P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER a62a935ff43d494ca35d0c417943133d (127.15.255.130) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: NON_VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { promote: true } } }
I20240502 21:29:22.859310 1684 raft_consensus.cc:2932] T 444c1cba2a6f4729aa01b75d06b6b517 P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER a62a935ff43d494ca35d0c417943133d (127.15.255.130) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: NON_VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { promote: true } } }
I20240502 21:29:22.860308 1413 raft_consensus.cc:2932] T 444c1cba2a6f4729aa01b75d06b6b517 P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER a62a935ff43d494ca35d0c417943133d (127.15.255.130) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: NON_VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { promote: true } } }
I20240502 21:29:22.860812 4120 raft_consensus.cc:2932] T 444c1cba2a6f4729aa01b75d06b6b517 P 1fab5794cf1740288abd82f2cebb9b1a [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER a62a935ff43d494ca35d0c417943133d (127.15.255.130) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: NON_VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { promote: true } } }
I20240502 21:29:22.874686 4119 raft_consensus.cc:2932] T aae7b5a01a0543afb1b5a7f44a9a864c P 1fab5794cf1740288abd82f2cebb9b1a [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } attrs { replace: true } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: NON_VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { promote: true } } }
I20240502 21:29:22.882220 2225 raft_consensus.cc:2932] T 444c1cba2a6f4729aa01b75d06b6b517 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER a62a935ff43d494ca35d0c417943133d (127.15.255.130) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: NON_VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { promote: true } } }
I20240502 21:29:22.898155 2087 consensus_queue.cc:237] T e45af71e3fc14c579b32ba4dbe2adf78 P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: NON_VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { promote: true } }
I20240502 21:29:22.904989 2225 consensus_queue.cc:237] T 4555e51179344acda81670fc9019d968 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } attrs { replace: true } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: NON_VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { promote: true } }
I20240502 21:29:22.917340 3986 raft_consensus.cc:1261] T e45af71e3fc14c579b32ba4dbe2adf78 P 16a97c1394f346f0b987f97cce7371b6 [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 21:29:22.919121 4120 raft_consensus.cc:1261] T e45af71e3fc14c579b32ba4dbe2adf78 P 1fab5794cf1740288abd82f2cebb9b1a [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 21:29:22.920037 1684 raft_consensus.cc:1261] T e45af71e3fc14c579b32ba4dbe2adf78 P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 21:29:22.920171 1413 raft_consensus.cc:1261] T e45af71e3fc14c579b32ba4dbe2adf78 P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 21:29:22.921063 6395 consensus_queue.cc:1035] T e45af71e3fc14c579b32ba4dbe2adf78 P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 21:29:22.921383 2226 raft_consensus.cc:1261] T e45af71e3fc14c579b32ba4dbe2adf78 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 21:29:22.921656 6395 consensus_queue.cc:1035] T e45af71e3fc14c579b32ba4dbe2adf78 P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
W20240502 21:29:22.922709 2023 consensus_peers.cc:485] T e45af71e3fc14c579b32ba4dbe2adf78 P 521bb93035324cecbebed2f6f2a630f8 -> Peer a62a935ff43d494ca35d0c417943133d (127.15.255.130:37991): Couldn't send request to peer a62a935ff43d494ca35d0c417943133d. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:29:22.922933 6817 consensus_queue.cc:1035] T e45af71e3fc14c579b32ba4dbe2adf78 P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 21:29:22.923544 6817 consensus_queue.cc:1035] T e45af71e3fc14c579b32ba4dbe2adf78 P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 21:29:22.924824 1817 raft_consensus.cc:1261] T e45af71e3fc14c579b32ba4dbe2adf78 P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 21:29:22.928819 6479 consensus_queue.cc:1035] T e45af71e3fc14c579b32ba4dbe2adf78 P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 21:29:22.929394 6479 consensus_queue.cc:1035] T e45af71e3fc14c579b32ba4dbe2adf78 P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 21:29:22.937345 1954 raft_consensus.cc:1261] T 4555e51179344acda81670fc9019d968 P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Refusing update from remote peer 6ab8df3c796f4b6cb3e29d5045ad3ea0: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 21:29:22.937860 6819 consensus_queue.cc:1035] T 4555e51179344acda81670fc9019d968 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 21:29:22.938004 4121 raft_consensus.cc:1261] T 4555e51179344acda81670fc9019d968 P 1fab5794cf1740288abd82f2cebb9b1a [term 1 FOLLOWER]: Refusing update from remote peer 6ab8df3c796f4b6cb3e29d5045ad3ea0: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
W20240502 21:29:22.939980 2160 consensus_peers.cc:485] T 4555e51179344acda81670fc9019d968 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 -> Peer 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132:44947): Couldn't send request to peer 153026dd7f8943cc93fedf6f48450fb4. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:29:22.941058 6819 consensus_queue.cc:1035] T 4555e51179344acda81670fc9019d968 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 21:29:22.941512 3988 raft_consensus.cc:1261] T 4555e51179344acda81670fc9019d968 P 16a97c1394f346f0b987f97cce7371b6 [term 1 FOLLOWER]: Refusing update from remote peer 6ab8df3c796f4b6cb3e29d5045ad3ea0: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 21:29:22.941809 2087 raft_consensus.cc:1261] T 4555e51179344acda81670fc9019d968 P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Refusing update from remote peer 6ab8df3c796f4b6cb3e29d5045ad3ea0: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 21:29:22.942167 6819 consensus_queue.cc:1035] T 4555e51179344acda81670fc9019d968 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 21:29:22.944089 6819 consensus_queue.cc:1035] T 4555e51179344acda81670fc9019d968 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 21:29:22.957299 1549 raft_consensus.cc:1261] T 4555e51179344acda81670fc9019d968 P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Refusing update from remote peer 6ab8df3c796f4b6cb3e29d5045ad3ea0: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 21:29:22.957934 6821 consensus_queue.cc:1035] T 4555e51179344acda81670fc9019d968 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 21:29:22.960804 1414 raft_consensus.cc:1261] T 4555e51179344acda81670fc9019d968 P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Refusing update from remote peer 6ab8df3c796f4b6cb3e29d5045ad3ea0: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 21:29:22.961418 6819 consensus_queue.cc:1035] T 4555e51179344acda81670fc9019d968 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 21:29:22.997893 6819 raft_consensus.cc:2932] T 4555e51179344acda81670fc9019d968 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } attrs { replace: true } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: NON_VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { promote: true } } }
I20240502 21:29:23.001848 4121 raft_consensus.cc:2932] T 4555e51179344acda81670fc9019d968 P 1fab5794cf1740288abd82f2cebb9b1a [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } attrs { replace: true } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: NON_VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { promote: true } } }
I20240502 21:29:23.002856 1412 raft_consensus.cc:2932] T 4555e51179344acda81670fc9019d968 P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } attrs { replace: true } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: NON_VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { promote: true } } }
I20240502 21:29:23.005085 1549 raft_consensus.cc:2932] T 4555e51179344acda81670fc9019d968 P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } attrs { replace: true } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: NON_VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { promote: true } } }
I20240502 21:29:23.007951 1954 raft_consensus.cc:2932] T 4555e51179344acda81670fc9019d968 P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } attrs { replace: true } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: NON_VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { promote: true } } }
I20240502 21:29:23.008183 3988 raft_consensus.cc:2932] T 4555e51179344acda81670fc9019d968 P 16a97c1394f346f0b987f97cce7371b6 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } attrs { replace: true } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: NON_VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { promote: true } } }
I20240502 21:29:23.014492 2087 raft_consensus.cc:2932] T 4555e51179344acda81670fc9019d968 P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } attrs { replace: true } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: NON_VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { promote: true } } }
I20240502 21:29:23.014816 1268 catalog_manager.cc:5484] T 4555e51179344acda81670fc9019d968 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 reported cstate change: config changed from index 10 to 11, NON_VOTER 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132) added. New cstate: current_term: 1 leader_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: NON_VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 21:29:23.038779 6479 raft_consensus.cc:2932] T e45af71e3fc14c579b32ba4dbe2adf78 P 521bb93035324cecbebed2f6f2a630f8 [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER a62a935ff43d494ca35d0c417943133d (127.15.255.130) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: NON_VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { promote: true } } }
I20240502 21:29:23.043718 1820 raft_consensus.cc:2932] T e45af71e3fc14c579b32ba4dbe2adf78 P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER a62a935ff43d494ca35d0c417943133d (127.15.255.130) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: NON_VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { promote: true } } }
I20240502 21:29:23.045125 1685 raft_consensus.cc:2932] T e45af71e3fc14c579b32ba4dbe2adf78 P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER a62a935ff43d494ca35d0c417943133d (127.15.255.130) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: NON_VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { promote: true } } }
I20240502 21:29:23.049454 2226 raft_consensus.cc:2932] T e45af71e3fc14c579b32ba4dbe2adf78 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER a62a935ff43d494ca35d0c417943133d (127.15.255.130) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: NON_VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { promote: true } } }
I20240502 21:29:23.049785 1263 catalog_manager.cc:5484] T e45af71e3fc14c579b32ba4dbe2adf78 P 153026dd7f8943cc93fedf6f48450fb4 reported cstate change: config changed from index 10 to 11, NON_VOTER a62a935ff43d494ca35d0c417943133d (127.15.255.130) added. New cstate: current_term: 1 leader_uuid: "521bb93035324cecbebed2f6f2a630f8" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: NON_VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { promote: true } } }
I20240502 21:29:23.070278 4120 raft_consensus.cc:2932] T e45af71e3fc14c579b32ba4dbe2adf78 P 1fab5794cf1740288abd82f2cebb9b1a [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER a62a935ff43d494ca35d0c417943133d (127.15.255.130) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: NON_VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { promote: true } } }
I20240502 21:29:23.071205 1413 raft_consensus.cc:2932] T e45af71e3fc14c579b32ba4dbe2adf78 P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER a62a935ff43d494ca35d0c417943133d (127.15.255.130) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: NON_VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { promote: true } } }
I20240502 21:29:23.079022 6479 raft_consensus.cc:1050] T c46e36b0bceb45b4bafbf1ffaf8e039e P 521bb93035324cecbebed2f6f2a630f8: attempting to promote NON_VOTER d8d7297186694f77bb98de9174e4279c to VOTER
I20240502 21:29:23.080188 6479 consensus_queue.cc:237] T c46e36b0bceb45b4bafbf1ffaf8e039e P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false replace: true } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } }
I20240502 21:29:23.085356 3854 raft_consensus.cc:1261] T c46e36b0bceb45b4bafbf1ffaf8e039e P d8d7297186694f77bb98de9174e4279c [term 1 LEARNER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:29:23.085842 1685 raft_consensus.cc:1261] T c46e36b0bceb45b4bafbf1ffaf8e039e P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:29:23.086731 6817 consensus_queue.cc:1035] T c46e36b0bceb45b4bafbf1ffaf8e039e P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:29:23.087179 3986 raft_consensus.cc:2932] T e45af71e3fc14c579b32ba4dbe2adf78 P 16a97c1394f346f0b987f97cce7371b6 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER a62a935ff43d494ca35d0c417943133d (127.15.255.130) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: NON_VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { promote: true } } }
I20240502 21:29:23.090440 6395 consensus_queue.cc:1035] T c46e36b0bceb45b4bafbf1ffaf8e039e P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:29:23.097854 4121 raft_consensus.cc:1261] T c46e36b0bceb45b4bafbf1ffaf8e039e P 1fab5794cf1740288abd82f2cebb9b1a [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:29:23.098069 1549 raft_consensus.cc:1261] T c46e36b0bceb45b4bafbf1ffaf8e039e P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:29:23.098786 1954 raft_consensus.cc:1261] T c46e36b0bceb45b4bafbf1ffaf8e039e P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:29:23.098784 6817 consensus_queue.cc:1035] T c46e36b0bceb45b4bafbf1ffaf8e039e P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:29:23.100201 6903 consensus_queue.cc:1035] T c46e36b0bceb45b4bafbf1ffaf8e039e P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:29:23.100795 2226 raft_consensus.cc:1261] T c46e36b0bceb45b4bafbf1ffaf8e039e P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:29:23.106304 6817 consensus_queue.cc:1035] T c46e36b0bceb45b4bafbf1ffaf8e039e P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:29:23.107398 6903 consensus_queue.cc:1035] T c46e36b0bceb45b4bafbf1ffaf8e039e P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:29:23.146291 6659 raft_consensus.cc:1050] T 05e885ae741c43808c6dfbcdc5769942 P 6ab8df3c796f4b6cb3e29d5045ad3ea0: attempting to promote NON_VOTER 16b3a337bc5e462f83ba895ad8660deb to VOTER
I20240502 21:29:23.147511 6659 consensus_queue.cc:237] T 05e885ae741c43808c6dfbcdc5769942 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { promote: false } }
I20240502 21:29:23.148460 1412 raft_consensus.cc:1261] T c46e36b0bceb45b4bafbf1ffaf8e039e P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:29:23.150514 6903 consensus_queue.cc:1035] T c46e36b0bceb45b4bafbf1ffaf8e039e P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:29:23.152248 6903 raft_consensus.cc:2932] T c46e36b0bceb45b4bafbf1ffaf8e039e P 521bb93035324cecbebed2f6f2a630f8 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, d8d7297186694f77bb98de9174e4279c (127.15.255.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false replace: true } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } }
I20240502 21:29:23.152601 2226 raft_consensus.cc:2932] T c46e36b0bceb45b4bafbf1ffaf8e039e P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, d8d7297186694f77bb98de9174e4279c (127.15.255.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false replace: true } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } }
I20240502 21:29:23.156320 1549 raft_consensus.cc:2932] T c46e36b0bceb45b4bafbf1ffaf8e039e P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, d8d7297186694f77bb98de9174e4279c (127.15.255.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false replace: true } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } }
I20240502 21:29:23.160849 6915 ts_tablet_manager.cc:902] T aae7b5a01a0543afb1b5a7f44a9a864c P 153026dd7f8943cc93fedf6f48450fb4: Initiating tablet copy from peer 7d1e5c1b047443cbb28d7b0a9744babf (127.15.255.129:37897)
I20240502 21:29:23.161412 4121 raft_consensus.cc:1261] T 05e885ae741c43808c6dfbcdc5769942 P 1fab5794cf1740288abd82f2cebb9b1a [term 1 FOLLOWER]: Refusing update from remote peer 6ab8df3c796f4b6cb3e29d5045ad3ea0: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240502 21:29:23.162098 6814 consensus_queue.cc:1035] T 05e885ae741c43808c6dfbcdc5769942 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 21:29:23.174795 3986 raft_consensus.cc:1261] T 05e885ae741c43808c6dfbcdc5769942 P 16a97c1394f346f0b987f97cce7371b6 [term 1 FOLLOWER]: Refusing update from remote peer 6ab8df3c796f4b6cb3e29d5045ad3ea0: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240502 21:29:23.176236 2087 raft_consensus.cc:1261] T 05e885ae741c43808c6dfbcdc5769942 P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Refusing update from remote peer 6ab8df3c796f4b6cb3e29d5045ad3ea0: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240502 21:29:23.177353 1954 raft_consensus.cc:1261] T 05e885ae741c43808c6dfbcdc5769942 P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Refusing update from remote peer 6ab8df3c796f4b6cb3e29d5045ad3ea0: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240502 21:29:23.180300 1269 catalog_manager.cc:5484] T c46e36b0bceb45b4bafbf1ffaf8e039e P a62a935ff43d494ca35d0c417943133d reported cstate change: config changed from index 8 to 9, d8d7297186694f77bb98de9174e4279c (127.15.255.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "521bb93035324cecbebed2f6f2a630f8" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false replace: true } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } }
I20240502 21:29:23.181460 4121 raft_consensus.cc:1203] T 3c88daaac3734d918955a774829532c6 P 1fab5794cf1740288abd82f2cebb9b1a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240502 21:29:23.182894 6819 consensus_queue.cc:1035] T 05e885ae741c43808c6dfbcdc5769942 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 21:29:23.182785 1415 raft_consensus.cc:2932] T c46e36b0bceb45b4bafbf1ffaf8e039e P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, d8d7297186694f77bb98de9174e4279c (127.15.255.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false replace: true } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } }
I20240502 21:29:23.187230 6819 consensus_queue.cc:1035] T 05e885ae741c43808c6dfbcdc5769942 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 21:29:23.187506 6821 consensus_queue.cc:1035] T 05e885ae741c43808c6dfbcdc5769942 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 21:29:23.189229 1414 raft_consensus.cc:1261] T 05e885ae741c43808c6dfbcdc5769942 P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Refusing update from remote peer 6ab8df3c796f4b6cb3e29d5045ad3ea0: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240502 21:29:23.190887 1685 raft_consensus.cc:1261] T 05e885ae741c43808c6dfbcdc5769942 P 16b3a337bc5e462f83ba895ad8660deb [term 1 LEARNER]: Refusing update from remote peer 6ab8df3c796f4b6cb3e29d5045ad3ea0: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240502 21:29:23.191004 1413 consensus_queue.cc:237] T 20a7af42e83c4639bba20f22dee69f2a P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } attrs { replace: true } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: NON_VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { promote: true } }
I20240502 21:29:23.192433 6819 consensus_queue.cc:1035] T 05e885ae741c43808c6dfbcdc5769942 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 21:29:23.192962 6819 consensus_queue.cc:1035] T 05e885ae741c43808c6dfbcdc5769942 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 21:29:23.194211 3854 raft_consensus.cc:2932] T c46e36b0bceb45b4bafbf1ffaf8e039e P d8d7297186694f77bb98de9174e4279c [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, d8d7297186694f77bb98de9174e4279c (127.15.255.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false replace: true } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } }
I20240502 21:29:23.188206 1412 consensus_queue.cc:237] T f8bee9bdcbff404280f000bdaeadf465 P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { replace: true } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: NON_VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { promote: true } }
I20240502 21:29:23.198683 1954 raft_consensus.cc:2932] T c46e36b0bceb45b4bafbf1ffaf8e039e P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, d8d7297186694f77bb98de9174e4279c (127.15.255.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false replace: true } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } }
I20240502 21:29:23.199146 4119 raft_consensus.cc:2932] T c46e36b0bceb45b4bafbf1ffaf8e039e P 1fab5794cf1740288abd82f2cebb9b1a [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, d8d7297186694f77bb98de9174e4279c (127.15.255.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false replace: true } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } }
I20240502 21:29:23.202107 1686 raft_consensus.cc:1261] T 20a7af42e83c4639bba20f22dee69f2a P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:29:23.202185 3988 raft_consensus.cc:1261] T 20a7af42e83c4639bba20f22dee69f2a P 16a97c1394f346f0b987f97cce7371b6 [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:29:23.202780 6839 consensus_queue.cc:1035] T 20a7af42e83c4639bba20f22dee69f2a P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240502 21:29:23.203305 1349 consensus_peers.cc:485] T 20a7af42e83c4639bba20f22dee69f2a P 7d1e5c1b047443cbb28d7b0a9744babf -> Peer 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132:44947): Couldn't send request to peer 153026dd7f8943cc93fedf6f48450fb4. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:29:23.203390 6839 consensus_queue.cc:1035] T 20a7af42e83c4639bba20f22dee69f2a P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:29:23.211093 1549 raft_consensus.cc:1261] T 05e885ae741c43808c6dfbcdc5769942 P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Refusing update from remote peer 6ab8df3c796f4b6cb3e29d5045ad3ea0: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240502 21:29:23.211504 2089 consensus_queue.cc:237] T c46e36b0bceb45b4bafbf1ffaf8e039e P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } }
I20240502 21:29:23.215135 3855 raft_consensus.cc:1261] T c46e36b0bceb45b4bafbf1ffaf8e039e P d8d7297186694f77bb98de9174e4279c [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 21:29:23.215883 1683 raft_consensus.cc:1261] T c46e36b0bceb45b4bafbf1ffaf8e039e P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 21:29:23.217706 2225 raft_consensus.cc:1261] T c46e36b0bceb45b4bafbf1ffaf8e039e P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 21:29:23.219341 6906 consensus_queue.cc:1035] T c46e36b0bceb45b4bafbf1ffaf8e039e P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 21:29:23.219956 6906 consensus_queue.cc:1035] T c46e36b0bceb45b4bafbf1ffaf8e039e P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:29:23.221565 1683 raft_consensus.cc:2932] T c46e36b0bceb45b4bafbf1ffaf8e039e P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, d8d7297186694f77bb98de9174e4279c (127.15.255.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false replace: true } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } }
I20240502 21:29:23.224736 1415 raft_consensus.cc:1261] T c46e36b0bceb45b4bafbf1ffaf8e039e P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 21:29:23.225549 1549 raft_consensus.cc:1261] T c46e36b0bceb45b4bafbf1ffaf8e039e P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 21:29:23.227485 6903 consensus_queue.cc:1035] T c46e36b0bceb45b4bafbf1ffaf8e039e P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 21:29:23.228078 6903 consensus_queue.cc:1035] T c46e36b0bceb45b4bafbf1ffaf8e039e P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 21:29:23.228667 6903 consensus_queue.cc:1035] T c46e36b0bceb45b4bafbf1ffaf8e039e P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 21:29:23.241046 6659 consensus_queue.cc:1035] T 05e885ae741c43808c6dfbcdc5769942 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.002s
I20240502 21:29:23.244659 1953 raft_consensus.cc:1261] T c46e36b0bceb45b4bafbf1ffaf8e039e P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 21:29:23.244297 6906 raft_consensus.cc:2932] T c46e36b0bceb45b4bafbf1ffaf8e039e P 521bb93035324cecbebed2f6f2a630f8 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } }
I20240502 21:29:23.246229 1548 raft_consensus.cc:1261] T f8bee9bdcbff404280f000bdaeadf465 P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 21:29:23.246963 6840 consensus_queue.cc:1035] T f8bee9bdcbff404280f000bdaeadf465 P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 21:29:23.247596 2089 raft_consensus.cc:1261] T f8bee9bdcbff404280f000bdaeadf465 P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 21:29:23.247833 1953 raft_consensus.cc:1261] T f8bee9bdcbff404280f000bdaeadf465 P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 21:29:23.248206 6840 consensus_queue.cc:1035] T f8bee9bdcbff404280f000bdaeadf465 P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 21:29:23.248543 6698 consensus_queue.cc:1035] T f8bee9bdcbff404280f000bdaeadf465 P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 21:29:23.249410 3988 raft_consensus.cc:1261] T f8bee9bdcbff404280f000bdaeadf465 P 16a97c1394f346f0b987f97cce7371b6 [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 21:29:23.249428 6906 consensus_queue.cc:1035] T c46e36b0bceb45b4bafbf1ffaf8e039e P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 21:29:23.251263 3854 raft_consensus.cc:1261] T f8bee9bdcbff404280f000bdaeadf465 P d8d7297186694f77bb98de9174e4279c [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 21:29:23.251492 1254 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c46e36b0bceb45b4bafbf1ffaf8e039e with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:29:23.252148 6839 consensus_queue.cc:1035] T f8bee9bdcbff404280f000bdaeadf465 P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 21:29:23.252458 2226 raft_consensus.cc:1261] T f8bee9bdcbff404280f000bdaeadf465 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240502 21:29:23.252761 6839 consensus_queue.cc:1035] T f8bee9bdcbff404280f000bdaeadf465 P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 21:29:23.252346 1954 raft_consensus.cc:2932] T c46e36b0bceb45b4bafbf1ffaf8e039e P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } }
I20240502 21:29:23.253072 6698 consensus_queue.cc:1035] T f8bee9bdcbff404280f000bdaeadf465 P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
W20240502 21:29:23.254303 1350 consensus_peers.cc:485] T f8bee9bdcbff404280f000bdaeadf465 P 7d1e5c1b047443cbb28d7b0a9744babf -> Peer 16b3a337bc5e462f83ba895ad8660deb (127.15.255.131:36033): Couldn't send request to peer 16b3a337bc5e462f83ba895ad8660deb. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:29:23.258395 1547 raft_consensus.cc:2932] T c46e36b0bceb45b4bafbf1ffaf8e039e P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } }
I20240502 21:29:23.260227 1268 catalog_manager.cc:5484] T c46e36b0bceb45b4bafbf1ffaf8e039e P 521bb93035324cecbebed2f6f2a630f8 reported cstate change: config changed from index 9 to 10, VOTER 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138) evicted. New cstate: current_term: 1 leader_uuid: "521bb93035324cecbebed2f6f2a630f8" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 21:29:23.262799 2225 raft_consensus.cc:2932] T c46e36b0bceb45b4bafbf1ffaf8e039e P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } }
I20240502 21:29:23.280851 1685 raft_consensus.cc:2932] T c46e36b0bceb45b4bafbf1ffaf8e039e P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } }
I20240502 21:29:23.283058 1411 raft_consensus.cc:2932] T c46e36b0bceb45b4bafbf1ffaf8e039e P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } }
I20240502 21:29:23.284482 3854 raft_consensus.cc:2932] T c46e36b0bceb45b4bafbf1ffaf8e039e P d8d7297186694f77bb98de9174e4279c [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } }
I20240502 21:29:23.290437 6814 raft_consensus.cc:2932] T 05e885ae741c43808c6dfbcdc5769942 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 16b3a337bc5e462f83ba895ad8660deb (127.15.255.131) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { promote: false } } }
I20240502 21:29:23.298509 1411 raft_consensus.cc:2932] T 05e885ae741c43808c6dfbcdc5769942 P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 16b3a337bc5e462f83ba895ad8660deb (127.15.255.131) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { promote: false } } }
I20240502 21:29:23.298959 4119 raft_consensus.cc:2932] T 05e885ae741c43808c6dfbcdc5769942 P 1fab5794cf1740288abd82f2cebb9b1a [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 16b3a337bc5e462f83ba895ad8660deb (127.15.255.131) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { promote: false } } }
I20240502 21:29:23.304327 1955 raft_consensus.cc:2932] T 05e885ae741c43808c6dfbcdc5769942 P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 16b3a337bc5e462f83ba895ad8660deb (127.15.255.131) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { promote: false } } }
I20240502 21:29:23.304888 1263 catalog_manager.cc:5484] T 05e885ae741c43808c6dfbcdc5769942 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 reported cstate change: config changed from index 11 to 12, 16b3a337bc5e462f83ba895ad8660deb (127.15.255.131) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 21:29:23.311280 2086 raft_consensus.cc:2932] T 05e885ae741c43808c6dfbcdc5769942 P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 16b3a337bc5e462f83ba895ad8660deb (127.15.255.131) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { promote: false } } }
I20240502 21:29:23.316192 1549 raft_consensus.cc:2932] T 05e885ae741c43808c6dfbcdc5769942 P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 16b3a337bc5e462f83ba895ad8660deb (127.15.255.131) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { promote: false } } }
I20240502 21:29:23.315086 3986 raft_consensus.cc:2932] T 05e885ae741c43808c6dfbcdc5769942 P 16a97c1394f346f0b987f97cce7371b6 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 16b3a337bc5e462f83ba895ad8660deb (127.15.255.131) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { promote: false } } }
I20240502 21:29:23.330343 6839 raft_consensus.cc:1050] T 815178fbd07b4631b507927683a475c9 P 7d1e5c1b047443cbb28d7b0a9744babf: attempting to promote NON_VOTER d8d7297186694f77bb98de9174e4279c to VOTER
I20240502 21:29:23.331795 6839 consensus_queue.cc:237] T 815178fbd07b4631b507927683a475c9 P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false replace: true } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } }
I20240502 21:29:23.335237 6933 ts_tablet_manager.cc:902] T 444c1cba2a6f4729aa01b75d06b6b517 P a62a935ff43d494ca35d0c417943133d: Initiating tablet copy from peer e632c553ff4341328acf1cbaca678392 (127.15.255.133:37439)
I20240502 21:29:23.335382 6698 raft_consensus.cc:2932] T f8bee9bdcbff404280f000bdaeadf465 P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 16b3a337bc5e462f83ba895ad8660deb (127.15.255.131) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { replace: true } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: NON_VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { promote: true } } }
I20240502 21:29:23.340375 3987 raft_consensus.cc:2932] T f8bee9bdcbff404280f000bdaeadf465 P 16a97c1394f346f0b987f97cce7371b6 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 16b3a337bc5e462f83ba895ad8660deb (127.15.255.131) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { replace: true } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: NON_VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { promote: true } } }
I20240502 21:29:23.346678 1686 raft_consensus.cc:2932] T 05e885ae741c43808c6dfbcdc5769942 P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 16b3a337bc5e462f83ba895ad8660deb (127.15.255.131) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { replace: true } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { promote: false } } }
I20240502 21:29:23.348219 2226 consensus_queue.cc:237] T 05e885ae741c43808c6dfbcdc5769942 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { promote: false } }
I20240502 21:29:23.352138 2086 raft_consensus.cc:1261] T 05e885ae741c43808c6dfbcdc5769942 P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Refusing update from remote peer 6ab8df3c796f4b6cb3e29d5045ad3ea0: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240502 21:29:23.350023 3854 raft_consensus.cc:2932] T f8bee9bdcbff404280f000bdaeadf465 P d8d7297186694f77bb98de9174e4279c [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 16b3a337bc5e462f83ba895ad8660deb (127.15.255.131) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { replace: true } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: NON_VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { promote: true } } }
I20240502 21:29:23.352787 6819 consensus_queue.cc:1035] T 05e885ae741c43808c6dfbcdc5769942 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240502 21:29:23.352628 1269 catalog_manager.cc:5484] T f8bee9bdcbff404280f000bdaeadf465 P 16a97c1394f346f0b987f97cce7371b6 reported cstate change: config changed from index 10 to 11, NON_VOTER 16b3a337bc5e462f83ba895ad8660deb (127.15.255.131) added. New cstate: current_term: 1 leader_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { replace: true } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: NON_VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { promote: true } } }
I20240502 21:29:23.351022 1955 raft_consensus.cc:2932] T f8bee9bdcbff404280f000bdaeadf465 P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 16b3a337bc5e462f83ba895ad8660deb (127.15.255.131) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { replace: true } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: NON_VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { promote: true } } }
I20240502 21:29:23.357328 1548 raft_consensus.cc:2932] T f8bee9bdcbff404280f000bdaeadf465 P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 16b3a337bc5e462f83ba895ad8660deb (127.15.255.131) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { replace: true } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: NON_VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { promote: true } } }
I20240502 21:29:23.358623 4119 raft_consensus.cc:1261] T 05e885ae741c43808c6dfbcdc5769942 P 1fab5794cf1740288abd82f2cebb9b1a [term 1 FOLLOWER]: Refusing update from remote peer 6ab8df3c796f4b6cb3e29d5045ad3ea0: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240502 21:29:23.359308 6814 consensus_queue.cc:1035] T 05e885ae741c43808c6dfbcdc5769942 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240502 21:29:23.359541 6718 raft_consensus.cc:1050] T 3c88daaac3734d918955a774829532c6 P a62a935ff43d494ca35d0c417943133d: attempting to promote NON_VOTER 1fab5794cf1740288abd82f2cebb9b1a to VOTER
I20240502 21:29:23.359906 1953 raft_consensus.cc:1261] T 05e885ae741c43808c6dfbcdc5769942 P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Refusing update from remote peer 6ab8df3c796f4b6cb3e29d5045ad3ea0: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240502 21:29:23.360448 6814 consensus_queue.cc:1035] T 05e885ae741c43808c6dfbcdc5769942 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240502 21:29:23.360632 6718 consensus_queue.cc:237] T 3c88daaac3734d918955a774829532c6 P a62a935ff43d494ca35d0c417943133d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false replace: true } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } }
I20240502 21:29:23.362865 3987 raft_consensus.cc:1261] T 05e885ae741c43808c6dfbcdc5769942 P 16a97c1394f346f0b987f97cce7371b6 [term 1 FOLLOWER]: Refusing update from remote peer 6ab8df3c796f4b6cb3e29d5045ad3ea0: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240502 21:29:23.363518 6814 consensus_queue.cc:1035] T 05e885ae741c43808c6dfbcdc5769942 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240502 21:29:23.366083 6814 raft_consensus.cc:2932] T 05e885ae741c43808c6dfbcdc5769942 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 LEADER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER a62a935ff43d494ca35d0c417943133d (127.15.255.130) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { promote: false } } }
I20240502 21:29:23.371353 3986 raft_consensus.cc:2932] T 05e885ae741c43808c6dfbcdc5769942 P 16a97c1394f346f0b987f97cce7371b6 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER a62a935ff43d494ca35d0c417943133d (127.15.255.130) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { promote: false } } }
I20240502 21:29:23.373064 1255 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 05e885ae741c43808c6dfbcdc5769942 with cas_config_opid_index 12: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:29:23.373265 2086 raft_consensus.cc:2932] T 05e885ae741c43808c6dfbcdc5769942 P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER a62a935ff43d494ca35d0c417943133d (127.15.255.130) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { promote: false } } }
I20240502 21:29:23.374322 3855 raft_consensus.cc:1261] T 815178fbd07b4631b507927683a475c9 P d8d7297186694f77bb98de9174e4279c [term 1 LEARNER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240502 21:29:23.374825 6698 consensus_queue.cc:1035] T 815178fbd07b4631b507927683a475c9 P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 21:29:23.374799 4119 raft_consensus.cc:2932] T 05e885ae741c43808c6dfbcdc5769942 P 1fab5794cf1740288abd82f2cebb9b1a [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER a62a935ff43d494ca35d0c417943133d (127.15.255.130) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { promote: false } } }
I20240502 21:29:23.377671 4121 raft_consensus.cc:1261] T 815178fbd07b4631b507927683a475c9 P 1fab5794cf1740288abd82f2cebb9b1a [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240502 21:29:23.378417 6840 consensus_queue.cc:1035] T 815178fbd07b4631b507927683a475c9 P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 21:29:23.378839 1550 raft_consensus.cc:1261] T 815178fbd07b4631b507927683a475c9 P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240502 21:29:23.378970 1954 raft_consensus.cc:1261] T 815178fbd07b4631b507927683a475c9 P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240502 21:29:23.379334 3987 raft_consensus.cc:1261] T 815178fbd07b4631b507927683a475c9 P 16a97c1394f346f0b987f97cce7371b6 [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240502 21:29:23.379371 6840 consensus_queue.cc:1035] T 815178fbd07b4631b507927683a475c9 P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 21:29:23.379634 6698 consensus_queue.cc:1035] T 815178fbd07b4631b507927683a475c9 P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 21:29:23.379990 6839 consensus_queue.cc:1035] T 815178fbd07b4631b507927683a475c9 P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 21:29:23.380473 1953 raft_consensus.cc:2932] T 05e885ae741c43808c6dfbcdc5769942 P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER a62a935ff43d494ca35d0c417943133d (127.15.255.130) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { promote: false } } }
I20240502 21:29:23.381507 2086 raft_consensus.cc:1261] T 815178fbd07b4631b507927683a475c9 P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240502 21:29:23.382336 6794 consensus_queue.cc:1035] T 815178fbd07b4631b507927683a475c9 P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 21:29:23.388314 1269 catalog_manager.cc:5484] T 05e885ae741c43808c6dfbcdc5769942 P 521bb93035324cecbebed2f6f2a630f8 reported cstate change: config changed from index 12 to 13, VOTER a62a935ff43d494ca35d0c417943133d (127.15.255.130) evicted. New cstate: current_term: 1 leader_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { promote: false } } }
I20240502 21:29:23.394093 1411 raft_consensus.cc:1261] T 05e885ae741c43808c6dfbcdc5769942 P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Refusing update from remote peer 6ab8df3c796f4b6cb3e29d5045ad3ea0: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240502 21:29:23.394752 6814 consensus_queue.cc:1035] T 05e885ae741c43808c6dfbcdc5769942 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240502 21:29:23.396458 1414 raft_consensus.cc:2932] T 05e885ae741c43808c6dfbcdc5769942 P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER a62a935ff43d494ca35d0c417943133d (127.15.255.130) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { promote: false } } }
I20240502 21:29:23.401537 2225 raft_consensus.cc:1261] T 815178fbd07b4631b507927683a475c9 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240502 21:29:23.403126 6840 consensus_queue.cc:1035] T 815178fbd07b4631b507927683a475c9 P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 21:29:23.406334 1412 raft_consensus.cc:1261] T 3c88daaac3734d918955a774829532c6 P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Refusing update from remote peer a62a935ff43d494ca35d0c417943133d: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:29:23.406937 6873 consensus_queue.cc:1035] T 3c88daaac3734d918955a774829532c6 P a62a935ff43d494ca35d0c417943133d [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:29:23.409088 1684 raft_consensus.cc:1261] T 3c88daaac3734d918955a774829532c6 P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Refusing update from remote peer a62a935ff43d494ca35d0c417943133d: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:29:23.409663 6873 consensus_queue.cc:1035] T 3c88daaac3734d918955a774829532c6 P a62a935ff43d494ca35d0c417943133d [LEADER]: Connected to new peer: Peer: permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:29:23.425756 1527 tablet_service.cc:1514] Processing DeleteTablet for tablet 05e885ae741c43808c6dfbcdc5769942 with delete_type TABLET_DATA_TOMBSTONED (TS a62a935ff43d494ca35d0c417943133d not found in new config with opid_index 13) from {username='slave'} at 127.0.0.1:43860
I20240502 21:29:23.427574 3856 raft_consensus.cc:1261] T 3c88daaac3734d918955a774829532c6 P d8d7297186694f77bb98de9174e4279c [term 1 FOLLOWER]: Refusing update from remote peer a62a935ff43d494ca35d0c417943133d: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:29:23.428246 6873 consensus_queue.cc:1035] T 3c88daaac3734d918955a774829532c6 P a62a935ff43d494ca35d0c417943133d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:29:23.444367 6698 raft_consensus.cc:2932] T 815178fbd07b4631b507927683a475c9 P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, d8d7297186694f77bb98de9174e4279c (127.15.255.136) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false replace: true } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } }
I20240502 21:29:23.451545 6955 ts_tablet_manager.cc:902] T 4555e51179344acda81670fc9019d968 P 153026dd7f8943cc93fedf6f48450fb4: Initiating tablet copy from peer 6ab8df3c796f4b6cb3e29d5045ad3ea0 (127.15.255.135:34701)
I20240502 21:29:23.451292 3987 raft_consensus.cc:2932] T 815178fbd07b4631b507927683a475c9 P 16a97c1394f346f0b987f97cce7371b6 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, d8d7297186694f77bb98de9174e4279c (127.15.255.136) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false replace: true } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } }
I20240502 21:29:23.452821 6956 tablet_replica.cc:330] T 05e885ae741c43808c6dfbcdc5769942 P a62a935ff43d494ca35d0c417943133d: stopping tablet replica
I20240502 21:29:23.453105 6956 raft_consensus.cc:2229] T 05e885ae741c43808c6dfbcdc5769942 P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:29:23.453586 6956 raft_consensus.cc:2258] T 05e885ae741c43808c6dfbcdc5769942 P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:29:23.454731 6956 ts_tablet_manager.cc:1875] T 05e885ae741c43808c6dfbcdc5769942 P a62a935ff43d494ca35d0c417943133d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:29:23.453738 2086 raft_consensus.cc:2932] T 815178fbd07b4631b507927683a475c9 P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, d8d7297186694f77bb98de9174e4279c (127.15.255.136) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false replace: true } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } }
I20240502 21:29:23.454790 3856 raft_consensus.cc:2932] T 815178fbd07b4631b507927683a475c9 P d8d7297186694f77bb98de9174e4279c [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, d8d7297186694f77bb98de9174e4279c (127.15.255.136) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false replace: true } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } }
I20240502 21:29:23.458168 1265 catalog_manager.cc:5484] T 815178fbd07b4631b507927683a475c9 P 16a97c1394f346f0b987f97cce7371b6 reported cstate change: config changed from index 11 to 12, d8d7297186694f77bb98de9174e4279c (127.15.255.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false replace: true } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } }
I20240502 21:29:23.463768 1685 raft_consensus.cc:1261] T 05e885ae741c43808c6dfbcdc5769942 P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Refusing update from remote peer 6ab8df3c796f4b6cb3e29d5045ad3ea0: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240502 21:29:23.464461 6659 consensus_queue.cc:1035] T 05e885ae741c43808c6dfbcdc5769942 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240502 21:29:23.466418 2087 raft_consensus.cc:1261] T 20a7af42e83c4639bba20f22dee69f2a P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:29:23.466472 2226 raft_consensus.cc:1261] T 20a7af42e83c4639bba20f22dee69f2a P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:29:23.466992 6794 consensus_queue.cc:1035] T 20a7af42e83c4639bba20f22dee69f2a P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:29:23.467633 4121 raft_consensus.cc:2932] T 815178fbd07b4631b507927683a475c9 P 1fab5794cf1740288abd82f2cebb9b1a [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, d8d7297186694f77bb98de9174e4279c (127.15.255.136) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false replace: true } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } }
I20240502 21:29:23.468808 1548 raft_consensus.cc:1261] T 20a7af42e83c4639bba20f22dee69f2a P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:29:23.472924 1685 raft_consensus.cc:2932] T 05e885ae741c43808c6dfbcdc5769942 P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER a62a935ff43d494ca35d0c417943133d (127.15.255.130) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { promote: false } } }
I20240502 21:29:23.483834 6839 consensus_queue.cc:1035] T 20a7af42e83c4639bba20f22dee69f2a P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:29:23.487356 1955 raft_consensus.cc:1261] T 20a7af42e83c4639bba20f22dee69f2a P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:29:23.488250 6840 consensus_queue.cc:1035] T 20a7af42e83c4639bba20f22dee69f2a P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:29:23.494283 6794 consensus_queue.cc:1035] T 20a7af42e83c4639bba20f22dee69f2a P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:29:23.495558 2086 raft_consensus.cc:2932] T f8bee9bdcbff404280f000bdaeadf465 P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 16b3a337bc5e462f83ba895ad8660deb (127.15.255.131) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { replace: true } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: NON_VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { promote: true } } }
I20240502 21:29:23.507762 1952 raft_consensus.cc:2932] T 815178fbd07b4631b507927683a475c9 P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, d8d7297186694f77bb98de9174e4279c (127.15.255.136) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false replace: true } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } }
I20240502 21:29:23.522187 6698 raft_consensus.cc:2932] T 20a7af42e83c4639bba20f22dee69f2a P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } attrs { replace: true } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: NON_VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { promote: true } } }
I20240502 21:29:23.527635 1549 raft_consensus.cc:2932] T 815178fbd07b4631b507927683a475c9 P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, d8d7297186694f77bb98de9174e4279c (127.15.255.136) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false replace: true } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } }
I20240502 21:29:23.534456 1547 raft_consensus.cc:2932] T 20a7af42e83c4639bba20f22dee69f2a P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } attrs { replace: true } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: NON_VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { promote: true } } }
I20240502 21:29:23.536690 6963 ts_tablet_manager.cc:902] T e45af71e3fc14c579b32ba4dbe2adf78 P a62a935ff43d494ca35d0c417943133d: Initiating tablet copy from peer 521bb93035324cecbebed2f6f2a630f8 (127.15.255.134:40821)
I20240502 21:29:23.542208 3986 raft_consensus.cc:2932] T 20a7af42e83c4639bba20f22dee69f2a P 16a97c1394f346f0b987f97cce7371b6 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } attrs { replace: true } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: NON_VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { promote: true } } }
I20240502 21:29:23.547336 4101 tablet_service.cc:1514] Processing DeleteTablet for tablet c46e36b0bceb45b4bafbf1ffaf8e039e with delete_type TABLET_DATA_TOMBSTONED (TS 1fab5794cf1740288abd82f2cebb9b1a not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:41856
I20240502 21:29:23.549531 1262 catalog_manager.cc:5484] T 20a7af42e83c4639bba20f22dee69f2a P 16a97c1394f346f0b987f97cce7371b6 reported cstate change: config changed from index 7 to 8, NON_VOTER 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132) added. New cstate: current_term: 1 leader_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } attrs { replace: true } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: NON_VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { promote: true } } }
I20240502 21:29:23.552279 2225 raft_consensus.cc:2932] T 815178fbd07b4631b507927683a475c9 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, d8d7297186694f77bb98de9174e4279c (127.15.255.136) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false replace: true } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } }
I20240502 21:29:23.555286 1955 raft_consensus.cc:2932] T 20a7af42e83c4639bba20f22dee69f2a P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } attrs { replace: true } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: NON_VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { promote: true } } }
I20240502 21:29:23.561460 2087 raft_consensus.cc:2932] T 20a7af42e83c4639bba20f22dee69f2a P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } attrs { replace: true } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: NON_VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { promote: true } } }
I20240502 21:29:23.566402 1686 raft_consensus.cc:2932] T 20a7af42e83c4639bba20f22dee69f2a P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } attrs { replace: true } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: NON_VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { promote: true } } }
I20240502 21:29:23.574296 2227 raft_consensus.cc:2932] T 20a7af42e83c4639bba20f22dee69f2a P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } attrs { replace: true } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: NON_VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { promote: true } } }
I20240502 21:29:23.590282 6976 tablet_replica.cc:330] T c46e36b0bceb45b4bafbf1ffaf8e039e P 1fab5794cf1740288abd82f2cebb9b1a: stopping tablet replica
I20240502 21:29:23.591003 6976 raft_consensus.cc:2229] T c46e36b0bceb45b4bafbf1ffaf8e039e P 1fab5794cf1740288abd82f2cebb9b1a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:29:23.591459 6976 raft_consensus.cc:2258] T c46e36b0bceb45b4bafbf1ffaf8e039e P 1fab5794cf1740288abd82f2cebb9b1a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:29:23.603138 1412 consensus_queue.cc:237] T 815178fbd07b4631b507927683a475c9 P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } }
I20240502 21:29:23.605943 3854 raft_consensus.cc:1261] T 815178fbd07b4631b507927683a475c9 P d8d7297186694f77bb98de9174e4279c [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240502 21:29:23.606546 1955 raft_consensus.cc:1261] T 815178fbd07b4631b507927683a475c9 P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240502 21:29:23.607021 2087 raft_consensus.cc:1261] T 815178fbd07b4631b507927683a475c9 P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240502 21:29:23.607137 4119 raft_consensus.cc:1261] T 815178fbd07b4631b507927683a475c9 P 1fab5794cf1740288abd82f2cebb9b1a [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240502 21:29:23.607729 6698 consensus_queue.cc:1035] T 815178fbd07b4631b507927683a475c9 P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240502 21:29:23.608577 6698 consensus_queue.cc:1035] T 815178fbd07b4631b507927683a475c9 P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240502 21:29:23.610706 6698 consensus_queue.cc:1035] T 815178fbd07b4631b507927683a475c9 P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240502 21:29:23.604352 6956 ts_tablet_manager.cc:1888] T 05e885ae741c43808c6dfbcdc5769942 P a62a935ff43d494ca35d0c417943133d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.12
I20240502 21:29:23.613548 6956 log.cc:1196] T 05e885ae741c43808c6dfbcdc5769942 P a62a935ff43d494ca35d0c417943133d: Deleting WAL directory at /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-1/wal/wals/05e885ae741c43808c6dfbcdc5769942
I20240502 21:29:23.614504 1255 catalog_manager.cc:4830] TS a62a935ff43d494ca35d0c417943133d (127.15.255.130:37991): tablet 05e885ae741c43808c6dfbcdc5769942 (table rebalance_test_table_4 [id=d523de924ba14cb493e559296c083a34]) successfully deleted
I20240502 21:29:23.614853 6839 raft_consensus.cc:2932] T 815178fbd07b4631b507927683a475c9 P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 LEADER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER 16a97c1394f346f0b987f97cce7371b6 (127.15.255.137) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } }
I20240502 21:29:23.615608 3854 raft_consensus.cc:2932] T 815178fbd07b4631b507927683a475c9 P d8d7297186694f77bb98de9174e4279c [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER 16a97c1394f346f0b987f97cce7371b6 (127.15.255.137) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } }
I20240502 21:29:23.617645 6976 ts_tablet_manager.cc:1875] T c46e36b0bceb45b4bafbf1ffaf8e039e P 1fab5794cf1740288abd82f2cebb9b1a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:29:23.622203 2227 raft_consensus.cc:1261] T 815178fbd07b4631b507927683a475c9 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240502 21:29:23.624524 1549 raft_consensus.cc:1261] T 815178fbd07b4631b507927683a475c9 P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240502 21:29:23.625370 6698 consensus_queue.cc:1035] T 815178fbd07b4631b507927683a475c9 P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240502 21:29:23.625372 2226 raft_consensus.cc:2932] T f8bee9bdcbff404280f000bdaeadf465 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 16b3a337bc5e462f83ba895ad8660deb (127.15.255.131) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { replace: true } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: NON_VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { promote: true } } }
I20240502 21:29:23.625852 6698 consensus_queue.cc:1035] T 815178fbd07b4631b507927683a475c9 P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240502 21:29:23.626335 1269 catalog_manager.cc:5484] T 815178fbd07b4631b507927683a475c9 P d8d7297186694f77bb98de9174e4279c reported cstate change: config changed from index 12 to 13, VOTER 16a97c1394f346f0b987f97cce7371b6 (127.15.255.137) evicted. New cstate: current_term: 1 leader_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } }
I20240502 21:29:23.627736 6955 tablet_copy_client.cc:308] T 4555e51179344acda81670fc9019d968 P 153026dd7f8943cc93fedf6f48450fb4: tablet copy: Beginning tablet copy session from remote peer at address 127.15.255.135:34701
I20240502 21:29:23.628669 2249 tablet_copy_service.cc:140] P 6ab8df3c796f4b6cb3e29d5045ad3ea0: Received BeginTabletCopySession request for tablet 4555e51179344acda81670fc9019d968 from peer 153026dd7f8943cc93fedf6f48450fb4 ({username='slave'} at 127.15.255.132:47173)
I20240502 21:29:23.628818 2249 tablet_copy_service.cc:161] P 6ab8df3c796f4b6cb3e29d5045ad3ea0: Beginning new tablet copy session on tablet 4555e51179344acda81670fc9019d968 from peer 153026dd7f8943cc93fedf6f48450fb4 at {username='slave'} at 127.15.255.132:47173: session id = 153026dd7f8943cc93fedf6f48450fb4-4555e51179344acda81670fc9019d968
I20240502 21:29:23.628002 1549 raft_consensus.cc:2932] T 815178fbd07b4631b507927683a475c9 P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER 16a97c1394f346f0b987f97cce7371b6 (127.15.255.137) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } }
I20240502 21:29:23.631664 2249 tablet_copy_source_session.cc:205] T 4555e51179344acda81670fc9019d968 P 6ab8df3c796f4b6cb3e29d5045ad3ea0: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:29:23.631873 1255 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 815178fbd07b4631b507927683a475c9 with cas_config_opid_index 12: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:29:23.643623 6955 ts_tablet_manager.cc:1875] T 4555e51179344acda81670fc9019d968 P 153026dd7f8943cc93fedf6f48450fb4: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240502 21:29:23.644196 6915 tablet_copy_client.cc:308] T aae7b5a01a0543afb1b5a7f44a9a864c P 153026dd7f8943cc93fedf6f48450fb4: tablet copy: Beginning tablet copy session from remote peer at address 127.15.255.129:37897
I20240502 21:29:23.644991 1434 tablet_copy_service.cc:140] P 7d1e5c1b047443cbb28d7b0a9744babf: Received BeginTabletCopySession request for tablet aae7b5a01a0543afb1b5a7f44a9a864c from peer 153026dd7f8943cc93fedf6f48450fb4 ({username='slave'} at 127.15.255.132:33345)
I20240502 21:29:23.645159 1434 tablet_copy_service.cc:161] P 7d1e5c1b047443cbb28d7b0a9744babf: Beginning new tablet copy session on tablet aae7b5a01a0543afb1b5a7f44a9a864c from peer 153026dd7f8943cc93fedf6f48450fb4 at {username='slave'} at 127.15.255.132:33345: session id = 153026dd7f8943cc93fedf6f48450fb4-aae7b5a01a0543afb1b5a7f44a9a864c
I20240502 21:29:23.646371 1955 raft_consensus.cc:2932] T 815178fbd07b4631b507927683a475c9 P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER 16a97c1394f346f0b987f97cce7371b6 (127.15.255.137) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } }
I20240502 21:29:23.647332 2087 raft_consensus.cc:2932] T 815178fbd07b4631b507927683a475c9 P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER 16a97c1394f346f0b987f97cce7371b6 (127.15.255.137) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } }
I20240502 21:29:23.648054 1434 tablet_copy_source_session.cc:205] T aae7b5a01a0543afb1b5a7f44a9a864c P 7d1e5c1b047443cbb28d7b0a9744babf: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:29:23.648331 2225 raft_consensus.cc:2932] T 815178fbd07b4631b507927683a475c9 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER 16a97c1394f346f0b987f97cce7371b6 (127.15.255.137) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } }
I20240502 21:29:23.653980 6995 ts_tablet_manager.cc:902] T f8bee9bdcbff404280f000bdaeadf465 P 16b3a337bc5e462f83ba895ad8660deb: Initiating tablet copy from peer 7d1e5c1b047443cbb28d7b0a9744babf (127.15.255.129:37897)
I20240502 21:29:23.654106 6990 ts_tablet_manager.cc:902] T 20a7af42e83c4639bba20f22dee69f2a P 153026dd7f8943cc93fedf6f48450fb4: Initiating tablet copy from peer 7d1e5c1b047443cbb28d7b0a9744babf (127.15.255.129:37897)
I20240502 21:29:23.656785 6915 ts_tablet_manager.cc:1875] T aae7b5a01a0543afb1b5a7f44a9a864c P 153026dd7f8943cc93fedf6f48450fb4: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240502 21:29:23.670253 6933 tablet_copy_client.cc:308] T 444c1cba2a6f4729aa01b75d06b6b517 P a62a935ff43d494ca35d0c417943133d: tablet copy: Beginning tablet copy session from remote peer at address 127.15.255.133:37439
I20240502 21:29:23.670814 6974 consensus_queue.cc:1035] T 815178fbd07b4631b507927683a475c9 P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240502 21:29:23.673825 2087 raft_consensus.cc:1261] T 3c88daaac3734d918955a774829532c6 P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Refusing update from remote peer a62a935ff43d494ca35d0c417943133d: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:29:23.674404 6873 consensus_queue.cc:1035] T 3c88daaac3734d918955a774829532c6 P a62a935ff43d494ca35d0c417943133d [LEADER]: Connected to new peer: Peer: permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:29:23.676323 4119 raft_consensus.cc:2932] T 815178fbd07b4631b507927683a475c9 P 1fab5794cf1740288abd82f2cebb9b1a [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER 16a97c1394f346f0b987f97cce7371b6 (127.15.255.137) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } }
I20240502 21:29:23.677433 4121 raft_consensus.cc:1261] T 3c88daaac3734d918955a774829532c6 P 1fab5794cf1740288abd82f2cebb9b1a [term 1 LEARNER]: Refusing update from remote peer a62a935ff43d494ca35d0c417943133d: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:29:23.679881 6873 consensus_queue.cc:1035] T 3c88daaac3734d918955a774829532c6 P a62a935ff43d494ca35d0c417943133d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:29:23.686650 1975 tablet_copy_service.cc:140] P e632c553ff4341328acf1cbaca678392: Received BeginTabletCopySession request for tablet 444c1cba2a6f4729aa01b75d06b6b517 from peer a62a935ff43d494ca35d0c417943133d ({username='slave'} at 127.15.255.130:36903)
I20240502 21:29:23.686807 1975 tablet_copy_service.cc:161] P e632c553ff4341328acf1cbaca678392: Beginning new tablet copy session on tablet 444c1cba2a6f4729aa01b75d06b6b517 from peer a62a935ff43d494ca35d0c417943133d at {username='slave'} at 127.15.255.130:36903: session id = a62a935ff43d494ca35d0c417943133d-444c1cba2a6f4729aa01b75d06b6b517
I20240502 21:29:23.689790 1975 tablet_copy_source_session.cc:205] T 444c1cba2a6f4729aa01b75d06b6b517 P e632c553ff4341328acf1cbaca678392: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:29:23.692260 3968 tablet_service.cc:1514] Processing DeleteTablet for tablet 815178fbd07b4631b507927683a475c9 with delete_type TABLET_DATA_TOMBSTONED (TS 16a97c1394f346f0b987f97cce7371b6 not found in new config with opid_index 13) from {username='slave'} at 127.0.0.1:46236
I20240502 21:29:23.672201 6963 tablet_copy_client.cc:308] T e45af71e3fc14c579b32ba4dbe2adf78 P a62a935ff43d494ca35d0c417943133d: tablet copy: Beginning tablet copy session from remote peer at address 127.15.255.134:40821
I20240502 21:29:23.693285 2113 tablet_copy_service.cc:140] P 521bb93035324cecbebed2f6f2a630f8: Received BeginTabletCopySession request for tablet e45af71e3fc14c579b32ba4dbe2adf78 from peer a62a935ff43d494ca35d0c417943133d ({username='slave'} at 127.15.255.130:60331)
I20240502 21:29:23.693331 2225 raft_consensus.cc:1261] T 3c88daaac3734d918955a774829532c6 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Refusing update from remote peer a62a935ff43d494ca35d0c417943133d: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:29:23.693450 2113 tablet_copy_service.cc:161] P 521bb93035324cecbebed2f6f2a630f8: Beginning new tablet copy session on tablet e45af71e3fc14c579b32ba4dbe2adf78 from peer a62a935ff43d494ca35d0c417943133d at {username='slave'} at 127.15.255.130:60331: session id = a62a935ff43d494ca35d0c417943133d-e45af71e3fc14c579b32ba4dbe2adf78
I20240502 21:29:23.693799 6873 raft_consensus.cc:2932] T 3c88daaac3734d918955a774829532c6 P a62a935ff43d494ca35d0c417943133d [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false replace: true } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } }
I20240502 21:29:23.695304 6950 consensus_queue.cc:1035] T 3c88daaac3734d918955a774829532c6 P a62a935ff43d494ca35d0c417943133d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:29:23.696310 2113 tablet_copy_source_session.cc:205] T e45af71e3fc14c579b32ba4dbe2adf78 P 521bb93035324cecbebed2f6f2a630f8: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:29:23.698137 6963 ts_tablet_manager.cc:1875] T e45af71e3fc14c579b32ba4dbe2adf78 P a62a935ff43d494ca35d0c417943133d: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240502 21:29:23.699036 4121 raft_consensus.cc:2932] T 3c88daaac3734d918955a774829532c6 P 1fab5794cf1740288abd82f2cebb9b1a [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false replace: true } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } }
I20240502 21:29:23.700516 3854 raft_consensus.cc:2932] T 3c88daaac3734d918955a774829532c6 P d8d7297186694f77bb98de9174e4279c [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false replace: true } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } }
I20240502 21:29:23.708731 1685 raft_consensus.cc:2932] T 3c88daaac3734d918955a774829532c6 P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false replace: true } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } }
I20240502 21:29:23.709952 6963 ts_tablet_manager.cc:1888] T e45af71e3fc14c579b32ba4dbe2adf78 P a62a935ff43d494ca35d0c417943133d: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.9
I20240502 21:29:23.709667 1263 catalog_manager.cc:5484] T 3c88daaac3734d918955a774829532c6 P 1fab5794cf1740288abd82f2cebb9b1a reported cstate change: config changed from index 8 to 9, 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a62a935ff43d494ca35d0c417943133d" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false replace: true } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } }
I20240502 21:29:23.714764 2225 raft_consensus.cc:2932] T 3c88daaac3734d918955a774829532c6 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false replace: true } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } }
I20240502 21:29:23.717504 2086 raft_consensus.cc:2932] T 3c88daaac3734d918955a774829532c6 P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false replace: true } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } }
I20240502 21:29:23.717972 1412 raft_consensus.cc:2932] T 3c88daaac3734d918955a774829532c6 P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false replace: true } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } }
I20240502 21:29:23.720099 7005 tablet_replica.cc:330] T 815178fbd07b4631b507927683a475c9 P 16a97c1394f346f0b987f97cce7371b6: stopping tablet replica
I20240502 21:29:23.724762 6933 ts_tablet_manager.cc:1875] T 444c1cba2a6f4729aa01b75d06b6b517 P a62a935ff43d494ca35d0c417943133d: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240502 21:29:23.725786 6963 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e45af71e3fc14c579b32ba4dbe2adf78. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:29:23.726994 7005 raft_consensus.cc:2229] T 815178fbd07b4631b507927683a475c9 P 16a97c1394f346f0b987f97cce7371b6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:29:23.731787 7005 raft_consensus.cc:2258] T 815178fbd07b4631b507927683a475c9 P 16a97c1394f346f0b987f97cce7371b6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:29:23.739606 7005 ts_tablet_manager.cc:1875] T 815178fbd07b4631b507927683a475c9 P 16a97c1394f346f0b987f97cce7371b6: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:29:23.745129 6995 tablet_copy_client.cc:308] T f8bee9bdcbff404280f000bdaeadf465 P 16b3a337bc5e462f83ba895ad8660deb: tablet copy: Beginning tablet copy session from remote peer at address 127.15.255.129:37897
I20240502 21:29:23.745887 6963 tablet_copy_client.cc:783] T e45af71e3fc14c579b32ba4dbe2adf78 P a62a935ff43d494ca35d0c417943133d: tablet copy: Starting download of 0 data blocks...
I20240502 21:29:23.745887 1434 tablet_copy_service.cc:140] P 7d1e5c1b047443cbb28d7b0a9744babf: Received BeginTabletCopySession request for tablet f8bee9bdcbff404280f000bdaeadf465 from peer 16b3a337bc5e462f83ba895ad8660deb ({username='slave'} at 127.15.255.131:43685)
I20240502 21:29:23.746071 1434 tablet_copy_service.cc:161] P 7d1e5c1b047443cbb28d7b0a9744babf: Beginning new tablet copy session on tablet f8bee9bdcbff404280f000bdaeadf465 from peer 16b3a337bc5e462f83ba895ad8660deb at {username='slave'} at 127.15.255.131:43685: session id = 16b3a337bc5e462f83ba895ad8660deb-f8bee9bdcbff404280f000bdaeadf465
I20240502 21:29:23.746292 6963 tablet_copy_client.cc:647] T e45af71e3fc14c579b32ba4dbe2adf78 P a62a935ff43d494ca35d0c417943133d: tablet copy: Starting download of 1 WAL segments...
I20240502 21:29:23.749312 1434 tablet_copy_source_session.cc:205] T f8bee9bdcbff404280f000bdaeadf465 P 7d1e5c1b047443cbb28d7b0a9744babf: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:29:23.749401 6963 tablet_copy_client.cc:523] T e45af71e3fc14c579b32ba4dbe2adf78 P a62a935ff43d494ca35d0c417943133d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:29:23.750541 6995 ts_tablet_manager.cc:1875] T f8bee9bdcbff404280f000bdaeadf465 P 16b3a337bc5e462f83ba895ad8660deb: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240502 21:29:23.752271 7005 ts_tablet_manager.cc:1888] T 815178fbd07b4631b507927683a475c9 P 16a97c1394f346f0b987f97cce7371b6: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.12
I20240502 21:29:23.752907 7005 log.cc:1196] T 815178fbd07b4631b507927683a475c9 P 16a97c1394f346f0b987f97cce7371b6: Deleting WAL directory at /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-8/wal/wals/815178fbd07b4631b507927683a475c9
I20240502 21:29:23.753863 1254 catalog_manager.cc:4830] TS 16a97c1394f346f0b987f97cce7371b6 (127.15.255.137:43717): tablet 815178fbd07b4631b507927683a475c9 (table rebalance_test_table_3 [id=a205513324dd4db5a5ee4ed1928aefa6]) successfully deleted
I20240502 21:29:23.755854 1817 raft_consensus.cc:1261] T 3c88daaac3734d918955a774829532c6 P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Refusing update from remote peer a62a935ff43d494ca35d0c417943133d: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:29:23.756665 6718 consensus_queue.cc:1035] T 3c88daaac3734d918955a774829532c6 P a62a935ff43d494ca35d0c417943133d [LEADER]: Connected to new peer: Peer: permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:29:23.758085 6963 tablet_bootstrap.cc:492] T e45af71e3fc14c579b32ba4dbe2adf78 P a62a935ff43d494ca35d0c417943133d: Bootstrap starting.
I20240502 21:29:23.762817 6995 ts_tablet_manager.cc:1888] T f8bee9bdcbff404280f000bdaeadf465 P 16b3a337bc5e462f83ba895ad8660deb: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.9
I20240502 21:29:23.769824 6933 ts_tablet_manager.cc:1888] T 444c1cba2a6f4729aa01b75d06b6b517 P a62a935ff43d494ca35d0c417943133d: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.9
I20240502 21:29:23.770754 6933 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 444c1cba2a6f4729aa01b75d06b6b517. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:29:23.771888 6995 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8bee9bdcbff404280f000bdaeadf465. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:29:23.774518 6976 ts_tablet_manager.cc:1888] T c46e36b0bceb45b4bafbf1ffaf8e039e P 1fab5794cf1740288abd82f2cebb9b1a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240502 21:29:23.775133 6976 log.cc:1196] T c46e36b0bceb45b4bafbf1ffaf8e039e P 1fab5794cf1740288abd82f2cebb9b1a: Deleting WAL directory at /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-9/wal/wals/c46e36b0bceb45b4bafbf1ffaf8e039e
I20240502 21:29:23.781342 6933 tablet_copy_client.cc:783] T 444c1cba2a6f4729aa01b75d06b6b517 P a62a935ff43d494ca35d0c417943133d: tablet copy: Starting download of 0 data blocks...
I20240502 21:29:23.784911 6933 tablet_copy_client.cc:647] T 444c1cba2a6f4729aa01b75d06b6b517 P a62a935ff43d494ca35d0c417943133d: tablet copy: Starting download of 1 WAL segments...
I20240502 21:29:23.786855 6933 tablet_copy_client.cc:523] T 444c1cba2a6f4729aa01b75d06b6b517 P a62a935ff43d494ca35d0c417943133d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:29:23.788205 6955 ts_tablet_manager.cc:1888] T 4555e51179344acda81670fc9019d968 P 153026dd7f8943cc93fedf6f48450fb4: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.6
I20240502 21:29:23.793560 1257 catalog_manager.cc:4830] TS 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138:33375): tablet c46e36b0bceb45b4bafbf1ffaf8e039e (table rebalance_test_table_4 [id=d523de924ba14cb493e559296c083a34]) successfully deleted
I20240502 21:29:23.796809 6933 tablet_bootstrap.cc:492] T 444c1cba2a6f4729aa01b75d06b6b517 P a62a935ff43d494ca35d0c417943133d: Bootstrap starting.
I20240502 21:29:23.797650 6995 tablet_copy_client.cc:783] T f8bee9bdcbff404280f000bdaeadf465 P 16b3a337bc5e462f83ba895ad8660deb: tablet copy: Starting download of 0 data blocks...
I20240502 21:29:23.797979 6995 tablet_copy_client.cc:647] T f8bee9bdcbff404280f000bdaeadf465 P 16b3a337bc5e462f83ba895ad8660deb: tablet copy: Starting download of 1 WAL segments...
I20240502 21:29:23.802364 6995 tablet_copy_client.cc:523] T f8bee9bdcbff404280f000bdaeadf465 P 16b3a337bc5e462f83ba895ad8660deb: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:29:23.811215 6915 ts_tablet_manager.cc:1888] T aae7b5a01a0543afb1b5a7f44a9a864c P 153026dd7f8943cc93fedf6f48450fb4: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.6
I20240502 21:29:23.811807 6955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4555e51179344acda81670fc9019d968. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:29:23.813890 1955 consensus_queue.cc:237] T 444c1cba2a6f4729aa01b75d06b6b517 P e632c553ff4341328acf1cbaca678392 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } attrs { replace: true } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: NON_VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { promote: true } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: NON_VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { promote: true } }
I20240502 21:29:23.822806 6995 tablet_bootstrap.cc:492] T f8bee9bdcbff404280f000bdaeadf465 P 16b3a337bc5e462f83ba895ad8660deb: Bootstrap starting.
I20240502 21:29:23.825323 6915 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aae7b5a01a0543afb1b5a7f44a9a864c. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 21:29:23.830068 1885 consensus_peers.cc:485] T 444c1cba2a6f4729aa01b75d06b6b517 P e632c553ff4341328acf1cbaca678392 -> Peer a62a935ff43d494ca35d0c417943133d (127.15.255.130:37991): Couldn't send request to peer a62a935ff43d494ca35d0c417943133d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:29:23.831132 3854 raft_consensus.cc:1261] T 444c1cba2a6f4729aa01b75d06b6b517 P d8d7297186694f77bb98de9174e4279c [term 1 FOLLOWER]: Refusing update from remote peer e632c553ff4341328acf1cbaca678392: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240502 21:29:23.831750 6750 consensus_queue.cc:1035] T 444c1cba2a6f4729aa01b75d06b6b517 P e632c553ff4341328acf1cbaca678392 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
W20240502 21:29:23.835958 1886 consensus_peers.cc:485] T 444c1cba2a6f4729aa01b75d06b6b517 P e632c553ff4341328acf1cbaca678392 -> Peer 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132:44947): Couldn't send request to peer 153026dd7f8943cc93fedf6f48450fb4. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:29:23.839740 1686 raft_consensus.cc:1261] T 444c1cba2a6f4729aa01b75d06b6b517 P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Refusing update from remote peer e632c553ff4341328acf1cbaca678392: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240502 21:29:23.840270 6750 consensus_queue.cc:1035] T 444c1cba2a6f4729aa01b75d06b6b517 P e632c553ff4341328acf1cbaca678392 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 21:29:23.845021 1412 raft_consensus.cc:1261] T 444c1cba2a6f4729aa01b75d06b6b517 P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Refusing update from remote peer e632c553ff4341328acf1cbaca678392: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240502 21:29:23.845618 6750 consensus_queue.cc:1035] T 444c1cba2a6f4729aa01b75d06b6b517 P e632c553ff4341328acf1cbaca678392 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 21:29:23.848886 4121 raft_consensus.cc:1261] T 444c1cba2a6f4729aa01b75d06b6b517 P 1fab5794cf1740288abd82f2cebb9b1a [term 1 FOLLOWER]: Refusing update from remote peer e632c553ff4341328acf1cbaca678392: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240502 21:29:23.849550 6864 consensus_queue.cc:1035] T 444c1cba2a6f4729aa01b75d06b6b517 P e632c553ff4341328acf1cbaca678392 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 21:29:23.851243 6915 tablet_copy_client.cc:783] T aae7b5a01a0543afb1b5a7f44a9a864c P 153026dd7f8943cc93fedf6f48450fb4: tablet copy: Starting download of 0 data blocks...
I20240502 21:29:23.851570 6915 tablet_copy_client.cc:647] T aae7b5a01a0543afb1b5a7f44a9a864c P 153026dd7f8943cc93fedf6f48450fb4: tablet copy: Starting download of 1 WAL segments...
I20240502 21:29:23.852212 6955 tablet_copy_client.cc:783] T 4555e51179344acda81670fc9019d968 P 153026dd7f8943cc93fedf6f48450fb4: tablet copy: Starting download of 0 data blocks...
I20240502 21:29:23.852483 6955 tablet_copy_client.cc:647] T 4555e51179344acda81670fc9019d968 P 153026dd7f8943cc93fedf6f48450fb4: tablet copy: Starting download of 1 WAL segments...
I20240502 21:29:23.853313 1817 raft_consensus.cc:2932] T 3c88daaac3734d918955a774829532c6 P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 1fab5794cf1740288abd82f2cebb9b1a (127.15.255.138) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false replace: true } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } }
I20240502 21:29:23.881970 6955 tablet_copy_client.cc:523] T 4555e51179344acda81670fc9019d968 P 153026dd7f8943cc93fedf6f48450fb4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:29:23.885938 6915 tablet_copy_client.cc:523] T aae7b5a01a0543afb1b5a7f44a9a864c P 153026dd7f8943cc93fedf6f48450fb4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:29:23.895880 2086 raft_consensus.cc:1261] T 444c1cba2a6f4729aa01b75d06b6b517 P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Refusing update from remote peer e632c553ff4341328acf1cbaca678392: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240502 21:29:23.896593 6864 consensus_queue.cc:1035] T 444c1cba2a6f4729aa01b75d06b6b517 P e632c553ff4341328acf1cbaca678392 [LEADER]: Connected to new peer: Peer: permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
W20240502 21:29:23.903152 1885 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.135:34701, user_credentials={real_user=slave}} blocked reactor thread for 46525.2us
I20240502 21:29:23.903978 2226 raft_consensus.cc:1261] T 444c1cba2a6f4729aa01b75d06b6b517 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Refusing update from remote peer e632c553ff4341328acf1cbaca678392: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240502 21:29:23.905110 6864 consensus_queue.cc:1035] T 444c1cba2a6f4729aa01b75d06b6b517 P e632c553ff4341328acf1cbaca678392 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 21:29:23.909044 6990 tablet_copy_client.cc:308] T 20a7af42e83c4639bba20f22dee69f2a P 153026dd7f8943cc93fedf6f48450fb4: tablet copy: Beginning tablet copy session from remote peer at address 127.15.255.129:37897
I20240502 21:29:23.909862 1434 tablet_copy_service.cc:140] P 7d1e5c1b047443cbb28d7b0a9744babf: Received BeginTabletCopySession request for tablet 20a7af42e83c4639bba20f22dee69f2a from peer 153026dd7f8943cc93fedf6f48450fb4 ({username='slave'} at 127.15.255.132:33345)
I20240502 21:29:23.910028 1434 tablet_copy_service.cc:161] P 7d1e5c1b047443cbb28d7b0a9744babf: Beginning new tablet copy session on tablet 20a7af42e83c4639bba20f22dee69f2a from peer 153026dd7f8943cc93fedf6f48450fb4 at {username='slave'} at 127.15.255.132:33345: session id = 153026dd7f8943cc93fedf6f48450fb4-20a7af42e83c4639bba20f22dee69f2a
I20240502 21:29:23.913103 1434 tablet_copy_source_session.cc:205] T 20a7af42e83c4639bba20f22dee69f2a P 7d1e5c1b047443cbb28d7b0a9744babf: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:29:23.914383 6990 ts_tablet_manager.cc:1875] T 20a7af42e83c4639bba20f22dee69f2a P 153026dd7f8943cc93fedf6f48450fb4: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240502 21:29:23.919291 3854 raft_consensus.cc:2932] T 444c1cba2a6f4729aa01b75d06b6b517 P d8d7297186694f77bb98de9174e4279c [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, NON_VOTER 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } attrs { replace: true } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: NON_VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { promote: true } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: NON_VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { promote: true } } }
I20240502 21:29:23.922974 6990 ts_tablet_manager.cc:1888] T 20a7af42e83c4639bba20f22dee69f2a P 153026dd7f8943cc93fedf6f48450fb4: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.6
I20240502 21:29:23.923295 6990 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20a7af42e83c4639bba20f22dee69f2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:29:23.926857 6990 tablet_copy_client.cc:783] T 20a7af42e83c4639bba20f22dee69f2a P 153026dd7f8943cc93fedf6f48450fb4: tablet copy: Starting download of 0 data blocks...
I20240502 21:29:23.927168 6990 tablet_copy_client.cc:647] T 20a7af42e83c4639bba20f22dee69f2a P 153026dd7f8943cc93fedf6f48450fb4: tablet copy: Starting download of 1 WAL segments...
I20240502 21:29:23.927587 6915 tablet_bootstrap.cc:492] T aae7b5a01a0543afb1b5a7f44a9a864c P 153026dd7f8943cc93fedf6f48450fb4: Bootstrap starting.
I20240502 21:29:23.929304 6990 tablet_copy_client.cc:523] T 20a7af42e83c4639bba20f22dee69f2a P 153026dd7f8943cc93fedf6f48450fb4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:29:23.932891 6955 tablet_bootstrap.cc:492] T 4555e51179344acda81670fc9019d968 P 153026dd7f8943cc93fedf6f48450fb4: Bootstrap starting.
I20240502 21:29:23.935146 6990 tablet_bootstrap.cc:492] T 20a7af42e83c4639bba20f22dee69f2a P 153026dd7f8943cc93fedf6f48450fb4: Bootstrap starting.
I20240502 21:29:23.935190 2226 raft_consensus.cc:2932] T 444c1cba2a6f4729aa01b75d06b6b517 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, NON_VOTER 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } attrs { replace: true } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: NON_VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { promote: true } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: NON_VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { promote: true } } }
I20240502 21:29:23.937534 4121 raft_consensus.cc:2932] T 444c1cba2a6f4729aa01b75d06b6b517 P 1fab5794cf1740288abd82f2cebb9b1a [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, NON_VOTER 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } attrs { replace: true } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: NON_VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { promote: true } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: NON_VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { promote: true } } }
I20240502 21:29:23.942325 1262 catalog_manager.cc:5484] T 444c1cba2a6f4729aa01b75d06b6b517 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 reported cstate change: config changed from index 11 to 12, NON_VOTER 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132) added. New cstate: current_term: 1 leader_uuid: "e632c553ff4341328acf1cbaca678392" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } attrs { replace: true } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: NON_VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { promote: true } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: NON_VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { promote: true } } }
I20240502 21:29:23.953262 2086 raft_consensus.cc:2932] T 444c1cba2a6f4729aa01b75d06b6b517 P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, NON_VOTER 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } attrs { replace: true } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: NON_VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { promote: true } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: NON_VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { promote: true } } }
I20240502 21:29:23.889568 7032 raft_consensus.cc:2932] T 444c1cba2a6f4729aa01b75d06b6b517 P e632c553ff4341328acf1cbaca678392 [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, NON_VOTER 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } attrs { replace: true } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: NON_VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { promote: true } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: NON_VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { promote: true } } }
I20240502 21:29:23.995985 1686 raft_consensus.cc:2932] T 444c1cba2a6f4729aa01b75d06b6b517 P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, NON_VOTER 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } attrs { replace: true } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: NON_VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { promote: true } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: NON_VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { promote: true } } }
I20240502 21:29:23.997256 1412 raft_consensus.cc:2932] T 444c1cba2a6f4729aa01b75d06b6b517 P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, NON_VOTER 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } attrs { replace: true } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: NON_VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { promote: true } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: NON_VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { promote: true } } }
I20240502 21:29:24.063910 6963 tablet_bootstrap.cc:492] T e45af71e3fc14c579b32ba4dbe2adf78 P a62a935ff43d494ca35d0c417943133d: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:29:24.064397 6963 tablet_bootstrap.cc:492] T e45af71e3fc14c579b32ba4dbe2adf78 P a62a935ff43d494ca35d0c417943133d: Bootstrap complete.
I20240502 21:29:24.067835 6963 ts_tablet_manager.cc:1367] T e45af71e3fc14c579b32ba4dbe2adf78 P a62a935ff43d494ca35d0c417943133d: Time spent bootstrapping tablet: real 0.310s user 0.099s sys 0.042s
I20240502 21:29:24.069170 6963 raft_consensus.cc:348] T e45af71e3fc14c579b32ba4dbe2adf78 P a62a935ff43d494ca35d0c417943133d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: NON_VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { promote: true } }
I20240502 21:29:24.069540 6963 raft_consensus.cc:729] T e45af71e3fc14c579b32ba4dbe2adf78 P a62a935ff43d494ca35d0c417943133d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a62a935ff43d494ca35d0c417943133d, State: Initialized, Role: LEARNER
I20240502 21:29:24.069775 6963 consensus_queue.cc:260] T e45af71e3fc14c579b32ba4dbe2adf78 P a62a935ff43d494ca35d0c417943133d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 1.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: NON_VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { promote: true } }
I20240502 21:29:24.070533 6963 ts_tablet_manager.cc:1398] T e45af71e3fc14c579b32ba4dbe2adf78 P a62a935ff43d494ca35d0c417943133d: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240502 21:29:24.081199 2113 tablet_copy_service.cc:342] P 521bb93035324cecbebed2f6f2a630f8: Request end of tablet copy session a62a935ff43d494ca35d0c417943133d-e45af71e3fc14c579b32ba4dbe2adf78 received from {username='slave'} at 127.15.255.130:60331
I20240502 21:29:24.081351 2113 tablet_copy_service.cc:433] P 521bb93035324cecbebed2f6f2a630f8: ending tablet copy session a62a935ff43d494ca35d0c417943133d-e45af71e3fc14c579b32ba4dbe2adf78 on tablet e45af71e3fc14c579b32ba4dbe2adf78 with peer a62a935ff43d494ca35d0c417943133d
I20240502 21:29:24.113729 1955 consensus_queue.cc:237] T 0658ecbf32754656876452b6f021926b P e632c553ff4341328acf1cbaca678392 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } attrs { replace: true } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: NON_VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { promote: true } }
W20240502 21:29:24.120218 1886 consensus_peers.cc:485] T 0658ecbf32754656876452b6f021926b P e632c553ff4341328acf1cbaca678392 -> Peer 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132:44947): Couldn't send request to peer 153026dd7f8943cc93fedf6f48450fb4. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240502 21:29:24.122386 1686 raft_consensus.cc:1261] T 0658ecbf32754656876452b6f021926b P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Refusing update from remote peer e632c553ff4341328acf1cbaca678392: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:29:24.122951 7032 consensus_queue.cc:1035] T 0658ecbf32754656876452b6f021926b P e632c553ff4341328acf1cbaca678392 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:29:24.123438 6933 tablet_bootstrap.cc:492] T 444c1cba2a6f4729aa01b75d06b6b517 P a62a935ff43d494ca35d0c417943133d: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:29:24.123862 6933 tablet_bootstrap.cc:492] T 444c1cba2a6f4729aa01b75d06b6b517 P a62a935ff43d494ca35d0c417943133d: Bootstrap complete.
I20240502 21:29:24.124154 6933 ts_tablet_manager.cc:1367] T 444c1cba2a6f4729aa01b75d06b6b517 P a62a935ff43d494ca35d0c417943133d: Time spent bootstrapping tablet: real 0.327s user 0.037s sys 0.014s
I20240502 21:29:24.124891 6933 raft_consensus.cc:348] T 444c1cba2a6f4729aa01b75d06b6b517 P a62a935ff43d494ca35d0c417943133d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: NON_VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { promote: true } }
I20240502 21:29:24.125226 6933 raft_consensus.cc:729] T 444c1cba2a6f4729aa01b75d06b6b517 P a62a935ff43d494ca35d0c417943133d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a62a935ff43d494ca35d0c417943133d, State: Initialized, Role: LEARNER
I20240502 21:29:24.125429 6933 consensus_queue.cc:260] T 444c1cba2a6f4729aa01b75d06b6b517 P a62a935ff43d494ca35d0c417943133d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 1.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: NON_VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { promote: true } }
I20240502 21:29:24.126821 6933 ts_tablet_manager.cc:1398] T 444c1cba2a6f4729aa01b75d06b6b517 P a62a935ff43d494ca35d0c417943133d: Time spent starting tablet: real 0.003s user 0.005s sys 0.000s
I20240502 21:29:24.127408 1975 tablet_copy_service.cc:342] P e632c553ff4341328acf1cbaca678392: Request end of tablet copy session a62a935ff43d494ca35d0c417943133d-444c1cba2a6f4729aa01b75d06b6b517 received from {username='slave'} at 127.15.255.130:36903
I20240502 21:29:24.127511 1975 tablet_copy_service.cc:433] P e632c553ff4341328acf1cbaca678392: ending tablet copy session a62a935ff43d494ca35d0c417943133d-444c1cba2a6f4729aa01b75d06b6b517 on tablet 444c1cba2a6f4729aa01b75d06b6b517 with peer a62a935ff43d494ca35d0c417943133d
I20240502 21:29:24.135466 1412 raft_consensus.cc:1261] T 0658ecbf32754656876452b6f021926b P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Refusing update from remote peer e632c553ff4341328acf1cbaca678392: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:29:24.135810 1547 raft_consensus.cc:1261] T 0658ecbf32754656876452b6f021926b P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Refusing update from remote peer e632c553ff4341328acf1cbaca678392: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:29:24.136353 7032 consensus_queue.cc:1035] T 0658ecbf32754656876452b6f021926b P e632c553ff4341328acf1cbaca678392 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:29:24.136834 7033 consensus_queue.cc:1035] T 0658ecbf32754656876452b6f021926b P e632c553ff4341328acf1cbaca678392 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:29:24.143908 4121 raft_consensus.cc:1261] T 0658ecbf32754656876452b6f021926b P 1fab5794cf1740288abd82f2cebb9b1a [term 1 FOLLOWER]: Refusing update from remote peer e632c553ff4341328acf1cbaca678392: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:29:24.145068 2086 raft_consensus.cc:1261] T 0658ecbf32754656876452b6f021926b P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Refusing update from remote peer e632c553ff4341328acf1cbaca678392: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:29:24.145807 6750 consensus_queue.cc:1035] T 0658ecbf32754656876452b6f021926b P e632c553ff4341328acf1cbaca678392 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:29:24.146337 6750 consensus_queue.cc:1035] T 0658ecbf32754656876452b6f021926b P e632c553ff4341328acf1cbaca678392 [LEADER]: Connected to new peer: Peer: permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:29:24.147392 2226 raft_consensus.cc:1261] T 0658ecbf32754656876452b6f021926b P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Refusing update from remote peer e632c553ff4341328acf1cbaca678392: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 21:29:24.150216 6750 consensus_queue.cc:1035] T 0658ecbf32754656876452b6f021926b P e632c553ff4341328acf1cbaca678392 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 21:29:24.156551 7032 raft_consensus.cc:2932] T 0658ecbf32754656876452b6f021926b P e632c553ff4341328acf1cbaca678392 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } attrs { replace: true } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: NON_VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { promote: true } } }
I20240502 21:29:24.159354 2226 raft_consensus.cc:2932] T 0658ecbf32754656876452b6f021926b P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } attrs { replace: true } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: NON_VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { promote: true } } }
I20240502 21:29:24.165354 6995 tablet_bootstrap.cc:492] T f8bee9bdcbff404280f000bdaeadf465 P 16b3a337bc5e462f83ba895ad8660deb: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:29:24.165738 6995 tablet_bootstrap.cc:492] T f8bee9bdcbff404280f000bdaeadf465 P 16b3a337bc5e462f83ba895ad8660deb: Bootstrap complete.
I20240502 21:29:24.166033 6995 ts_tablet_manager.cc:1367] T f8bee9bdcbff404280f000bdaeadf465 P 16b3a337bc5e462f83ba895ad8660deb: Time spent bootstrapping tablet: real 0.343s user 0.060s sys 0.011s
I20240502 21:29:24.166828 6995 raft_consensus.cc:348] T f8bee9bdcbff404280f000bdaeadf465 P 16b3a337bc5e462f83ba895ad8660deb [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { replace: true } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: NON_VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { promote: true } }
I20240502 21:29:24.167189 6995 raft_consensus.cc:729] T f8bee9bdcbff404280f000bdaeadf465 P 16b3a337bc5e462f83ba895ad8660deb [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 16b3a337bc5e462f83ba895ad8660deb, State: Initialized, Role: LEARNER
I20240502 21:29:24.167414 6995 consensus_queue.cc:260] T f8bee9bdcbff404280f000bdaeadf465 P 16b3a337bc5e462f83ba895ad8660deb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 1.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { replace: true } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: NON_VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { promote: true } }
I20240502 21:29:24.169785 1547 consensus_queue.cc:237] T 3c88daaac3734d918955a774829532c6 P a62a935ff43d494ca35d0c417943133d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } }
I20240502 21:29:24.171248 1263 catalog_manager.cc:5484] T 0658ecbf32754656876452b6f021926b P 6ab8df3c796f4b6cb3e29d5045ad3ea0 reported cstate change: config changed from index 7 to 8, NON_VOTER 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132) added. New cstate: current_term: 1 leader_uuid: "e632c553ff4341328acf1cbaca678392" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } attrs { replace: true } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: NON_VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { promote: true } } }
I20240502 21:29:24.172951 1685 raft_consensus.cc:1261] T 3c88daaac3734d918955a774829532c6 P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Refusing update from remote peer a62a935ff43d494ca35d0c417943133d: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 21:29:24.175426 6950 consensus_queue.cc:1035] T 3c88daaac3734d918955a774829532c6 P a62a935ff43d494ca35d0c417943133d [LEADER]: Connected to new peer: Peer: permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 21:29:24.177477 1411 raft_consensus.cc:1261] T 3c88daaac3734d918955a774829532c6 P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Refusing update from remote peer a62a935ff43d494ca35d0c417943133d: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 21:29:24.177626 1683 raft_consensus.cc:2932] T 0658ecbf32754656876452b6f021926b P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } attrs { replace: true } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: NON_VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { promote: true } } }
I20240502 21:29:24.178555 6950 consensus_queue.cc:1035] T 3c88daaac3734d918955a774829532c6 P a62a935ff43d494ca35d0c417943133d [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 21:29:24.181252 6995 ts_tablet_manager.cc:1398] T f8bee9bdcbff404280f000bdaeadf465 P 16b3a337bc5e462f83ba895ad8660deb: Time spent starting tablet: real 0.015s user 0.003s sys 0.000s
I20240502 21:29:24.181917 1434 tablet_copy_service.cc:342] P 7d1e5c1b047443cbb28d7b0a9744babf: Request end of tablet copy session 16b3a337bc5e462f83ba895ad8660deb-f8bee9bdcbff404280f000bdaeadf465 received from {username='slave'} at 127.15.255.131:43685
I20240502 21:29:24.182041 1434 tablet_copy_service.cc:433] P 7d1e5c1b047443cbb28d7b0a9744babf: ending tablet copy session 16b3a337bc5e462f83ba895ad8660deb-f8bee9bdcbff404280f000bdaeadf465 on tablet f8bee9bdcbff404280f000bdaeadf465 with peer 16b3a337bc5e462f83ba895ad8660deb
I20240502 21:29:24.183194 4119 raft_consensus.cc:1261] T 3c88daaac3734d918955a774829532c6 P 1fab5794cf1740288abd82f2cebb9b1a [term 1 FOLLOWER]: Refusing update from remote peer a62a935ff43d494ca35d0c417943133d: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 21:29:24.183606 4121 raft_consensus.cc:2932] T 0658ecbf32754656876452b6f021926b P 1fab5794cf1740288abd82f2cebb9b1a [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } attrs { replace: true } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: NON_VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { promote: true } } }
I20240502 21:29:24.184198 6950 consensus_queue.cc:1035] T 3c88daaac3734d918955a774829532c6 P a62a935ff43d494ca35d0c417943133d [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 21:29:24.184464 1549 raft_consensus.cc:2932] T 0658ecbf32754656876452b6f021926b P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } attrs { replace: true } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: NON_VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { promote: true } } }
I20240502 21:29:24.186780 6718 raft_consensus.cc:2932] T 3c88daaac3734d918955a774829532c6 P a62a935ff43d494ca35d0c417943133d [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER d8d7297186694f77bb98de9174e4279c (127.15.255.136) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } }
I20240502 21:29:24.189406 2086 raft_consensus.cc:1261] T 3c88daaac3734d918955a774829532c6 P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Refusing update from remote peer a62a935ff43d494ca35d0c417943133d: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 21:29:24.191565 2226 raft_consensus.cc:1261] T 3c88daaac3734d918955a774829532c6 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Refusing update from remote peer a62a935ff43d494ca35d0c417943133d: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 21:29:24.192113 6878 consensus_queue.cc:1035] T 3c88daaac3734d918955a774829532c6 P a62a935ff43d494ca35d0c417943133d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 21:29:24.193748 2225 raft_consensus.cc:2932] T 3c88daaac3734d918955a774829532c6 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER d8d7297186694f77bb98de9174e4279c (127.15.255.136) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } }
I20240502 21:29:24.194465 1255 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 3c88daaac3734d918955a774829532c6 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:29:24.194309 4121 raft_consensus.cc:2932] T 3c88daaac3734d918955a774829532c6 P 1fab5794cf1740288abd82f2cebb9b1a [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER d8d7297186694f77bb98de9174e4279c (127.15.255.136) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } }
I20240502 21:29:24.195436 6950 consensus_queue.cc:1035] T 3c88daaac3734d918955a774829532c6 P a62a935ff43d494ca35d0c417943133d [LEADER]: Connected to new peer: Peer: permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 21:29:24.197206 2086 raft_consensus.cc:2932] T 3c88daaac3734d918955a774829532c6 P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER d8d7297186694f77bb98de9174e4279c (127.15.255.136) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } }
I20240502 21:29:24.198006 6915 tablet_bootstrap.cc:492] T aae7b5a01a0543afb1b5a7f44a9a864c P 153026dd7f8943cc93fedf6f48450fb4: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:29:24.198480 6915 tablet_bootstrap.cc:492] T aae7b5a01a0543afb1b5a7f44a9a864c P 153026dd7f8943cc93fedf6f48450fb4: Bootstrap complete.
I20240502 21:29:24.198786 6915 ts_tablet_manager.cc:1367] T aae7b5a01a0543afb1b5a7f44a9a864c P 153026dd7f8943cc93fedf6f48450fb4: Time spent bootstrapping tablet: real 0.271s user 0.056s sys 0.001s
I20240502 21:29:24.199636 6915 raft_consensus.cc:348] T aae7b5a01a0543afb1b5a7f44a9a864c P 153026dd7f8943cc93fedf6f48450fb4 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } attrs { replace: true } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: NON_VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { promote: true } }
I20240502 21:29:24.200001 6915 raft_consensus.cc:729] T aae7b5a01a0543afb1b5a7f44a9a864c P 153026dd7f8943cc93fedf6f48450fb4 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 153026dd7f8943cc93fedf6f48450fb4, State: Initialized, Role: LEARNER
I20240502 21:29:24.200223 6915 consensus_queue.cc:260] T aae7b5a01a0543afb1b5a7f44a9a864c P 153026dd7f8943cc93fedf6f48450fb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 1.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } attrs { replace: true } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: NON_VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { promote: true } }
I20240502 21:29:24.200956 6915 ts_tablet_manager.cc:1398] T aae7b5a01a0543afb1b5a7f44a9a864c P 153026dd7f8943cc93fedf6f48450fb4: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:29:24.202206 1434 tablet_copy_service.cc:342] P 7d1e5c1b047443cbb28d7b0a9744babf: Request end of tablet copy session 153026dd7f8943cc93fedf6f48450fb4-aae7b5a01a0543afb1b5a7f44a9a864c received from {username='slave'} at 127.15.255.132:33345
I20240502 21:29:24.202373 1434 tablet_copy_service.cc:433] P 7d1e5c1b047443cbb28d7b0a9744babf: ending tablet copy session 153026dd7f8943cc93fedf6f48450fb4-aae7b5a01a0543afb1b5a7f44a9a864c on tablet aae7b5a01a0543afb1b5a7f44a9a864c with peer 153026dd7f8943cc93fedf6f48450fb4
I20240502 21:29:24.203795 1270 catalog_manager.cc:5484] T 3c88daaac3734d918955a774829532c6 P 521bb93035324cecbebed2f6f2a630f8 reported cstate change: config changed from index 9 to 10, VOTER d8d7297186694f77bb98de9174e4279c (127.15.255.136) evicted. New cstate: current_term: 1 leader_uuid: "a62a935ff43d494ca35d0c417943133d" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } }
I20240502 21:29:24.210922 1817 raft_consensus.cc:1261] T 3c88daaac3734d918955a774829532c6 P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Refusing update from remote peer a62a935ff43d494ca35d0c417943133d: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 21:29:24.211107 1411 raft_consensus.cc:2932] T 3c88daaac3734d918955a774829532c6 P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER d8d7297186694f77bb98de9174e4279c (127.15.255.136) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } }
I20240502 21:29:24.216938 6878 consensus_queue.cc:1035] T 3c88daaac3734d918955a774829532c6 P a62a935ff43d494ca35d0c417943133d [LEADER]: Connected to new peer: Peer: permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 21:29:24.218673 1817 raft_consensus.cc:2932] T 3c88daaac3734d918955a774829532c6 P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER d8d7297186694f77bb98de9174e4279c (127.15.255.136) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } }
I20240502 21:29:24.219534 2086 raft_consensus.cc:2932] T 0658ecbf32754656876452b6f021926b P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } attrs { replace: true } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: NON_VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { promote: true } } }
I20240502 21:29:24.225205 1683 raft_consensus.cc:2932] T 3c88daaac3734d918955a774829532c6 P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER d8d7297186694f77bb98de9174e4279c (127.15.255.136) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } }
I20240502 21:29:24.261875 1412 raft_consensus.cc:2932] T 0658ecbf32754656876452b6f021926b P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } attrs { replace: true } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: NON_VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { promote: true } } }
I20240502 21:29:24.265230 6915 ts_tablet_manager.cc:902] T 444c1cba2a6f4729aa01b75d06b6b517 P 153026dd7f8943cc93fedf6f48450fb4: Initiating tablet copy from peer e632c553ff4341328acf1cbaca678392 (127.15.255.133:37439)
I20240502 21:29:24.305550 6955 tablet_bootstrap.cc:492] T 4555e51179344acda81670fc9019d968 P 153026dd7f8943cc93fedf6f48450fb4: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:29:24.307587 3836 tablet_service.cc:1514] Processing DeleteTablet for tablet 3c88daaac3734d918955a774829532c6 with delete_type TABLET_DATA_TOMBSTONED (TS d8d7297186694f77bb98de9174e4279c not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:56586
I20240502 21:29:24.314697 6955 tablet_bootstrap.cc:492] T 4555e51179344acda81670fc9019d968 P 153026dd7f8943cc93fedf6f48450fb4: Bootstrap complete.
I20240502 21:29:24.314999 6955 ts_tablet_manager.cc:1367] T 4555e51179344acda81670fc9019d968 P 153026dd7f8943cc93fedf6f48450fb4: Time spent bootstrapping tablet: real 0.382s user 0.048s sys 0.004s
I20240502 21:29:24.314204 6990 tablet_bootstrap.cc:492] T 20a7af42e83c4639bba20f22dee69f2a P 153026dd7f8943cc93fedf6f48450fb4: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:29:24.315961 6990 tablet_bootstrap.cc:492] T 20a7af42e83c4639bba20f22dee69f2a P 153026dd7f8943cc93fedf6f48450fb4: Bootstrap complete.
I20240502 21:29:24.316296 6990 ts_tablet_manager.cc:1367] T 20a7af42e83c4639bba20f22dee69f2a P 153026dd7f8943cc93fedf6f48450fb4: Time spent bootstrapping tablet: real 0.381s user 0.051s sys 0.016s
I20240502 21:29:24.317420 6955 raft_consensus.cc:348] T 4555e51179344acda81670fc9019d968 P 153026dd7f8943cc93fedf6f48450fb4 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } attrs { replace: true } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: NON_VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { promote: true } }
I20240502 21:29:24.317746 6955 raft_consensus.cc:729] T 4555e51179344acda81670fc9019d968 P 153026dd7f8943cc93fedf6f48450fb4 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 153026dd7f8943cc93fedf6f48450fb4, State: Initialized, Role: LEARNER
I20240502 21:29:24.317974 6955 consensus_queue.cc:260] T 4555e51179344acda81670fc9019d968 P 153026dd7f8943cc93fedf6f48450fb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 1.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } attrs { replace: true } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: NON_VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { promote: true } }
I20240502 21:29:24.319375 6915 tablet_copy_client.cc:308] T 444c1cba2a6f4729aa01b75d06b6b517 P 153026dd7f8943cc93fedf6f48450fb4: tablet copy: Beginning tablet copy session from remote peer at address 127.15.255.133:37439
I20240502 21:29:24.319491 6990 raft_consensus.cc:348] T 20a7af42e83c4639bba20f22dee69f2a P 153026dd7f8943cc93fedf6f48450fb4 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } attrs { replace: true } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: NON_VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { promote: true } }
I20240502 21:29:24.319803 6990 raft_consensus.cc:729] T 20a7af42e83c4639bba20f22dee69f2a P 153026dd7f8943cc93fedf6f48450fb4 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 153026dd7f8943cc93fedf6f48450fb4, State: Initialized, Role: LEARNER
I20240502 21:29:24.319993 6990 consensus_queue.cc:260] T 20a7af42e83c4639bba20f22dee69f2a P 153026dd7f8943cc93fedf6f48450fb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } attrs { replace: true } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: NON_VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { promote: true } }
I20240502 21:29:24.320667 6990 ts_tablet_manager.cc:1398] T 20a7af42e83c4639bba20f22dee69f2a P 153026dd7f8943cc93fedf6f48450fb4: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 21:29:24.322157 1434 tablet_copy_service.cc:342] P 7d1e5c1b047443cbb28d7b0a9744babf: Request end of tablet copy session 153026dd7f8943cc93fedf6f48450fb4-20a7af42e83c4639bba20f22dee69f2a received from {username='slave'} at 127.15.255.132:33345
I20240502 21:29:24.322314 1434 tablet_copy_service.cc:433] P 7d1e5c1b047443cbb28d7b0a9744babf: ending tablet copy session 153026dd7f8943cc93fedf6f48450fb4-20a7af42e83c4639bba20f22dee69f2a on tablet 20a7af42e83c4639bba20f22dee69f2a with peer 153026dd7f8943cc93fedf6f48450fb4
I20240502 21:29:24.325845 6955 ts_tablet_manager.cc:1398] T 4555e51179344acda81670fc9019d968 P 153026dd7f8943cc93fedf6f48450fb4: Time spent starting tablet: real 0.011s user 0.003s sys 0.000s
I20240502 21:29:24.329383 2249 tablet_copy_service.cc:342] P 6ab8df3c796f4b6cb3e29d5045ad3ea0: Request end of tablet copy session 153026dd7f8943cc93fedf6f48450fb4-4555e51179344acda81670fc9019d968 received from {username='slave'} at 127.15.255.132:47173
I20240502 21:29:24.329526 2249 tablet_copy_service.cc:433] P 6ab8df3c796f4b6cb3e29d5045ad3ea0: ending tablet copy session 153026dd7f8943cc93fedf6f48450fb4-4555e51179344acda81670fc9019d968 on tablet 4555e51179344acda81670fc9019d968 with peer 153026dd7f8943cc93fedf6f48450fb4
I20240502 21:29:24.330206 1975 tablet_copy_service.cc:140] P e632c553ff4341328acf1cbaca678392: Received BeginTabletCopySession request for tablet 444c1cba2a6f4729aa01b75d06b6b517 from peer 153026dd7f8943cc93fedf6f48450fb4 ({username='slave'} at 127.15.255.132:49633)
I20240502 21:29:24.330381 1975 tablet_copy_service.cc:161] P e632c553ff4341328acf1cbaca678392: Beginning new tablet copy session on tablet 444c1cba2a6f4729aa01b75d06b6b517 from peer 153026dd7f8943cc93fedf6f48450fb4 at {username='slave'} at 127.15.255.132:49633: session id = 153026dd7f8943cc93fedf6f48450fb4-444c1cba2a6f4729aa01b75d06b6b517
I20240502 21:29:24.331063 1817 raft_consensus.cc:1203] T 4555e51179344acda81670fc9019d968 P 153026dd7f8943cc93fedf6f48450fb4 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.10->[1.11-1.11] Dedup: 1.11->[]
I20240502 21:29:24.336760 1975 tablet_copy_source_session.cc:205] T 444c1cba2a6f4729aa01b75d06b6b517 P e632c553ff4341328acf1cbaca678392: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:29:24.337841 6915 ts_tablet_manager.cc:1875] T 444c1cba2a6f4729aa01b75d06b6b517 P 153026dd7f8943cc93fedf6f48450fb4: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240502 21:29:24.341220 7105 tablet_replica.cc:330] T 3c88daaac3734d918955a774829532c6 P d8d7297186694f77bb98de9174e4279c: stopping tablet replica
I20240502 21:29:24.341815 7105 raft_consensus.cc:2229] T 3c88daaac3734d918955a774829532c6 P d8d7297186694f77bb98de9174e4279c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:29:24.342283 7105 raft_consensus.cc:2258] T 3c88daaac3734d918955a774829532c6 P d8d7297186694f77bb98de9174e4279c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:29:24.343827 7105 ts_tablet_manager.cc:1875] T 3c88daaac3734d918955a774829532c6 P d8d7297186694f77bb98de9174e4279c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:29:24.344719 6915 ts_tablet_manager.cc:1888] T 444c1cba2a6f4729aa01b75d06b6b517 P 153026dd7f8943cc93fedf6f48450fb4: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.6
I20240502 21:29:24.344970 6915 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 444c1cba2a6f4729aa01b75d06b6b517. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:29:24.350582 7105 ts_tablet_manager.cc:1888] T 3c88daaac3734d918955a774829532c6 P d8d7297186694f77bb98de9174e4279c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240502 21:29:24.350847 7105 log.cc:1196] T 3c88daaac3734d918955a774829532c6 P d8d7297186694f77bb98de9174e4279c: Deleting WAL directory at /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-7/wal/wals/3c88daaac3734d918955a774829532c6
I20240502 21:29:24.351871 1256 catalog_manager.cc:4830] TS d8d7297186694f77bb98de9174e4279c (127.15.255.136:41459): tablet 3c88daaac3734d918955a774829532c6 (table rebalance_test_table_0 [id=82737b0de4b945dc8ca0c7d6b92af37c]) successfully deleted
I20240502 21:29:24.353751 6915 tablet_copy_client.cc:783] T 444c1cba2a6f4729aa01b75d06b6b517 P 153026dd7f8943cc93fedf6f48450fb4: tablet copy: Starting download of 0 data blocks...
I20240502 21:29:24.354226 6915 tablet_copy_client.cc:647] T 444c1cba2a6f4729aa01b75d06b6b517 P 153026dd7f8943cc93fedf6f48450fb4: tablet copy: Starting download of 1 WAL segments...
I20240502 21:29:24.356179 6915 tablet_copy_client.cc:523] T 444c1cba2a6f4729aa01b75d06b6b517 P 153026dd7f8943cc93fedf6f48450fb4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:29:24.363423 6915 tablet_bootstrap.cc:492] T 444c1cba2a6f4729aa01b75d06b6b517 P 153026dd7f8943cc93fedf6f48450fb4: Bootstrap starting.
I20240502 21:29:24.442196 1547 raft_consensus.cc:2932] T 444c1cba2a6f4729aa01b75d06b6b517 P a62a935ff43d494ca35d0c417943133d [term 1 LEARNER]: Committing config change with OpId 1.12: config changed from index 11 to 12, NON_VOTER 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } attrs { replace: true } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: NON_VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { promote: true } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: NON_VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { promote: true } } }
I20240502 21:29:24.451140 6915 tablet_bootstrap.cc:492] T 444c1cba2a6f4729aa01b75d06b6b517 P 153026dd7f8943cc93fedf6f48450fb4: Bootstrap replayed 1/1 log segments. Stats: ops{read=12 overwritten=0 applied=12 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:29:24.451596 6915 tablet_bootstrap.cc:492] T 444c1cba2a6f4729aa01b75d06b6b517 P 153026dd7f8943cc93fedf6f48450fb4: Bootstrap complete.
I20240502 21:29:24.451884 6915 ts_tablet_manager.cc:1367] T 444c1cba2a6f4729aa01b75d06b6b517 P 153026dd7f8943cc93fedf6f48450fb4: Time spent bootstrapping tablet: real 0.089s user 0.040s sys 0.019s
I20240502 21:29:24.452764 6915 raft_consensus.cc:348] T 444c1cba2a6f4729aa01b75d06b6b517 P 153026dd7f8943cc93fedf6f48450fb4 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } attrs { replace: true } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: NON_VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { promote: true } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: NON_VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { promote: true } }
I20240502 21:29:24.453104 6915 raft_consensus.cc:729] T 444c1cba2a6f4729aa01b75d06b6b517 P 153026dd7f8943cc93fedf6f48450fb4 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 153026dd7f8943cc93fedf6f48450fb4, State: Initialized, Role: LEARNER
I20240502 21:29:24.453404 6915 consensus_queue.cc:260] T 444c1cba2a6f4729aa01b75d06b6b517 P 153026dd7f8943cc93fedf6f48450fb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } attrs { replace: true } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: NON_VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { promote: true } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: NON_VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { promote: true } }
I20240502 21:29:24.454111 6915 ts_tablet_manager.cc:1398] T 444c1cba2a6f4729aa01b75d06b6b517 P 153026dd7f8943cc93fedf6f48450fb4: Time spent starting tablet: real 0.002s user 0.005s sys 0.000s
I20240502 21:29:24.454730 1975 tablet_copy_service.cc:342] P e632c553ff4341328acf1cbaca678392: Request end of tablet copy session 153026dd7f8943cc93fedf6f48450fb4-444c1cba2a6f4729aa01b75d06b6b517 received from {username='slave'} at 127.15.255.132:49633
I20240502 21:29:24.454841 1975 tablet_copy_service.cc:433] P e632c553ff4341328acf1cbaca678392: ending tablet copy session 153026dd7f8943cc93fedf6f48450fb4-444c1cba2a6f4729aa01b75d06b6b517 on tablet 444c1cba2a6f4729aa01b75d06b6b517 with peer 153026dd7f8943cc93fedf6f48450fb4
I20240502 21:29:24.490532 1547 raft_consensus.cc:1203] T e45af71e3fc14c579b32ba4dbe2adf78 P a62a935ff43d494ca35d0c417943133d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.10->[1.11-1.11] Dedup: 1.11->[]
I20240502 21:29:24.507792 1686 raft_consensus.cc:1203] T f8bee9bdcbff404280f000bdaeadf465 P 16b3a337bc5e462f83ba895ad8660deb [term 1 LEARNER]: Deduplicated request from leader. Original: 1.10->[1.11-1.11] Dedup: 1.11->[]
I20240502 21:29:24.566965 3968 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:24.578203 1798 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 21:29:24.579336 4101 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:24.590867 6915 ts_tablet_manager.cc:902] T 0658ecbf32754656876452b6f021926b P 153026dd7f8943cc93fedf6f48450fb4: Initiating tablet copy from peer e632c553ff4341328acf1cbaca678392 (127.15.255.133:37439)
I20240502 21:29:24.611796 2205 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:29:24.612399 1664 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:24.617548 2068 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 21:29:24.623306 1817 raft_consensus.cc:1203] T aae7b5a01a0543afb1b5a7f44a9a864c P 153026dd7f8943cc93fedf6f48450fb4 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.10->[1.11-1.11] Dedup: 1.11->[]
I20240502 21:29:24.640527 1527 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 21:29:24.655364 1394 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 21:29:24.671736 3836 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 21:29:24.692564 1931 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 21:29:24.705546 6915 tablet_copy_client.cc:308] T 0658ecbf32754656876452b6f021926b P 153026dd7f8943cc93fedf6f48450fb4: tablet copy: Beginning tablet copy session from remote peer at address 127.15.255.133:37439
I20240502 21:29:24.706184 1975 tablet_copy_service.cc:140] P e632c553ff4341328acf1cbaca678392: Received BeginTabletCopySession request for tablet 0658ecbf32754656876452b6f021926b from peer 153026dd7f8943cc93fedf6f48450fb4 ({username='slave'} at 127.15.255.132:49633)
I20240502 21:29:24.706349 1975 tablet_copy_service.cc:161] P e632c553ff4341328acf1cbaca678392: Beginning new tablet copy session on tablet 0658ecbf32754656876452b6f021926b from peer 153026dd7f8943cc93fedf6f48450fb4 at {username='slave'} at 127.15.255.132:49633: session id = 153026dd7f8943cc93fedf6f48450fb4-0658ecbf32754656876452b6f021926b
I20240502 21:29:24.710305 1975 tablet_copy_source_session.cc:205] T 0658ecbf32754656876452b6f021926b P e632c553ff4341328acf1cbaca678392: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 21:29:24.712347 6915 ts_tablet_manager.cc:1875] T 0658ecbf32754656876452b6f021926b P 153026dd7f8943cc93fedf6f48450fb4: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240502 21:29:24.719638 6915 ts_tablet_manager.cc:1888] T 0658ecbf32754656876452b6f021926b P 153026dd7f8943cc93fedf6f48450fb4: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.6
I20240502 21:29:24.732193 6915 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0658ecbf32754656876452b6f021926b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 21:29:24.737071 1817 raft_consensus.cc:1203] T 20a7af42e83c4639bba20f22dee69f2a P 153026dd7f8943cc93fedf6f48450fb4 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240502 21:29:24.744187 1817 raft_consensus.cc:1203] T 444c1cba2a6f4729aa01b75d06b6b517 P 153026dd7f8943cc93fedf6f48450fb4 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.11->[1.12-1.12] Dedup: 1.12->[]
I20240502 21:29:24.749375 6915 tablet_copy_client.cc:783] T 0658ecbf32754656876452b6f021926b P 153026dd7f8943cc93fedf6f48450fb4: tablet copy: Starting download of 0 data blocks...
I20240502 21:29:24.749720 6915 tablet_copy_client.cc:647] T 0658ecbf32754656876452b6f021926b P 153026dd7f8943cc93fedf6f48450fb4: tablet copy: Starting download of 1 WAL segments...
I20240502 21:29:24.751593 6915 tablet_copy_client.cc:523] T 0658ecbf32754656876452b6f021926b P 153026dd7f8943cc93fedf6f48450fb4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 21:29:24.781044 6915 tablet_bootstrap.cc:492] T 0658ecbf32754656876452b6f021926b P 153026dd7f8943cc93fedf6f48450fb4: Bootstrap starting.
I20240502 21:29:24.846464 6915 tablet_bootstrap.cc:492] T 0658ecbf32754656876452b6f021926b P 153026dd7f8943cc93fedf6f48450fb4: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 21:29:24.847059 6915 tablet_bootstrap.cc:492] T 0658ecbf32754656876452b6f021926b P 153026dd7f8943cc93fedf6f48450fb4: Bootstrap complete.
I20240502 21:29:24.847455 6915 ts_tablet_manager.cc:1367] T 0658ecbf32754656876452b6f021926b P 153026dd7f8943cc93fedf6f48450fb4: Time spent bootstrapping tablet: real 0.067s user 0.044s sys 0.005s
I20240502 21:29:24.848327 6915 raft_consensus.cc:348] T 0658ecbf32754656876452b6f021926b P 153026dd7f8943cc93fedf6f48450fb4 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } attrs { replace: true } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: NON_VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { promote: true } }
I20240502 21:29:24.848820 6915 raft_consensus.cc:729] T 0658ecbf32754656876452b6f021926b P 153026dd7f8943cc93fedf6f48450fb4 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 153026dd7f8943cc93fedf6f48450fb4, State: Initialized, Role: LEARNER
I20240502 21:29:24.849109 6915 consensus_queue.cc:260] T 0658ecbf32754656876452b6f021926b P 153026dd7f8943cc93fedf6f48450fb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } attrs { replace: true } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: NON_VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { promote: true } }
I20240502 21:29:24.849871 6915 ts_tablet_manager.cc:1398] T 0658ecbf32754656876452b6f021926b P 153026dd7f8943cc93fedf6f48450fb4: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 21:29:24.850746 1975 tablet_copy_service.cc:342] P e632c553ff4341328acf1cbaca678392: Request end of tablet copy session 153026dd7f8943cc93fedf6f48450fb4-0658ecbf32754656876452b6f021926b received from {username='slave'} at 127.15.255.132:49633
I20240502 21:29:24.850893 1975 tablet_copy_service.cc:433] P e632c553ff4341328acf1cbaca678392: ending tablet copy session 153026dd7f8943cc93fedf6f48450fb4-0658ecbf32754656876452b6f021926b on tablet 0658ecbf32754656876452b6f021926b with peer 153026dd7f8943cc93fedf6f48450fb4
I20240502 21:29:24.858903 6659 raft_consensus.cc:1050] T 4555e51179344acda81670fc9019d968 P 6ab8df3c796f4b6cb3e29d5045ad3ea0: attempting to promote NON_VOTER 153026dd7f8943cc93fedf6f48450fb4 to VOTER
I20240502 21:29:24.860168 6659 consensus_queue.cc:237] T 4555e51179344acda81670fc9019d968 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } attrs { replace: true } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { promote: false } }
I20240502 21:29:24.883256 1817 raft_consensus.cc:1261] T 4555e51179344acda81670fc9019d968 P 153026dd7f8943cc93fedf6f48450fb4 [term 1 LEARNER]: Refusing update from remote peer 6ab8df3c796f4b6cb3e29d5045ad3ea0: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240502 21:29:24.887189 7197 consensus_queue.cc:1035] T 4555e51179344acda81670fc9019d968 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 21:29:24.901445 4121 raft_consensus.cc:1261] T 4555e51179344acda81670fc9019d968 P 1fab5794cf1740288abd82f2cebb9b1a [term 1 FOLLOWER]: Refusing update from remote peer 6ab8df3c796f4b6cb3e29d5045ad3ea0: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240502 21:29:24.910221 7201 consensus_queue.cc:1035] T 4555e51179344acda81670fc9019d968 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 21:29:24.921308 3986 raft_consensus.cc:1261] T 4555e51179344acda81670fc9019d968 P 16a97c1394f346f0b987f97cce7371b6 [term 1 FOLLOWER]: Refusing update from remote peer 6ab8df3c796f4b6cb3e29d5045ad3ea0: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240502 21:29:24.924271 2086 raft_consensus.cc:1261] T 4555e51179344acda81670fc9019d968 P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Refusing update from remote peer 6ab8df3c796f4b6cb3e29d5045ad3ea0: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240502 21:29:24.925266 1955 raft_consensus.cc:1261] T 4555e51179344acda81670fc9019d968 P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Refusing update from remote peer 6ab8df3c796f4b6cb3e29d5045ad3ea0: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240502 21:29:24.926106 7199 consensus_queue.cc:1035] T 4555e51179344acda81670fc9019d968 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 21:29:24.926460 6814 consensus_queue.cc:1035] T 4555e51179344acda81670fc9019d968 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 21:29:24.928475 7197 consensus_queue.cc:1035] T 4555e51179344acda81670fc9019d968 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 21:29:24.929195 1547 raft_consensus.cc:1261] T 4555e51179344acda81670fc9019d968 P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Refusing update from remote peer 6ab8df3c796f4b6cb3e29d5045ad3ea0: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240502 21:29:24.929803 1412 raft_consensus.cc:1261] T 4555e51179344acda81670fc9019d968 P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Refusing update from remote peer 6ab8df3c796f4b6cb3e29d5045ad3ea0: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240502 21:29:24.929824 6814 consensus_queue.cc:1035] T 4555e51179344acda81670fc9019d968 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 21:29:24.930339 7201 consensus_queue.cc:1035] T 4555e51179344acda81670fc9019d968 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 21:29:24.956279 1547 raft_consensus.cc:2932] T 4555e51179344acda81670fc9019d968 P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } attrs { replace: true } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { promote: false } } }
I20240502 21:29:24.958228 7203 raft_consensus.cc:2932] T 4555e51179344acda81670fc9019d968 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } attrs { replace: true } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { promote: false } } }
I20240502 21:29:24.958606 1953 raft_consensus.cc:2932] T 4555e51179344acda81670fc9019d968 P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } attrs { replace: true } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { promote: false } } }
I20240502 21:29:24.966249 1263 catalog_manager.cc:5484] T 4555e51179344acda81670fc9019d968 P a62a935ff43d494ca35d0c417943133d reported cstate change: config changed from index 11 to 12, 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } attrs { replace: true } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { promote: false } } }
I20240502 21:29:24.968943 7034 raft_consensus.cc:1050] T 444c1cba2a6f4729aa01b75d06b6b517 P e632c553ff4341328acf1cbaca678392: attempting to promote NON_VOTER a62a935ff43d494ca35d0c417943133d to VOTER
I20240502 21:29:24.970156 7034 consensus_queue.cc:237] T 444c1cba2a6f4729aa01b75d06b6b517 P e632c553ff4341328acf1cbaca678392 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } attrs { replace: true } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { promote: false } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: NON_VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { promote: true } }
I20240502 21:29:24.986353 3986 raft_consensus.cc:2932] T 4555e51179344acda81670fc9019d968 P 16a97c1394f346f0b987f97cce7371b6 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } attrs { replace: true } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { promote: false } } }
I20240502 21:29:24.998133 1817 raft_consensus.cc:2932] T 4555e51179344acda81670fc9019d968 P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } attrs { replace: true } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { promote: false } } }
I20240502 21:29:25.013424 6974 raft_consensus.cc:1050] T f8bee9bdcbff404280f000bdaeadf465 P 7d1e5c1b047443cbb28d7b0a9744babf: attempting to promote NON_VOTER 16b3a337bc5e462f83ba895ad8660deb to VOTER
I20240502 21:29:25.014797 6974 consensus_queue.cc:237] T f8bee9bdcbff404280f000bdaeadf465 P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { replace: true } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { promote: false } }
I20240502 21:29:25.029954 4121 raft_consensus.cc:2932] T 4555e51179344acda81670fc9019d968 P 1fab5794cf1740288abd82f2cebb9b1a [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } attrs { replace: true } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { promote: false } } }
I20240502 21:29:25.044893 2086 raft_consensus.cc:2932] T 4555e51179344acda81670fc9019d968 P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } attrs { replace: true } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { promote: false } } }
I20240502 21:29:25.059020 6987 raft_consensus.cc:1050] T aae7b5a01a0543afb1b5a7f44a9a864c P 7d1e5c1b047443cbb28d7b0a9744babf: attempting to promote NON_VOTER 153026dd7f8943cc93fedf6f48450fb4 to VOTER
I20240502 21:29:25.060461 6987 consensus_queue.cc:237] T aae7b5a01a0543afb1b5a7f44a9a864c P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } attrs { replace: true } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { promote: false } }
I20240502 21:29:25.062635 6908 raft_consensus.cc:1050] T e45af71e3fc14c579b32ba4dbe2adf78 P 521bb93035324cecbebed2f6f2a630f8: attempting to promote NON_VOTER a62a935ff43d494ca35d0c417943133d to VOTER
I20240502 21:29:25.063897 6908 consensus_queue.cc:237] T e45af71e3fc14c579b32ba4dbe2adf78 P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { promote: false } }
I20240502 21:29:25.074245 1547 raft_consensus.cc:1261] T e45af71e3fc14c579b32ba4dbe2adf78 P a62a935ff43d494ca35d0c417943133d [term 1 LEARNER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240502 21:29:25.074484 3986 raft_consensus.cc:1261] T e45af71e3fc14c579b32ba4dbe2adf78 P 16a97c1394f346f0b987f97cce7371b6 [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240502 21:29:25.074826 6817 consensus_queue.cc:1035] T e45af71e3fc14c579b32ba4dbe2adf78 P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 21:29:25.075398 6895 consensus_queue.cc:1035] T e45af71e3fc14c579b32ba4dbe2adf78 P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 21:29:25.078259 1686 raft_consensus.cc:1261] T e45af71e3fc14c579b32ba4dbe2adf78 P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240502 21:29:25.079138 1820 raft_consensus.cc:1261] T e45af71e3fc14c579b32ba4dbe2adf78 P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240502 21:29:25.079456 4119 raft_consensus.cc:1261] T e45af71e3fc14c579b32ba4dbe2adf78 P 1fab5794cf1740288abd82f2cebb9b1a [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240502 21:29:25.079890 6817 consensus_queue.cc:1035] T e45af71e3fc14c579b32ba4dbe2adf78 P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 21:29:25.080390 2225 raft_consensus.cc:1261] T e45af71e3fc14c579b32ba4dbe2adf78 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240502 21:29:25.080477 6817 consensus_queue.cc:1035] T e45af71e3fc14c579b32ba4dbe2adf78 P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 21:29:25.081545 1411 raft_consensus.cc:1261] T e45af71e3fc14c579b32ba4dbe2adf78 P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240502 21:29:25.082070 6895 consensus_queue.cc:1035] T e45af71e3fc14c579b32ba4dbe2adf78 P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 21:29:25.082593 6895 consensus_queue.cc:1035] T e45af71e3fc14c579b32ba4dbe2adf78 P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 21:29:25.083055 6895 consensus_queue.cc:1035] T e45af71e3fc14c579b32ba4dbe2adf78 P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 21:29:25.083523 1820 raft_consensus.cc:1261] T aae7b5a01a0543afb1b5a7f44a9a864c P 153026dd7f8943cc93fedf6f48450fb4 [term 1 LEARNER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240502 21:29:25.084700 4120 raft_consensus.cc:1261] T aae7b5a01a0543afb1b5a7f44a9a864c P 1fab5794cf1740288abd82f2cebb9b1a [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240502 21:29:25.085322 6794 consensus_queue.cc:1035] T aae7b5a01a0543afb1b5a7f44a9a864c P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 21:29:25.088407 3854 raft_consensus.cc:1261] T aae7b5a01a0543afb1b5a7f44a9a864c P d8d7297186694f77bb98de9174e4279c [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240502 21:29:25.090356 1818 raft_consensus.cc:1203] T 0658ecbf32754656876452b6f021926b P 153026dd7f8943cc93fedf6f48450fb4 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240502 21:29:25.091591 6794 consensus_queue.cc:1035] T aae7b5a01a0543afb1b5a7f44a9a864c P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 21:29:25.094341 6794 consensus_queue.cc:1035] T aae7b5a01a0543afb1b5a7f44a9a864c P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 21:29:25.096477 2225 raft_consensus.cc:1261] T aae7b5a01a0543afb1b5a7f44a9a864c P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240502 21:29:25.103534 6794 consensus_queue.cc:1035] T aae7b5a01a0543afb1b5a7f44a9a864c P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 21:29:25.127334 7223 raft_consensus.cc:2932] T e45af71e3fc14c579b32ba4dbe2adf78 P 521bb93035324cecbebed2f6f2a630f8 [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, a62a935ff43d494ca35d0c417943133d (127.15.255.130) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { promote: false } } }
I20240502 21:29:25.127436 1547 raft_consensus.cc:2932] T e45af71e3fc14c579b32ba4dbe2adf78 P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, a62a935ff43d494ca35d0c417943133d (127.15.255.130) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { promote: false } } }
I20240502 21:29:25.144671 1683 raft_consensus.cc:2932] T e45af71e3fc14c579b32ba4dbe2adf78 P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, a62a935ff43d494ca35d0c417943133d (127.15.255.130) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { promote: false } } }
I20240502 21:29:25.144140 1817 raft_consensus.cc:2932] T e45af71e3fc14c579b32ba4dbe2adf78 P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, a62a935ff43d494ca35d0c417943133d (127.15.255.130) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { promote: false } } }
I20240502 21:29:25.158250 3986 raft_consensus.cc:2932] T e45af71e3fc14c579b32ba4dbe2adf78 P 16a97c1394f346f0b987f97cce7371b6 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, a62a935ff43d494ca35d0c417943133d (127.15.255.130) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { promote: false } } }
I20240502 21:29:25.158509 2225 raft_consensus.cc:2932] T e45af71e3fc14c579b32ba4dbe2adf78 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, a62a935ff43d494ca35d0c417943133d (127.15.255.130) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { promote: false } } }
I20240502 21:29:25.158301 1412 raft_consensus.cc:2932] T e45af71e3fc14c579b32ba4dbe2adf78 P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, a62a935ff43d494ca35d0c417943133d (127.15.255.130) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { promote: false } } }
I20240502 21:29:25.170250 1412 raft_consensus.cc:2932] T 4555e51179344acda81670fc9019d968 P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } attrs { replace: true } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { promote: false } } }
I20240502 21:29:25.180841 4119 raft_consensus.cc:2932] T e45af71e3fc14c579b32ba4dbe2adf78 P 1fab5794cf1740288abd82f2cebb9b1a [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, a62a935ff43d494ca35d0c417943133d (127.15.255.130) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { promote: false } } }
I20240502 21:29:25.188306 3854 raft_consensus.cc:1261] T f8bee9bdcbff404280f000bdaeadf465 P d8d7297186694f77bb98de9174e4279c [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240502 21:29:25.188371 3986 raft_consensus.cc:1261] T f8bee9bdcbff404280f000bdaeadf465 P 16a97c1394f346f0b987f97cce7371b6 [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240502 21:29:25.189080 1820 raft_consensus.cc:1261] T 444c1cba2a6f4729aa01b75d06b6b517 P 153026dd7f8943cc93fedf6f48450fb4 [term 1 LEARNER]: Refusing update from remote peer e632c553ff4341328acf1cbaca678392: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240502 21:29:25.189257 7242 consensus_queue.cc:1035] T f8bee9bdcbff404280f000bdaeadf465 P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 21:29:25.189672 6750 consensus_queue.cc:1035] T 444c1cba2a6f4729aa01b75d06b6b517 P e632c553ff4341328acf1cbaca678392 [LEADER]: Connected to new peer: Peer: permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: NON_VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240502 21:29:25.190761 1549 raft_consensus.cc:1261] T f8bee9bdcbff404280f000bdaeadf465 P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240502 21:29:25.190860 7242 consensus_queue.cc:1035] T f8bee9bdcbff404280f000bdaeadf465 P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 21:29:25.191100 3854 raft_consensus.cc:1261] T 444c1cba2a6f4729aa01b75d06b6b517 P d8d7297186694f77bb98de9174e4279c [term 1 FOLLOWER]: Refusing update from remote peer e632c553ff4341328acf1cbaca678392: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240502 21:29:25.192047 1955 raft_consensus.cc:1261] T f8bee9bdcbff404280f000bdaeadf465 P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240502 21:29:25.192260 2086 raft_consensus.cc:1261] T f8bee9bdcbff404280f000bdaeadf465 P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240502 21:29:25.192786 6750 consensus_queue.cc:1035] T 444c1cba2a6f4729aa01b75d06b6b517 P e632c553ff4341328acf1cbaca678392 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240502 21:29:25.193277 7242 consensus_queue.cc:1035] T f8bee9bdcbff404280f000bdaeadf465 P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 21:29:25.193773 7242 consensus_queue.cc:1035] T f8bee9bdcbff404280f000bdaeadf465 P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 21:29:25.193871 1547 raft_consensus.cc:1261] T 444c1cba2a6f4729aa01b75d06b6b517 P a62a935ff43d494ca35d0c417943133d [term 1 LEARNER]: Refusing update from remote peer e632c553ff4341328acf1cbaca678392: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240502 21:29:25.195652 6794 consensus_queue.cc:1035] T f8bee9bdcbff404280f000bdaeadf465 P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 21:29:25.197746 6750 consensus_queue.cc:1035] T 444c1cba2a6f4729aa01b75d06b6b517 P e632c553ff4341328acf1cbaca678392 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240502 21:29:25.198168 1686 raft_consensus.cc:1261] T f8bee9bdcbff404280f000bdaeadf465 P 16b3a337bc5e462f83ba895ad8660deb [term 1 LEARNER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240502 21:29:25.201726 2087 raft_consensus.cc:1261] T aae7b5a01a0543afb1b5a7f44a9a864c P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240502 21:29:25.202347 7242 consensus_queue.cc:1035] T aae7b5a01a0543afb1b5a7f44a9a864c P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 21:29:25.204814 7237 raft_consensus.cc:1050] T 20a7af42e83c4639bba20f22dee69f2a P 7d1e5c1b047443cbb28d7b0a9744babf: attempting to promote NON_VOTER 153026dd7f8943cc93fedf6f48450fb4 to VOTER
I20240502 21:29:25.206277 7237 consensus_queue.cc:237] T 20a7af42e83c4639bba20f22dee69f2a P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } attrs { replace: true } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { promote: false } }
I20240502 21:29:25.212553 1953 raft_consensus.cc:1261] T aae7b5a01a0543afb1b5a7f44a9a864c P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240502 21:29:25.214107 7236 consensus_queue.cc:1035] T aae7b5a01a0543afb1b5a7f44a9a864c P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 21:29:25.214589 1266 catalog_manager.cc:5484] T e45af71e3fc14c579b32ba4dbe2adf78 P 16b3a337bc5e462f83ba895ad8660deb reported cstate change: config changed from index 11 to 12, a62a935ff43d494ca35d0c417943133d (127.15.255.130) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "521bb93035324cecbebed2f6f2a630f8" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { promote: false } } }
I20240502 21:29:25.219554 6987 consensus_queue.cc:1035] T f8bee9bdcbff404280f000bdaeadf465 P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 21:29:25.229472 7242 raft_consensus.cc:2932] T aae7b5a01a0543afb1b5a7f44a9a864c P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } attrs { replace: true } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { promote: false } } }
I20240502 21:29:25.236534 7238 raft_consensus.cc:2932] T f8bee9bdcbff404280f000bdaeadf465 P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 16b3a337bc5e462f83ba895ad8660deb (127.15.255.131) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { replace: true } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { promote: false } } }
I20240502 21:29:25.242882 3987 raft_consensus.cc:2932] T f8bee9bdcbff404280f000bdaeadf465 P 16a97c1394f346f0b987f97cce7371b6 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 16b3a337bc5e462f83ba895ad8660deb (127.15.255.131) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { replace: true } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { promote: false } } }
I20240502 21:29:25.243016 2087 raft_consensus.cc:2932] T f8bee9bdcbff404280f000bdaeadf465 P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 16b3a337bc5e462f83ba895ad8660deb (127.15.255.131) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { replace: true } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { promote: false } } }
I20240502 21:29:25.243894 3854 raft_consensus.cc:2932] T f8bee9bdcbff404280f000bdaeadf465 P d8d7297186694f77bb98de9174e4279c [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 16b3a337bc5e462f83ba895ad8660deb (127.15.255.131) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { replace: true } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { promote: false } } }
I20240502 21:29:25.245193 6750 raft_consensus.cc:1013] T 444c1cba2a6f4729aa01b75d06b6b517 P e632c553ff4341328acf1cbaca678392 [term 1 LEADER]: attempt to promote peer 153026dd7f8943cc93fedf6f48450fb4: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240502 21:29:25.246012 1686 raft_consensus.cc:1261] T 444c1cba2a6f4729aa01b75d06b6b517 P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Refusing update from remote peer e632c553ff4341328acf1cbaca678392: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240502 21:29:25.246618 7032 consensus_queue.cc:1035] T 444c1cba2a6f4729aa01b75d06b6b517 P e632c553ff4341328acf1cbaca678392 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240502 21:29:25.247506 1820 raft_consensus.cc:2932] T aae7b5a01a0543afb1b5a7f44a9a864c P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } attrs { replace: true } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { promote: false } } }
I20240502 21:29:25.248811 1549 raft_consensus.cc:1261] T aae7b5a01a0543afb1b5a7f44a9a864c P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240502 21:29:25.249819 2225 raft_consensus.cc:1261] T f8bee9bdcbff404280f000bdaeadf465 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240502 21:29:25.250819 7236 consensus_queue.cc:1035] T aae7b5a01a0543afb1b5a7f44a9a864c P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 21:29:25.251145 6974 consensus_queue.cc:1035] T f8bee9bdcbff404280f000bdaeadf465 P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240502 21:29:25.231874 2086 raft_consensus.cc:2932] T aae7b5a01a0543afb1b5a7f44a9a864c P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } attrs { replace: true } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { promote: false } } }
I20240502 21:29:25.260363 3855 raft_consensus.cc:2932] T aae7b5a01a0543afb1b5a7f44a9a864c P d8d7297186694f77bb98de9174e4279c [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } attrs { replace: true } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { promote: false } } }
I20240502 21:29:25.262166 1412 raft_consensus.cc:1261] T 444c1cba2a6f4729aa01b75d06b6b517 P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Refusing update from remote peer e632c553ff4341328acf1cbaca678392: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240502 21:29:25.262717 6750 consensus_queue.cc:1035] T 444c1cba2a6f4729aa01b75d06b6b517 P e632c553ff4341328acf1cbaca678392 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240502 21:29:25.263551 4119 raft_consensus.cc:1261] T 444c1cba2a6f4729aa01b75d06b6b517 P 1fab5794cf1740288abd82f2cebb9b1a [term 1 FOLLOWER]: Refusing update from remote peer e632c553ff4341328acf1cbaca678392: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240502 21:29:25.261708 1955 raft_consensus.cc:2932] T f8bee9bdcbff404280f000bdaeadf465 P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 16b3a337bc5e462f83ba895ad8660deb (127.15.255.131) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { replace: true } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { promote: false } } }
I20240502 21:29:25.265990 6750 consensus_queue.cc:1035] T 444c1cba2a6f4729aa01b75d06b6b517 P e632c553ff4341328acf1cbaca678392 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240502 21:29:25.272034 4121 raft_consensus.cc:2932] T aae7b5a01a0543afb1b5a7f44a9a864c P 1fab5794cf1740288abd82f2cebb9b1a [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } attrs { replace: true } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { promote: false } } }
I20240502 21:29:25.275264 2226 raft_consensus.cc:1261] T 444c1cba2a6f4729aa01b75d06b6b517 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Refusing update from remote peer e632c553ff4341328acf1cbaca678392: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240502 21:29:25.275871 6750 consensus_queue.cc:1035] T 444c1cba2a6f4729aa01b75d06b6b517 P e632c553ff4341328acf1cbaca678392 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240502 21:29:25.279664 2227 raft_consensus.cc:2932] T aae7b5a01a0543afb1b5a7f44a9a864c P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } attrs { replace: true } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { promote: false } } }
I20240502 21:29:25.287230 2087 raft_consensus.cc:1261] T 444c1cba2a6f4729aa01b75d06b6b517 P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Refusing update from remote peer e632c553ff4341328acf1cbaca678392: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240502 21:29:25.291280 7034 consensus_queue.cc:1035] T 444c1cba2a6f4729aa01b75d06b6b517 P e632c553ff4341328acf1cbaca678392 [LEADER]: Connected to new peer: Peer: permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240502 21:29:25.292281 1954 raft_consensus.cc:2932] T aae7b5a01a0543afb1b5a7f44a9a864c P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } attrs { replace: true } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { promote: false } } }
I20240502 21:29:25.299173 2225 raft_consensus.cc:2932] T f8bee9bdcbff404280f000bdaeadf465 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 16b3a337bc5e462f83ba895ad8660deb (127.15.255.131) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { replace: true } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { promote: false } } }
I20240502 21:29:25.302536 1820 raft_consensus.cc:1261] T 20a7af42e83c4639bba20f22dee69f2a P 153026dd7f8943cc93fedf6f48450fb4 [term 1 LEARNER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:29:25.303210 7233 consensus_queue.cc:1035] T 20a7af42e83c4639bba20f22dee69f2a P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:29:25.307678 1685 raft_consensus.cc:1261] T 20a7af42e83c4639bba20f22dee69f2a P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:29:25.308651 2089 raft_consensus.cc:1261] T 20a7af42e83c4639bba20f22dee69f2a P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:29:25.309280 6974 consensus_queue.cc:1035] T 20a7af42e83c4639bba20f22dee69f2a P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:29:25.310155 6794 consensus_queue.cc:1035] T 20a7af42e83c4639bba20f22dee69f2a P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:29:25.314283 2227 raft_consensus.cc:1261] T 20a7af42e83c4639bba20f22dee69f2a P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:29:25.314657 1548 raft_consensus.cc:1261] T 20a7af42e83c4639bba20f22dee69f2a P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:29:25.315287 7233 consensus_queue.cc:1035] T 20a7af42e83c4639bba20f22dee69f2a P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:29:25.317643 1954 raft_consensus.cc:1261] T 20a7af42e83c4639bba20f22dee69f2a P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:29:25.318614 6794 consensus_queue.cc:1035] T 20a7af42e83c4639bba20f22dee69f2a P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:29:25.319319 1264 catalog_manager.cc:5484] T f8bee9bdcbff404280f000bdaeadf465 P 521bb93035324cecbebed2f6f2a630f8 reported cstate change: config changed from index 11 to 12, 16b3a337bc5e462f83ba895ad8660deb (127.15.255.131) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { replace: true } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { promote: false } } }
I20240502 21:29:25.325608 7237 consensus_queue.cc:1035] T 20a7af42e83c4639bba20f22dee69f2a P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:29:25.326154 3986 raft_consensus.cc:1261] T 20a7af42e83c4639bba20f22dee69f2a P 16a97c1394f346f0b987f97cce7371b6 [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:29:25.326689 7237 consensus_queue.cc:1035] T 20a7af42e83c4639bba20f22dee69f2a P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:29:25.328109 1548 raft_consensus.cc:2932] T f8bee9bdcbff404280f000bdaeadf465 P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 16b3a337bc5e462f83ba895ad8660deb (127.15.255.131) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { replace: true } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { promote: false } } }
I20240502 21:29:25.329140 1683 raft_consensus.cc:2932] T f8bee9bdcbff404280f000bdaeadf465 P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 16b3a337bc5e462f83ba895ad8660deb (127.15.255.131) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { replace: true } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { promote: false } } }
I20240502 21:29:25.343380 1267 catalog_manager.cc:5484] T aae7b5a01a0543afb1b5a7f44a9a864c P 1fab5794cf1740288abd82f2cebb9b1a reported cstate change: config changed from index 11 to 12, 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } attrs { replace: true } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { promote: false } } }
I20240502 21:29:25.344957 2089 consensus_queue.cc:237] T e45af71e3fc14c579b32ba4dbe2adf78 P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { promote: false } }
I20240502 21:29:25.359201 6750 raft_consensus.cc:2932] T 444c1cba2a6f4729aa01b75d06b6b517 P e632c553ff4341328acf1cbaca678392 [term 1 LEADER]: Committing config change with OpId 1.13: config changed from index 12 to 13, a62a935ff43d494ca35d0c417943133d (127.15.255.130) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } attrs { replace: true } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { promote: false } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: NON_VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { promote: true } } }
I20240502 21:29:25.360381 1549 raft_consensus.cc:2932] T aae7b5a01a0543afb1b5a7f44a9a864c P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } attrs { replace: true } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { promote: false } } }
I20240502 21:29:25.362336 2227 raft_consensus.cc:2932] T 444c1cba2a6f4729aa01b75d06b6b517 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, a62a935ff43d494ca35d0c417943133d (127.15.255.130) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } attrs { replace: true } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { promote: false } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: NON_VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { promote: true } } }
I20240502 21:29:25.364012 1818 raft_consensus.cc:2932] T 444c1cba2a6f4729aa01b75d06b6b517 P 153026dd7f8943cc93fedf6f48450fb4 [term 1 LEARNER]: Committing config change with OpId 1.13: config changed from index 12 to 13, a62a935ff43d494ca35d0c417943133d (127.15.255.130) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } attrs { replace: true } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { promote: false } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: NON_VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { promote: true } } }
I20240502 21:29:25.366973 1548 raft_consensus.cc:1261] T e45af71e3fc14c579b32ba4dbe2adf78 P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240502 21:29:25.367518 4119 raft_consensus.cc:1261] T e45af71e3fc14c579b32ba4dbe2adf78 P 1fab5794cf1740288abd82f2cebb9b1a [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240502 21:29:25.367991 1817 raft_consensus.cc:1261] T e45af71e3fc14c579b32ba4dbe2adf78 P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240502 21:29:25.368654 6974 raft_consensus.cc:2932] T 20a7af42e83c4639bba20f22dee69f2a P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } attrs { replace: true } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { promote: false } } }
I20240502 21:29:25.369333 1411 raft_consensus.cc:1261] T e45af71e3fc14c579b32ba4dbe2adf78 P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240502 21:29:25.369884 7223 consensus_queue.cc:1035] T e45af71e3fc14c579b32ba4dbe2adf78 P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240502 21:29:25.371652 7223 consensus_queue.cc:1035] T e45af71e3fc14c579b32ba4dbe2adf78 P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240502 21:29:25.372042 2225 consensus_queue.cc:237] T 4555e51179344acda81670fc9019d968 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { promote: false } }
I20240502 21:29:25.372910 6908 consensus_queue.cc:1035] T e45af71e3fc14c579b32ba4dbe2adf78 P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240502 21:29:25.375629 4119 raft_consensus.cc:1261] T 4555e51179344acda81670fc9019d968 P 1fab5794cf1740288abd82f2cebb9b1a [term 1 FOLLOWER]: Refusing update from remote peer 6ab8df3c796f4b6cb3e29d5045ad3ea0: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240502 21:29:25.375829 6895 consensus_queue.cc:1035] T e45af71e3fc14c579b32ba4dbe2adf78 P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240502 21:29:25.376493 3988 raft_consensus.cc:2932] T 20a7af42e83c4639bba20f22dee69f2a P 16a97c1394f346f0b987f97cce7371b6 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } attrs { replace: true } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { promote: false } } }
I20240502 21:29:25.376987 2089 raft_consensus.cc:2932] T 20a7af42e83c4639bba20f22dee69f2a P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } attrs { replace: true } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { promote: false } } }
I20240502 21:29:25.378628 1411 raft_consensus.cc:1261] T 4555e51179344acda81670fc9019d968 P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Refusing update from remote peer 6ab8df3c796f4b6cb3e29d5045ad3ea0: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240502 21:29:25.379997 2226 raft_consensus.cc:1261] T e45af71e3fc14c579b32ba4dbe2adf78 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240502 21:29:25.380604 3986 raft_consensus.cc:1261] T e45af71e3fc14c579b32ba4dbe2adf78 P 16a97c1394f346f0b987f97cce7371b6 [term 1 FOLLOWER]: Refusing update from remote peer 521bb93035324cecbebed2f6f2a630f8: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240502 21:29:25.381450 6817 consensus_queue.cc:1035] T e45af71e3fc14c579b32ba4dbe2adf78 P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240502 21:29:25.381300 2226 raft_consensus.cc:2932] T 20a7af42e83c4639bba20f22dee69f2a P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } attrs { replace: true } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { promote: false } } }
I20240502 21:29:25.381987 6659 consensus_queue.cc:1035] T 4555e51179344acda81670fc9019d968 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240502 21:29:25.383360 7200 consensus_queue.cc:1035] T 4555e51179344acda81670fc9019d968 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240502 21:29:25.384995 1550 raft_consensus.cc:1261] T 4555e51179344acda81670fc9019d968 P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Refusing update from remote peer 6ab8df3c796f4b6cb3e29d5045ad3ea0: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240502 21:29:25.385771 6895 consensus_queue.cc:1035] T e45af71e3fc14c579b32ba4dbe2adf78 P 521bb93035324cecbebed2f6f2a630f8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240502 21:29:25.386250 3985 raft_consensus.cc:2932] T e45af71e3fc14c579b32ba4dbe2adf78 P 16a97c1394f346f0b987f97cce7371b6 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER 16b3a337bc5e462f83ba895ad8660deb (127.15.255.131) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { promote: false } } }
I20240502 21:29:25.386273 6895 raft_consensus.cc:2932] T e45af71e3fc14c579b32ba4dbe2adf78 P 521bb93035324cecbebed2f6f2a630f8 [term 1 LEADER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER 16b3a337bc5e462f83ba895ad8660deb (127.15.255.131) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { promote: false } } }
I20240502 21:29:25.387763 2086 raft_consensus.cc:2932] T 444c1cba2a6f4729aa01b75d06b6b517 P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, a62a935ff43d494ca35d0c417943133d (127.15.255.130) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } attrs { replace: true } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { promote: false } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: NON_VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { promote: true } } }
I20240502 21:29:25.389505 7200 consensus_queue.cc:1035] T 4555e51179344acda81670fc9019d968 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240502 21:29:25.389811 1550 raft_consensus.cc:2932] T e45af71e3fc14c579b32ba4dbe2adf78 P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER 16b3a337bc5e462f83ba895ad8660deb (127.15.255.131) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { promote: false } } }
I20240502 21:29:25.394351 1270 catalog_manager.cc:5484] T 444c1cba2a6f4729aa01b75d06b6b517 P 153026dd7f8943cc93fedf6f48450fb4 reported cstate change: config changed from index 12 to 13, a62a935ff43d494ca35d0c417943133d (127.15.255.130) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e632c553ff4341328acf1cbaca678392" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } attrs { replace: true } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { promote: false } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: NON_VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { promote: true } } }
I20240502 21:29:25.395834 3987 raft_consensus.cc:1261] T 4555e51179344acda81670fc9019d968 P 16a97c1394f346f0b987f97cce7371b6 [term 1 FOLLOWER]: Refusing update from remote peer 6ab8df3c796f4b6cb3e29d5045ad3ea0: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240502 21:29:25.396908 7201 raft_consensus.cc:2932] T 4555e51179344acda81670fc9019d968 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 LEADER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER 521bb93035324cecbebed2f6f2a630f8 (127.15.255.134) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { promote: false } } }
I20240502 21:29:25.397379 2227 raft_consensus.cc:2932] T e45af71e3fc14c579b32ba4dbe2adf78 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER 16b3a337bc5e462f83ba895ad8660deb (127.15.255.131) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { promote: false } } }
I20240502 21:29:25.400779 6659 consensus_queue.cc:1035] T 4555e51179344acda81670fc9019d968 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240502 21:29:25.402004 1818 raft_consensus.cc:1261] T 4555e51179344acda81670fc9019d968 P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Refusing update from remote peer 6ab8df3c796f4b6cb3e29d5045ad3ea0: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240502 21:29:25.403000 1262 catalog_manager.cc:5484] T e45af71e3fc14c579b32ba4dbe2adf78 P a62a935ff43d494ca35d0c417943133d reported cstate change: config changed from index 12 to 13, VOTER 16b3a337bc5e462f83ba895ad8660deb (127.15.255.131) evicted. New cstate: current_term: 1 leader_uuid: "521bb93035324cecbebed2f6f2a630f8" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { promote: false } } }
I20240502 21:29:25.403213 1411 raft_consensus.cc:2932] T 4555e51179344acda81670fc9019d968 P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER 521bb93035324cecbebed2f6f2a630f8 (127.15.255.134) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { promote: false } } }
I20240502 21:29:25.403796 3987 raft_consensus.cc:2932] T 4555e51179344acda81670fc9019d968 P 16a97c1394f346f0b987f97cce7371b6 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER 521bb93035324cecbebed2f6f2a630f8 (127.15.255.134) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { promote: false } } }
I20240502 21:29:25.407353 6659 consensus_queue.cc:1035] T 4555e51179344acda81670fc9019d968 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240502 21:29:25.409065 1817 raft_consensus.cc:2932] T 4555e51179344acda81670fc9019d968 P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER 521bb93035324cecbebed2f6f2a630f8 (127.15.255.134) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { promote: false } } }
I20240502 21:29:25.414815 4119 raft_consensus.cc:2932] T 4555e51179344acda81670fc9019d968 P 1fab5794cf1740288abd82f2cebb9b1a [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER 521bb93035324cecbebed2f6f2a630f8 (127.15.255.134) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { promote: false } } }
I20240502 21:29:25.405938 1268 catalog_manager.cc:5484] T 20a7af42e83c4639bba20f22dee69f2a P 7d1e5c1b047443cbb28d7b0a9744babf reported cstate change: config changed from index 8 to 9, 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 21:29:25.415511 1550 raft_consensus.cc:2932] T 4555e51179344acda81670fc9019d968 P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER 521bb93035324cecbebed2f6f2a630f8 (127.15.255.134) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { promote: false } } }
I20240502 21:29:25.412994 1254 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet e45af71e3fc14c579b32ba4dbe2adf78 with cas_config_opid_index 12: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:29:25.413784 1255 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 4555e51179344acda81670fc9019d968 with cas_config_opid_index 12: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:29:25.422032 1269 catalog_manager.cc:5484] T 4555e51179344acda81670fc9019d968 P 1fab5794cf1740288abd82f2cebb9b1a reported cstate change: config changed from index 12 to 13, VOTER 521bb93035324cecbebed2f6f2a630f8 (127.15.255.134) evicted. New cstate: current_term: 1 leader_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { promote: false } } }
I20240502 21:29:25.424623 1953 raft_consensus.cc:1261] T 4555e51179344acda81670fc9019d968 P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Refusing update from remote peer 6ab8df3c796f4b6cb3e29d5045ad3ea0: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240502 21:29:25.427476 7034 raft_consensus.cc:1050] T 444c1cba2a6f4729aa01b75d06b6b517 P e632c553ff4341328acf1cbaca678392: attempting to promote NON_VOTER 153026dd7f8943cc93fedf6f48450fb4 to VOTER
I20240502 21:29:25.428197 1664 tablet_service.cc:1514] Processing DeleteTablet for tablet e45af71e3fc14c579b32ba4dbe2adf78 with delete_type TABLET_DATA_TOMBSTONED (TS 16b3a337bc5e462f83ba895ad8660deb not found in new config with opid_index 13) from {username='slave'} at 127.0.0.1:42590
I20240502 21:29:25.433828 2068 tablet_service.cc:1514] Processing DeleteTablet for tablet 4555e51179344acda81670fc9019d968 with delete_type TABLET_DATA_TOMBSTONED (TS 521bb93035324cecbebed2f6f2a630f8 not found in new config with opid_index 13) from {username='slave'} at 127.0.0.1:50230
W20240502 21:29:25.449748 1885 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.135:34701, user_credentials={real_user=slave}} blocked reactor thread for 69314.7us
I20240502 21:29:25.450515 1550 raft_consensus.cc:2932] T 444c1cba2a6f4729aa01b75d06b6b517 P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, a62a935ff43d494ca35d0c417943133d (127.15.255.130) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } attrs { replace: true } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { promote: false } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: NON_VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { promote: true } } }
I20240502 21:29:25.451433 1820 raft_consensus.cc:2932] T 20a7af42e83c4639bba20f22dee69f2a P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } attrs { replace: true } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { promote: false } } }
I20240502 21:29:25.454728 1686 raft_consensus.cc:2932] T 20a7af42e83c4639bba20f22dee69f2a P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } attrs { replace: true } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { promote: false } } }
I20240502 21:29:25.457018 1411 raft_consensus.cc:2932] T 444c1cba2a6f4729aa01b75d06b6b517 P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, a62a935ff43d494ca35d0c417943133d (127.15.255.130) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } attrs { replace: true } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { promote: false } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: NON_VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { promote: true } } }
I20240502 21:29:25.458679 1547 raft_consensus.cc:2932] T 20a7af42e83c4639bba20f22dee69f2a P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } attrs { replace: true } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { promote: false } } }
I20240502 21:29:25.463529 7287 tablet_replica.cc:330] T e45af71e3fc14c579b32ba4dbe2adf78 P 16b3a337bc5e462f83ba895ad8660deb: stopping tablet replica
I20240502 21:29:25.463790 7287 raft_consensus.cc:2229] T e45af71e3fc14c579b32ba4dbe2adf78 P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Raft consensus shutting down.
W20240502 21:29:25.466218 2023 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.135:34701, user_credentials={real_user=slave}} blocked reactor thread for 47446.2us
W20240502 21:29:25.464831 2025 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.138:33375, user_credentials={real_user=slave}} blocked reactor thread for 51942.1us
I20240502 21:29:25.468019 7288 tablet_replica.cc:330] T 4555e51179344acda81670fc9019d968 P 521bb93035324cecbebed2f6f2a630f8: stopping tablet replica
I20240502 21:29:25.467958 1411 consensus_queue.cc:237] T 20a7af42e83c4639bba20f22dee69f2a P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { promote: false } }
I20240502 21:29:25.468216 7288 raft_consensus.cc:2229] T 4555e51179344acda81670fc9019d968 P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:29:25.469301 4119 raft_consensus.cc:2932] T e45af71e3fc14c579b32ba4dbe2adf78 P 1fab5794cf1740288abd82f2cebb9b1a [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER 16b3a337bc5e462f83ba895ad8660deb (127.15.255.131) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { promote: false } } }
I20240502 21:29:25.473309 7288 raft_consensus.cc:2258] T 4555e51179344acda81670fc9019d968 P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:29:25.471082 1412 raft_consensus.cc:2932] T e45af71e3fc14c579b32ba4dbe2adf78 P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER 16b3a337bc5e462f83ba895ad8660deb (127.15.255.131) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { promote: false } } }
W20240502 21:29:25.474738 1887 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.138:33375, user_credentials={real_user=slave}} blocked reactor thread for 78195.1us
I20240502 21:29:25.475703 7288 ts_tablet_manager.cc:1875] T 4555e51179344acda81670fc9019d968 P 521bb93035324cecbebed2f6f2a630f8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:29:25.475936 4119 raft_consensus.cc:2932] T 444c1cba2a6f4729aa01b75d06b6b517 P 1fab5794cf1740288abd82f2cebb9b1a [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, a62a935ff43d494ca35d0c417943133d (127.15.255.130) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } attrs { replace: true } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { promote: false } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: NON_VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { promote: true } } }
I20240502 21:29:25.477061 7287 raft_consensus.cc:2258] T e45af71e3fc14c579b32ba4dbe2adf78 P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:29:25.478801 7287 ts_tablet_manager.cc:1875] T e45af71e3fc14c579b32ba4dbe2adf78 P 16b3a337bc5e462f83ba895ad8660deb: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:29:25.481463 7201 consensus_queue.cc:1035] T 4555e51179344acda81670fc9019d968 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240502 21:29:25.482973 1686 raft_consensus.cc:2932] T 444c1cba2a6f4729aa01b75d06b6b517 P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, a62a935ff43d494ca35d0c417943133d (127.15.255.130) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } attrs { replace: true } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { promote: false } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: NON_VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { promote: true } } }
I20240502 21:29:25.485304 1955 raft_consensus.cc:2932] T 4555e51179344acda81670fc9019d968 P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER 521bb93035324cecbebed2f6f2a630f8 (127.15.255.134) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { promote: false } } }
I20240502 21:29:25.488515 7034 consensus_queue.cc:237] T 444c1cba2a6f4729aa01b75d06b6b517 P e632c553ff4341328acf1cbaca678392 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 1.13, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } attrs { replace: true } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { promote: false } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { promote: false } }
W20240502 21:29:25.489255 1886 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.15.255.132:44947, user_credentials={real_user=slave}} blocked reactor thread for 115386us
I20240502 21:29:25.489964 3854 raft_consensus.cc:2932] T 444c1cba2a6f4729aa01b75d06b6b517 P d8d7297186694f77bb98de9174e4279c [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, a62a935ff43d494ca35d0c417943133d (127.15.255.130) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } attrs { replace: true } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { promote: false } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: NON_VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { promote: true } } }
I20240502 21:29:25.493455 1685 raft_consensus.cc:1261] T 20a7af42e83c4639bba20f22dee69f2a P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 21:29:25.494740 7237 consensus_queue.cc:1035] T 20a7af42e83c4639bba20f22dee69f2a P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 21:29:25.495847 2227 raft_consensus.cc:1261] T 20a7af42e83c4639bba20f22dee69f2a P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 21:29:25.496492 7237 consensus_queue.cc:1035] T 20a7af42e83c4639bba20f22dee69f2a P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 21:29:25.501057 1547 raft_consensus.cc:1261] T 20a7af42e83c4639bba20f22dee69f2a P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 21:29:25.501638 6974 consensus_queue.cc:1035] T 20a7af42e83c4639bba20f22dee69f2a P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 21:29:25.502540 1820 raft_consensus.cc:1261] T 20a7af42e83c4639bba20f22dee69f2a P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 21:29:25.503122 1550 raft_consensus.cc:1261] T 444c1cba2a6f4729aa01b75d06b6b517 P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Refusing update from remote peer e632c553ff4341328acf1cbaca678392: Log matching property violated. Preceding OpId in replica: term: 1 index: 13. Preceding OpId from leader: term: 1 index: 14. (index mismatch)
I20240502 21:29:25.503731 6974 consensus_queue.cc:1035] T 20a7af42e83c4639bba20f22dee69f2a P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 21:29:25.503739 7032 consensus_queue.cc:1035] T 444c1cba2a6f4729aa01b75d06b6b517 P e632c553ff4341328acf1cbaca678392 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240502 21:29:25.505874 3985 raft_consensus.cc:1261] T 20a7af42e83c4639bba20f22dee69f2a P 16a97c1394f346f0b987f97cce7371b6 [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 21:29:25.505379 7236 raft_consensus.cc:2932] T 20a7af42e83c4639bba20f22dee69f2a P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER e632c553ff4341328acf1cbaca678392 (127.15.255.133) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { promote: false } } }
I20240502 21:29:25.506471 6974 consensus_queue.cc:1035] T 20a7af42e83c4639bba20f22dee69f2a P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 21:29:25.507290 1817 raft_consensus.cc:1261] T 444c1cba2a6f4729aa01b75d06b6b517 P 153026dd7f8943cc93fedf6f48450fb4 [term 1 LEARNER]: Refusing update from remote peer e632c553ff4341328acf1cbaca678392: Log matching property violated. Preceding OpId in replica: term: 1 index: 13. Preceding OpId from leader: term: 1 index: 14. (index mismatch)
I20240502 21:29:25.507014 1953 raft_consensus.cc:2932] T 20a7af42e83c4639bba20f22dee69f2a P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } attrs { replace: true } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { promote: false } } }
I20240502 21:29:25.509539 1255 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 20a7af42e83c4639bba20f22dee69f2a with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 21:29:25.509443 3985 raft_consensus.cc:2932] T 20a7af42e83c4639bba20f22dee69f2a P 16a97c1394f346f0b987f97cce7371b6 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER e632c553ff4341328acf1cbaca678392 (127.15.255.133) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { promote: false } } }
I20240502 21:29:25.511329 1546 raft_consensus.cc:2932] T 20a7af42e83c4639bba20f22dee69f2a P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER e632c553ff4341328acf1cbaca678392 (127.15.255.133) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { promote: false } } }
I20240502 21:29:25.513213 2227 raft_consensus.cc:2932] T 20a7af42e83c4639bba20f22dee69f2a P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER e632c553ff4341328acf1cbaca678392 (127.15.255.133) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { promote: false } } }
I20240502 21:29:25.513702 1270 catalog_manager.cc:5484] T 20a7af42e83c4639bba20f22dee69f2a P 7d1e5c1b047443cbb28d7b0a9744babf reported cstate change: config changed from index 9 to 10, VOTER e632c553ff4341328acf1cbaca678392 (127.15.255.133) evicted. New cstate: current_term: 1 leader_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 21:29:25.516439 1685 raft_consensus.cc:2932] T 20a7af42e83c4639bba20f22dee69f2a P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER e632c553ff4341328acf1cbaca678392 (127.15.255.133) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { promote: false } } }
I20240502 21:29:25.517257 3854 raft_consensus.cc:1261] T 444c1cba2a6f4729aa01b75d06b6b517 P d8d7297186694f77bb98de9174e4279c [term 1 FOLLOWER]: Refusing update from remote peer e632c553ff4341328acf1cbaca678392: Log matching property violated. Preceding OpId in replica: term: 1 index: 13. Preceding OpId from leader: term: 1 index: 14. (index mismatch)
I20240502 21:29:25.518412 7032 raft_consensus.cc:1050] T 0658ecbf32754656876452b6f021926b P e632c553ff4341328acf1cbaca678392: attempting to promote NON_VOTER 153026dd7f8943cc93fedf6f48450fb4 to VOTER
I20240502 21:29:25.519490 7032 consensus_queue.cc:237] T 0658ecbf32754656876452b6f021926b P e632c553ff4341328acf1cbaca678392 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 5, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } attrs { replace: true } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { promote: false } }
I20240502 21:29:25.520947 2086 raft_consensus.cc:1261] T 20a7af42e83c4639bba20f22dee69f2a P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Refusing update from remote peer 7d1e5c1b047443cbb28d7b0a9744babf: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 21:29:25.524618 1817 raft_consensus.cc:2932] T 20a7af42e83c4639bba20f22dee69f2a P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER e632c553ff4341328acf1cbaca678392 (127.15.255.133) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { promote: false } } }
I20240502 21:29:25.525552 1818 raft_consensus.cc:2932] T e45af71e3fc14c579b32ba4dbe2adf78 P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER 16b3a337bc5e462f83ba895ad8660deb (127.15.255.131) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { promote: false } } }
I20240502 21:29:25.527303 2086 raft_consensus.cc:1261] T 444c1cba2a6f4729aa01b75d06b6b517 P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Refusing update from remote peer e632c553ff4341328acf1cbaca678392: Log matching property violated. Preceding OpId in replica: term: 1 index: 13. Preceding OpId from leader: term: 1 index: 14. (index mismatch)
I20240502 21:29:25.527841 6750 consensus_queue.cc:1035] T 444c1cba2a6f4729aa01b75d06b6b517 P e632c553ff4341328acf1cbaca678392 [LEADER]: Connected to new peer: Peer: permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240502 21:29:25.530969 6750 consensus_queue.cc:1035] T 444c1cba2a6f4729aa01b75d06b6b517 P e632c553ff4341328acf1cbaca678392 [LEADER]: Connected to new peer: Peer: permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240502 21:29:25.532254 7034 consensus_queue.cc:1035] T 444c1cba2a6f4729aa01b75d06b6b517 P e632c553ff4341328acf1cbaca678392 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240502 21:29:25.532570 1931 tablet_service.cc:1514] Processing DeleteTablet for tablet 20a7af42e83c4639bba20f22dee69f2a with delete_type TABLET_DATA_TOMBSTONED (TS e632c553ff4341328acf1cbaca678392 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:51198
I20240502 21:29:25.535746 2226 raft_consensus.cc:1261] T 444c1cba2a6f4729aa01b75d06b6b517 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Refusing update from remote peer e632c553ff4341328acf1cbaca678392: Log matching property violated. Preceding OpId in replica: term: 1 index: 13. Preceding OpId from leader: term: 1 index: 14. (index mismatch)
I20240502 21:29:25.536125 7287 ts_tablet_manager.cc:1888] T e45af71e3fc14c579b32ba4dbe2adf78 P 16b3a337bc5e462f83ba895ad8660deb: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.12
I20240502 21:29:25.536262 7287 log.cc:1196] T e45af71e3fc14c579b32ba4dbe2adf78 P 16b3a337bc5e462f83ba895ad8660deb: Deleting WAL directory at /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-2/wal/wals/e45af71e3fc14c579b32ba4dbe2adf78
I20240502 21:29:25.536959 7237 consensus_queue.cc:1035] T 20a7af42e83c4639bba20f22dee69f2a P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Connected to new peer: Peer: permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 21:29:25.538810 7278 consensus_queue.cc:1035] T 444c1cba2a6f4729aa01b75d06b6b517 P e632c553ff4341328acf1cbaca678392 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240502 21:29:25.544232 2086 raft_consensus.cc:2932] T 20a7af42e83c4639bba20f22dee69f2a P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER e632c553ff4341328acf1cbaca678392 (127.15.255.133) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { promote: false } } }
I20240502 21:29:25.544862 1411 raft_consensus.cc:1261] T 444c1cba2a6f4729aa01b75d06b6b517 P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Refusing update from remote peer e632c553ff4341328acf1cbaca678392: Log matching property violated. Preceding OpId in replica: term: 1 index: 13. Preceding OpId from leader: term: 1 index: 14. (index mismatch)
I20240502 21:29:25.545325 7278 consensus_queue.cc:1035] T 444c1cba2a6f4729aa01b75d06b6b517 P e632c553ff4341328acf1cbaca678392 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240502 21:29:25.551777 7288 ts_tablet_manager.cc:1888] T 4555e51179344acda81670fc9019d968 P 521bb93035324cecbebed2f6f2a630f8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.12
I20240502 21:29:25.552456 7034 raft_consensus.cc:2932] T 444c1cba2a6f4729aa01b75d06b6b517 P e632c553ff4341328acf1cbaca678392 [term 1 LEADER]: Committing config change with OpId 1.14: config changed from index 13 to 14, 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132) changed from NON_VOTER to VOTER. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } attrs { replace: true } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { promote: false } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { promote: false } } }
I20240502 21:29:25.553100 1412 raft_consensus.cc:2932] T 444c1cba2a6f4729aa01b75d06b6b517 P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Committing config change with OpId 1.14: config changed from index 13 to 14, 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132) changed from NON_VOTER to VOTER. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } attrs { replace: true } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { promote: false } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { promote: false } } }
I20240502 21:29:25.558701 2226 raft_consensus.cc:2932] T 444c1cba2a6f4729aa01b75d06b6b517 P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Committing config change with OpId 1.14: config changed from index 13 to 14, 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132) changed from NON_VOTER to VOTER. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } attrs { replace: true } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { promote: false } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { promote: false } } }
I20240502 21:29:25.561786 4120 raft_consensus.cc:1261] T 444c1cba2a6f4729aa01b75d06b6b517 P 1fab5794cf1740288abd82f2cebb9b1a [term 1 FOLLOWER]: Refusing update from remote peer e632c553ff4341328acf1cbaca678392: Log matching property violated. Preceding OpId in replica: term: 1 index: 13. Preceding OpId from leader: term: 1 index: 14. (index mismatch)
I20240502 21:29:25.561779 1547 raft_consensus.cc:2932] T 444c1cba2a6f4729aa01b75d06b6b517 P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Committing config change with OpId 1.14: config changed from index 13 to 14, 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132) changed from NON_VOTER to VOTER. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } attrs { replace: true } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { promote: false } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { promote: false } } }
I20240502 21:29:25.563741 7280 consensus_queue.cc:1035] T 444c1cba2a6f4729aa01b75d06b6b517 P e632c553ff4341328acf1cbaca678392 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240502 21:29:25.567054 1818 raft_consensus.cc:2932] T 444c1cba2a6f4729aa01b75d06b6b517 P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Committing config change with OpId 1.14: config changed from index 13 to 14, 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132) changed from NON_VOTER to VOTER. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } attrs { replace: true } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { promote: false } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { promote: false } } }
I20240502 21:29:25.567734 1267 catalog_manager.cc:5484] T 444c1cba2a6f4729aa01b75d06b6b517 P e632c553ff4341328acf1cbaca678392 reported cstate change: config changed from index 13 to 14, 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e632c553ff4341328acf1cbaca678392" committed_config { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 21:29:25.569020 1686 raft_consensus.cc:1261] T 444c1cba2a6f4729aa01b75d06b6b517 P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Refusing update from remote peer e632c553ff4341328acf1cbaca678392: Log matching property violated. Preceding OpId in replica: term: 1 index: 13. Preceding OpId from leader: term: 1 index: 14. (index mismatch)
I20240502 21:29:25.570677 1257 catalog_manager.cc:4830] TS 16b3a337bc5e462f83ba895ad8660deb (127.15.255.131:36033): tablet e45af71e3fc14c579b32ba4dbe2adf78 (table rebalance_test_table_5 [id=9cb803e5bb2545c8b992c90a894085af]) successfully deleted
I20240502 21:29:25.574174 3854 raft_consensus.cc:2932] T 444c1cba2a6f4729aa01b75d06b6b517 P d8d7297186694f77bb98de9174e4279c [term 1 FOLLOWER]: Committing config change with OpId 1.14: config changed from index 13 to 14, 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132) changed from NON_VOTER to VOTER. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } attrs { replace: true } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { promote: false } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { promote: false } } }
I20240502 21:29:25.565436 2086 raft_consensus.cc:2932] T 444c1cba2a6f4729aa01b75d06b6b517 P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Committing config change with OpId 1.14: config changed from index 13 to 14, 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132) changed from NON_VOTER to VOTER. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } attrs { replace: true } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { promote: false } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { promote: false } } }
I20240502 21:29:25.581498 4120 raft_consensus.cc:2932] T 444c1cba2a6f4729aa01b75d06b6b517 P 1fab5794cf1740288abd82f2cebb9b1a [term 1 FOLLOWER]: Committing config change with OpId 1.14: config changed from index 13 to 14, 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132) changed from NON_VOTER to VOTER. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } attrs { replace: true } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { promote: false } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { promote: false } } }
I20240502 21:29:25.582892 7288 log.cc:1196] T 4555e51179344acda81670fc9019d968 P 521bb93035324cecbebed2f6f2a630f8: Deleting WAL directory at /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-5/wal/wals/4555e51179344acda81670fc9019d968
I20240502 21:29:25.583711 1254 catalog_manager.cc:4830] TS 521bb93035324cecbebed2f6f2a630f8 (127.15.255.134:40821): tablet 4555e51179344acda81670fc9019d968 (table rebalance_test_table_6 [id=552d53f5242c4a2eb8e89b63e8eb6a7a]) successfully deleted
I20240502 21:29:25.591426 7244 consensus_queue.cc:1035] T 444c1cba2a6f4729aa01b75d06b6b517 P e632c553ff4341328acf1cbaca678392 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240502 21:29:25.593683 1686 raft_consensus.cc:2932] T 444c1cba2a6f4729aa01b75d06b6b517 P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Committing config change with OpId 1.14: config changed from index 13 to 14, 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132) changed from NON_VOTER to VOTER. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } attrs { replace: true } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { replace: true } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } attrs { promote: false } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { promote: false } } }
I20240502 21:29:25.601307 7293 tablet_replica.cc:330] T 20a7af42e83c4639bba20f22dee69f2a P e632c553ff4341328acf1cbaca678392: stopping tablet replica
I20240502 21:29:25.601529 7293 raft_consensus.cc:2229] T 20a7af42e83c4639bba20f22dee69f2a P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 21:29:25.601831 7293 raft_consensus.cc:2258] T 20a7af42e83c4639bba20f22dee69f2a P e632c553ff4341328acf1cbaca678392 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 21:29:25.603829 7293 ts_tablet_manager.cc:1875] T 20a7af42e83c4639bba20f22dee69f2a P e632c553ff4341328acf1cbaca678392: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 21:29:25.609138 1818 raft_consensus.cc:1261] T 0658ecbf32754656876452b6f021926b P 153026dd7f8943cc93fedf6f48450fb4 [term 1 LEARNER]: Refusing update from remote peer e632c553ff4341328acf1cbaca678392: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:29:25.610782 1412 raft_consensus.cc:1261] T 0658ecbf32754656876452b6f021926b P 7d1e5c1b047443cbb28d7b0a9744babf [term 1 FOLLOWER]: Refusing update from remote peer e632c553ff4341328acf1cbaca678392: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:29:25.611423 1547 raft_consensus.cc:1261] T 0658ecbf32754656876452b6f021926b P a62a935ff43d494ca35d0c417943133d [term 1 FOLLOWER]: Refusing update from remote peer e632c553ff4341328acf1cbaca678392: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:29:25.612183 2086 raft_consensus.cc:1261] T 0658ecbf32754656876452b6f021926b P 521bb93035324cecbebed2f6f2a630f8 [term 1 FOLLOWER]: Refusing update from remote peer e632c553ff4341328acf1cbaca678392: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:29:25.612473 7034 consensus_queue.cc:1035] T 0658ecbf32754656876452b6f021926b P e632c553ff4341328acf1cbaca678392 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:29:25.612763 7280 consensus_queue.cc:1035] T 0658ecbf32754656876452b6f021926b P e632c553ff4341328acf1cbaca678392 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:29:25.613454 2227 raft_consensus.cc:1261] T 0658ecbf32754656876452b6f021926b P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Refusing update from remote peer e632c553ff4341328acf1cbaca678392: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:29:25.614126 4120 raft_consensus.cc:1261] T 0658ecbf32754656876452b6f021926b P 1fab5794cf1740288abd82f2cebb9b1a [term 1 FOLLOWER]: Refusing update from remote peer e632c553ff4341328acf1cbaca678392: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:29:25.615206 7278 consensus_queue.cc:1035] T 0658ecbf32754656876452b6f021926b P e632c553ff4341328acf1cbaca678392 [LEADER]: Connected to new peer: Peer: permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.002s
I20240502 21:29:25.617791 7032 consensus_queue.cc:1035] T 0658ecbf32754656876452b6f021926b P e632c553ff4341328acf1cbaca678392 [LEADER]: Connected to new peer: Peer: permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20240502 21:29:25.618095 7244 consensus_queue.cc:1035] T 0658ecbf32754656876452b6f021926b P e632c553ff4341328acf1cbaca678392 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:29:25.619408 1686 raft_consensus.cc:1261] T 0658ecbf32754656876452b6f021926b P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Refusing update from remote peer e632c553ff4341328acf1cbaca678392: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 21:29:25.625981 7244 consensus_queue.cc:1035] T 0658ecbf32754656876452b6f021926b P e632c553ff4341328acf1cbaca678392 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20240502 21:29:25.626535 7032 consensus_queue.cc:1035] T 0658ecbf32754656876452b6f021926b P e632c553ff4341328acf1cbaca678392 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 21:29:25.652231 7244 raft_consensus.cc:2932] T 0658ecbf32754656876452b6f021926b P e632c553ff4341328acf1cbaca678392 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } attrs { replace: true } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { promote: false } } }
I20240502 21:29:25.656971 1413 consensus_queue.cc:237] T f8bee9bdcbff404280f000bdaeadf465 P 7d1e5c1b047443cbb28d7b0a9744babf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } } peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "d8d7297186694f77bb98de9174e4279c" member_type: VOTER last_known_addr { host: "127.15.255.136" port: 41459 } attrs { promote: false } } peers { permanent_uuid: "16a97c1394f346f0b987f97cce7371b6" member_type: VOTER last_known_addr { host: "127.15.255.137" port: 43717 } attrs { promote: false } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } attrs { promote: false } }
I20240502 21:29:25.658730 7293 ts_tablet_manager.cc:1888] T 20a7af42e83c4639bba20f22dee69f2a P e632c553ff4341328acf1cbaca678392: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240502 21:29:25.658870 7293 log.cc:1196] T 20a7af42e83c4639bba20f22dee69f2a P e632c553ff4341328acf1cbaca678392: Deleting WAL directory at /tmp/dist-test-taskvNciQZ/test-tmp/rebalancer_tool-test.7.LocationAwareRebalancingParamTest.RebalanceWithIgnoredTServer_idx6_rf7_t7_l_3_4_3.1714685120732928-16382-0/raft_consensus-itest-cluster/ts-4/wal/wals/20a7af42e83c4639bba20f22dee69f2a
I20240502 21:29:25.659065 1818 raft_consensus.cc:2932] T 0658ecbf32754656876452b6f021926b P 153026dd7f8943cc93fedf6f48450fb4 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } attrs { replace: true } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { promote: false } } }
I20240502 21:29:25.660569 2226 raft_consensus.cc:2932] T 0658ecbf32754656876452b6f021926b P 6ab8df3c796f4b6cb3e29d5045ad3ea0 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } attrs { replace: true } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb" member_type: VOTER last_known_addr { host: "127.15.255.131" port: 36033 } } peers { permanent_uuid: "1fab5794cf1740288abd82f2cebb9b1a" member_type: VOTER last_known_addr { host: "127.15.255.138" port: 33375 } attrs { promote: false } } peers { permanent_uuid: "153026dd7f8943cc93fedf6f48450fb4" member_type: VOTER last_known_addr { host: "127.15.255.132" port: 44947 } attrs { promote: false } } }
I20240502 21:29:25.664207 1685 raft_consensus.cc:2932] T 0658ecbf32754656876452b6f021926b P 16b3a337bc5e462f83ba895ad8660deb [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 153026dd7f8943cc93fedf6f48450fb4 (127.15.255.132) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e632c553ff4341328acf1cbaca678392" member_type: VOTER last_known_addr { host: "127.15.255.133" port: 37439 } } peers { permanent_uuid: "521bb93035324cecbebed2f6f2a630f8" member_type: VOTER last_known_addr { host: "127.15.255.134" port: 40821 } attrs { replace: true } } peers { permanent_uuid: "6ab8df3c796f4b6cb3e29d5045ad3ea0" member_type: VOTER last_known_addr { host: "127.15.255.135" port: 34701 } } peers { permanent_uuid: "a62a935ff43d494ca35d0c417943133d" member_type: VOTER last_known_addr { host: "127.15.255.130" port: 37991 } } peers { permanent_uuid: "7d1e5c1b047443cbb28d7b0a9744babf" member_type: VOTER last_known_addr { host: "127.15.255.129" port: 37897 } } peers { permanent_uuid: "16b3a337bc5e462f83ba895ad8660deb"